aws ssm start-session --target i-1234567890abcdef0 aws ssm start-session --target i-1234567890abcdef0 aws ssm start-session --target i-1234567890abcdef0 # Generate and add a new key pair ssh-keygen -t ed25519 -f /tmp/newkey cat /tmp/newkey.pub >> /home/ec2-user/.ssh/authorized_keys # Or just keep using Session Manager forever (best practice) # Generate and add a new key pair ssh-keygen -t ed25519 -f /tmp/newkey cat /tmp/newkey.pub >> /home/ec2-user/.ssh/authorized_keys # Or just keep using Session Manager forever (best practice) # Generate and add a new key pair ssh-keygen -t ed25519 -f /tmp/newkey cat /tmp/newkey.pub >> /home/ec2-user/.ssh/authorized_keys # Or just keep using Session Manager forever (best practice) aws ssm get-parameter --name "/ec2rl/openssh/<instance-id>/key" \ --with-decryption \ --query "Parameter.Value" \ --output text > newkey.pem chmod 400 newkey.pem ssh -i newkey.pem ec2-user@your-ip aws ssm get-parameter --name "/ec2rl/openssh/<instance-id>/key" \ --with-decryption \ --query "Parameter.Value" \ --output text > newkey.pem chmod 400 newkey.pem ssh -i newkey.pem ec2-user@your-ip aws ssm get-parameter --name "/ec2rl/openssh/<instance-id>/key" \ --with-decryption \ --query "Parameter.Value" \ --output text > newkey.pem chmod 400 newkey.pem ssh -i newkey.pem ec2-user@your-ip aws ec2-instance-connect send-ssh-public-key \ --instance-id i-1234567890abcdef0 \ --instance-os-user ec2-user \ --ssh-public-key-file ~/.ssh/id_rsa.pub aws ec2-instance-connect send-ssh-public-key \ --instance-id i-1234567890abcdef0 \ --instance-os-user ec2-user \ --ssh-public-key-file ~/.ssh/id_rsa.pub aws ec2-instance-connect send-ssh-public-key \ --instance-id i-1234567890abcdef0 \ --instance-os-user ec2-user \ --ssh-public-key-file ~/.ssh/id_rsa.pub sudo mkdir /mnt/rescue sudo mount /dev/xvdf1 /mnt/rescue # confirm with lsblk sudo mkdir /mnt/rescue sudo mount /dev/xvdf1 /mnt/rescue # confirm with lsblk sudo mkdir /mnt/rescue sudo mount /dev/xvdf1 /mnt/rescue # confirm with lsblk cat <<EOF >> /mnt/rescue/home/ec2-user/.ssh/authorized_keys ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... your-new-public-key EOF cat <<EOF >> /mnt/rescue/home/ec2-user/.ssh/authorized_keys ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... your-new-public-key EOF cat <<EOF >> /mnt/rescue/home/ec2-user/.ssh/authorized_keys ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... your-new-public-key EOF sudo chown -R 1000:1000 /mnt/rescue/home/ec2-user/.ssh sudo chmod 700 /mnt/rescue/home/ec2-user/.ssh sudo chmod 600 /mnt/rescue/home/ec2-user/.ssh/authorized_keys sudo chown -R 1000:1000 /mnt/rescue/home/ec2-user/.ssh sudo chmod 700 /mnt/rescue/home/ec2-user/.ssh sudo chmod 600 /mnt/rescue/home/ec2-user/.ssh/authorized_keys sudo chown -R 1000:1000 /mnt/rescue/home/ec2-user/.ssh sudo chmod 700 /mnt/rescue/home/ec2-user/.ssh sudo chmod 600 /mnt/rescue/home/ec2-user/.ssh/authorized_keys **Support if you found this helpful😉** No **Money** 🙅🏻‍♀️ just **Subscribe** to my [YouTube](https://shorturl.at/lVi2G) channel. **Linktree Profile:** https://linktr.ee/DevOps_Descent **GitHub**: https://github.com/devopsdescent **Support if you found this helpful😉** No **Money** 🙅🏻‍♀️ just **Subscribe** to my [YouTube](https://shorturl.at/lVi2G) channel. **Linktree Profile:** https://linktr.ee/DevOps_Descent **GitHub**: https://github.com/devopsdescent **Support if you found this helpful😉** No **Money** 🙅🏻‍♀️ just **Subscribe** to my [YouTube](https://shorturl.at/lVi2G) channel. **Linktree Profile:** https://linktr.ee/DevOps_Descent **GitHub**: https://github.com/devopsdescent - This guide is for EBS-backed Linux instances (the vast majority). - Instance store-backed instances generally cannot be recovered this way — terminate and relaunch instead. - You need IAM permissions for EC2, SSM, and (in some cases) Systems Manager Automation. - Always take a backup first — create an EBS snapshot or AMI before making changes. - Some methods require the instance to be running; others require it to be stopped. - Suggestion: Enable detailed monitoring and CloudWatch alarms on your critical instances to detect issues early. - SSM Agent installed (pre-installed on Amazon Linux 2/2023, Ubuntu, etc.) - Instance has an IAM role with the AmazonSSMManagedInstanceCore managed policy - SSM VPC endpoints (or outbound internet on port 443) - Go to AWS Systems Manager → Session Manager - Click Start session - Select your instance and click Start session - No downtime - Fully auditable - Works on private subnets - No SSH exposure - Requires prior SSM setup - Stops your instance - Launches a temporary helper instance - Mounts your root volume - Generates a brand new SSH key pair - Injects the public key - Stores the private key encrypted in SSM Parameter Store (/ec2rl/openssh/<instance-id>/key) - Restarts your original instance - Open Systems Manager → Automation → Execute automation - Search for and select AWSSupport-ResetAccess - Choose Simple execution - Enter your Instance ID - (Optional) Provide a Subnet ID in the same AZ - Fully automated - Official AWS-supported method - Can create a backup AMI automatically - Requires SSM + Automation IAM permissions - Causes brief downtime - ec2-instance-connect package installed (default on Amazon Linux) - Security group allows inbound TCP 22 - IAM permission: ec2-instance-connect:SendSSHPublicKey - In EC2 console → Select instance → Connect - Choose EC2 Instance Connect - Click Connect → browser-based terminal opens - Quick and simple - No volume detachment - Good for emergency access - Requires SSH port 22 open - Package must already be installed - Access is temporary - Nitro-based instance - Serial Console enabled at the account level - IAM permissions for serial console access - In EC2 console → Select instance - Go to Actions → Monitor and troubleshoot → EC2 Serial Console → Connect - Works even if network/SSH is broken - Excellent for deep troubleshooting - Requires account-level enablement - Less beginner-friendly - Create a new key pair in the EC2 console - Stop the problematic instance - Note the root volume ID and device name - Launch a temporary rescue instance in the same Availability Zone - Detach root volume from original instance - Attach it to rescue instance as secondary volume (/dev/sdf) - SSH into rescue instance and mount: - Add your new public key: - Fix permissions: - Unmount, detach, reattach to original instance with correct device name, and start. - Works on almost any EBS-backed Linux instance - Most reliable fallback - Requires downtime - Manual process - Select instance → Actions → Image and templates → Create image - Once AMI is ready, launch a new instance with a new key pair - Reassign Elastic IP, security groups, and DNS records - Clean rebuild - Good for fixing other config issues - New instance ID and possibly new IP - Use Session Manager as your primary access method instead of SSH keys - Add at least 2–3 public keys to authorized_keys - Store private keys securely in AWS Secrets Manager or a password manager - Enable EC2 Instance Connect and Serial Console proactively - Use Infrastructure as Code (Terraform/CloudFormation) with proper IAM roles - Set up automated EBS snapshots and AMI creation - Tag instances clearly: AccessMethod: SSM, Environment: Prod