INC-001: I deleted my own home directory on a production server

While removing an old project, a pasted command lost its line breaks and turned into rm -rf on my entire home directory. I recovered SSH access by rescuing the boot volume through a second...

severity: SEV-1 | kind: outage | date: 2026-06-10 | duration: ~3h | status: RESOLVED

Impact

4 production apps for Prasad Academic (PIMS Evalu Pro, EaseExit, Elective Registration, Status Dashboard) lost their code and config; SSH access to the server was lost.

Summary

While removing an old project, a pasted command lost its line breaks and turned into rm -rf on my entire home directory. I recovered SSH access by rescuing the boot volume through a second instance, rebuilt every service, and hardened the server so a single typo can't do this again.

Timeline

• Removed old ccd-backend from PM2 (stop, delete, save)
• Pasted cleanup commands; ls -a showed an empty home directory
• SSH failed: Permission denied (publickey) — authorized_keys gone
• Tried OCI serial console: no password set on the ubuntu user
• Tried GRUB init=/bin/bash: boot window too short, missed repeatedly
• Switched strategy: boot volume rescue
• Stopped instance, detached boot volume, launched a rescue instance with my key, attached the old boot volume as a data volume
• Mounted it, recreated .ssh/authorized_keys (700/600); hit an ownership trap: ubuntu was UID 1001 on this image, not 1000
• Reattached boot volume, started the instance, SSH restored
• Confirmed MongoDB data intact in /var/lib/mongodb; Nginx + SSL intact
• Re-cloned 4 repos, recreated .env files, rebuilt React frontends, restarted everything under PM2 (startup + save)
• Restored CI/CD: re-added deploy key, updated GitHub Actions secrets

Root cause

Pasting multi-line commands merged rm -rf ~/CCD-App-Backend and cd ~ into rm -rf ~/CCD-App-Backendcd ~. The trailing ~ was passed to rm as a second target, deleting everything inside /home/ubuntu — including all app code, .env files and .ssh/authorized_keys. No confirmation, no trash, no backup of home.

rm -rf ~/CCD-App-Backendcd ~

What didn't work

• Serial console login (no password)
• GRUB single-user mode (couldn't catch the boot menu)

Resolution

• Boot volume rescue via a temporary instance
• Full service rebuild from GitHub + recreated secrets

Action items

• [x] Daily cron backups of all .env files and the home directory
• [x] OCI boot volume backup policy enabled
• [x] trash-cli + rm -i alias: deletes now confirm and are recoverable
• [x] PM2 startup + save so services survive reboots
• [x] PM2 log rotation and a swap file to prevent disk/OOM failures
• [x] SSH keepalive configured; personal and CI keys both authorised

Lesson

_Data in the database survived because it lived outside the home directory. Everything else survived because it was in Git. The only things truly lost were the things I never backed up: secrets and keys. Now those are backed up daily._

tags: oracle-cloud, linux, ssh, pm2, nginx, incident-response