akyldash/docs/operations/RECOVERY.md

Recovery on a clean VPS

Scope and safety

The bootstrap below prepares Ubuntu packages and directories only. It is not a complete recovery: production secrets, data, reverse proxy, TLS, registry credentials, backup restore, and deploy-helper still require owner-controlled setup. Run only on the new recovery host after verifying its identity. Never run recovery commands against the existing production by accident.

Procedure

1. Provision a clean Ubuntu Server 24.04 host and confirm hostname, IP, console access, and a tested owner recovery path.
2. Create the SSH-authorized codex account without sudo or Docker group access. Preserve the host's existing root SSH and firewall policy until an owner has reviewed the change.
3. Check out the approved Git revision, root-owned and not group/world-writable, at /opt/akyldash/current, then run sudo bash infra/bootstrap/bootstrap-ubuntu-24.04.sh --recovery-target. The recovery mode refuses the inventoried production hostname/IP and writes a root-only marker at /etc/akyldash/recovery-target. It installs Docker Engine, Compose, Git, Docker log limits, and target directories.
4. Install the limited read-only verifier and rollback helper with sudo bash infra/deploy/install-helper.sh. It validates the root-owned checkout and gives codex access only to those fixed commands.
5. Configure DNS to point through Cloudflare to the recovery host and allow the owner to review/open TCP 80 and 443. The production Compose Caddy service obtains its origin certificate automatically. Verify Cloudflare Full (strict) end to end before cutover.
6. Transfer the protected backup to the new host through the owner's approved channel. The archive contains the production environment and review secret; keep it root-only and never paste secret values into chat.
7. Extract the verified archive to a new /var/tmp/akyldash-restore-* staging directory with infra/backup/archive.py, then run sudo python3 infra/backup/apply_restore.py --staging-dir <staging-dir>. Apply requires the recovery marker, matching clean Git revision, empty data path, absent Compose volumes/config files, and no running containers. It refuses the inventoried production host and leaves services stopped.
8. Build the backend and OpenSearch images from the matching checked-out Git revision and explicitly start Compose with --wait. Verify HTTPS and container health, record the backend image ID as described in DEPLOYMENT.md, then record actual versions and checks in CURRENT_STATE.md.

Do not repoint DNS or retire the old host until the owner verifies recovery and approves the cutover. The restore and cutover have not been practiced yet.