Files
akyldash/docs/operations/RECOVERY.md
agent 02cf4e6e95
CI / tests-and-images (pull_request) Has been cancelled
Expose drift verifier through a fixed sudo helper
2026-10-11 19:39:02 +03:00

48 lines
2.7 KiB
Markdown

# Recovery on a clean VPS
## Scope and safety
The bootstrap below prepares Ubuntu packages and directories only. It is not a
complete recovery: production secrets, data, reverse proxy, TLS, registry
credentials, backup restore, and deploy-helper still require owner-controlled
setup. Run only on the new recovery host after verifying its identity. Never
run recovery commands against the existing production by accident.
## Procedure
1. Provision a clean Ubuntu Server 24.04 host and confirm hostname, IP, console
access, and a tested owner recovery path.
2. Create the SSH-authorized `codex` account without sudo or Docker group access.
Preserve the host's existing root SSH and firewall policy until an owner has
reviewed the change.
3. Check out the approved Git revision, root-owned and not group/world-writable,
at `/opt/akyldash/current`, then run
`sudo bash infra/bootstrap/bootstrap-ubuntu-24.04.sh --recovery-target`. The
recovery mode refuses the inventoried production hostname/IP and writes a
root-only marker at `/etc/akyldash/recovery-target`. It installs Docker
Engine, Compose, Git, Docker log limits, and target directories.
4. Install the limited read-only verifier and rollback helper with
`sudo bash infra/deploy/install-helper.sh`. It validates the root-owned
checkout and gives `codex` access only to those fixed commands.
5. Configure DNS to point through Cloudflare to the recovery host and allow the
owner to review/open TCP `80` and `443`. The production Compose Caddy service
obtains its origin certificate automatically. Verify Cloudflare Full
(strict) end to end before cutover.
6. Transfer the protected backup to the new host through the owner's approved
channel. The archive contains the production environment and review secret;
keep it root-only and never paste secret values into chat.
7. Extract the verified archive to a new `/var/tmp/akyldash-restore-*` staging
directory with `infra/backup/archive.py`, then run
`sudo python3 infra/backup/apply_restore.py --staging-dir <staging-dir>`.
Apply requires the recovery marker, matching clean Git revision, empty data
path, absent Compose volumes/config files, and no running containers. It
refuses the inventoried production host and leaves services stopped.
8. Build the backend and OpenSearch images from the matching checked-out Git
revision and explicitly start Compose with `--wait`. Verify HTTPS and
container health, record the backend image ID as described in
[DEPLOYMENT.md](DEPLOYMENT.md), then record actual versions and checks in
[CURRENT_STATE.md](CURRENT_STATE.md).
Do not repoint DNS or retire the old host until the owner verifies recovery and
approves the cutover. The restore and cutover have not been practiced yet.