48 lines
2.7 KiB
Markdown
48 lines
2.7 KiB
Markdown
# Recovery on a clean VPS
|
|
|
|
## Scope and safety
|
|
|
|
The bootstrap below prepares Ubuntu packages and directories only. It is not a
|
|
complete recovery: production secrets, data, reverse proxy, TLS, registry
|
|
credentials, backup restore, and deploy-helper still require owner-controlled
|
|
setup. Run only on the new recovery host after verifying its identity. Never
|
|
run recovery commands against the existing production by accident.
|
|
|
|
## Procedure
|
|
|
|
1. Provision a clean Ubuntu Server 24.04 host and confirm hostname, IP, console
|
|
access, and a tested owner recovery path.
|
|
2. Create the SSH-authorized `codex` account without sudo or Docker group access.
|
|
Preserve the host's existing root SSH and firewall policy until an owner has
|
|
reviewed the change.
|
|
3. Check out the approved Git revision, root-owned and not group/world-writable,
|
|
at `/opt/akyldash/current`, then run
|
|
`sudo bash infra/bootstrap/bootstrap-ubuntu-24.04.sh --recovery-target`. The
|
|
recovery mode refuses the inventoried production hostname/IP and writes a
|
|
root-only marker at `/etc/akyldash/recovery-target`. It installs Docker
|
|
Engine, Compose, Git, Docker log limits, and target directories.
|
|
4. Install the limited read-only verifier and rollback helper with
|
|
`sudo bash infra/deploy/install-helper.sh`. It validates the root-owned
|
|
checkout and gives `codex` access only to those fixed commands.
|
|
5. Configure DNS to point through Cloudflare to the recovery host and allow the
|
|
owner to review/open TCP `80` and `443`. The production Compose Caddy service
|
|
obtains its origin certificate automatically. Verify Cloudflare Full
|
|
(strict) end to end before cutover.
|
|
6. Transfer the protected backup to the new host through the owner's approved
|
|
channel. The archive contains the production environment and review secret;
|
|
keep it root-only and never paste secret values into chat.
|
|
7. Extract the verified archive to a new `/var/tmp/akyldash-restore-*` staging
|
|
directory with `infra/backup/archive.py`, then run
|
|
`sudo python3 infra/backup/apply_restore.py --staging-dir <staging-dir>`.
|
|
Apply requires the recovery marker, matching clean Git revision, empty data
|
|
path, absent Compose volumes/config files, and no running containers. It
|
|
refuses the inventoried production host and leaves services stopped.
|
|
8. Build the backend and OpenSearch images from the matching checked-out Git
|
|
revision and explicitly start Compose with `--wait`. Verify HTTPS and
|
|
container health, record the backend image ID as described in
|
|
[DEPLOYMENT.md](DEPLOYMENT.md), then record actual versions and checks in
|
|
[CURRENT_STATE.md](CURRENT_STATE.md).
|
|
|
|
Do not repoint DNS or retire the old host until the owner verifies recovery and
|
|
approves the cutover. The restore and cutover have not been practiced yet.
|