Files
akyldash/docs/operations/BACKUP.md
T
agent 6f809d3196
CI / tests-and-images (pull_request) Has been cancelled
Add guarded rollback and recovery apply tools
2026-10-11 19:30:22 +03:00

3.3 KiB

Backup and restore

Current status

No production data or backup existed during the 2026-10-11 audit. The repository now contains infra/backup/archive.py; it is not installed or run on the VPS, and /var/backups/akyldash remains a planned root-only location.

Required release policy

  • Create a backup before every production change, including any future schema or index migration.
  • Include the normalized corpus, search-users.sqlite3, search-reviews.sqlite3, the OpenSearch index/snapshot, deployment digest, Caddy's certificate data, and required secret file in the protected backup set. Store secrets and Caddy private keys with equivalent access controls.
  • Verify archive readability, checksums, and SQLite integrity before replacing the previous known-good backup.
  • Keep one current pre-release backup. If deployment fails, retain the backup required to restore the previous state.
  • Test restore into an isolated target before relying on the backup. Do not restore over production as a test.

Cold backup procedure

The script requires every Compose container to be stopped. Schedule downtime, then run from the approved checkout on the VPS:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
sudo python3 infra/backup/archive.py create \
  --compose-file deploy/production/compose.yaml \
  --env-file /etc/akyldash/production.env \
  --output /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz

Replace the timestamp placeholder with one real UTC timestamp before running. The create command refuses running services and archives the application data, OpenSearch volume, Caddy data/config volumes, production environment file, and review secret. It records the checked-out Git revision, deployment-file hashes, and image references/IDs for each stopped Compose service, writes a SHA-256 sidecar, and refuses to overwrite an existing archive. It also runs SQLite quick_check. Start the stack again only after verification succeeds:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d

Keep the local archive root-only, then copy it to separately managed encrypted storage. A local archive alone does not protect against loss of the VPS disk. Restore extracts only to a new akyldash-restore-* directory under /tmp or /var/tmp and validates the checksum and SQLite databases. Applying the staged contents requires the separate recovery-target tool; it refuses the inventoried production host, existing data/volumes, and an unmarked host. Example isolated restore:

sudo python3 infra/backup/archive.py restore /var/backups/akyldash/backup.tar.gz \
  --target-dir /var/tmp/akyldash-restore-test

This validates and stages the archive. On the replacement host only, after running bootstrap with --recovery-target and checking out the matching Git revision, apply the staged data:

sudo python3 infra/backup/apply_restore.py \
  --staging-dir /var/tmp/akyldash-restore-test

The apply command restores data and named volumes but leaves services stopped. Test the full procedure on a separate recovery host before relying on it. Do not run an unreviewed restore over existing data.