Files
akyldash/docs/operations/BACKUP.md
agent 3b41d1f20f
CI / tests-and-images (pull_request) Has been cancelled
Guard volume gaps and back up initialized production
2026-10-11 23:17:48 +03:00

6.2 KiB

Backup and restore

Current status

No production data or backup existed during the 2026-10-11 audit. The repository now contains infra/backup/archive.py; it is not installed or run on the VPS, and /var/backups/akyldash remains a planned root-only location.

Required release policy

  • Create a backup before every production change, including any future schema or index migration.
  • Include the normalized corpus, search-users.sqlite3, search-reviews.sqlite3, the OpenSearch index/snapshot, deployment digest, Caddy's certificate data, and required secret file in the protected backup set. Store secrets and Caddy private keys with equivalent access controls.
  • Verify archive readability, checksums, and SQLite integrity before relying on it. Never overwrite or delete the only usable backup.
  • Keep one current pre-release backup. If deployment fails, retain the backup required to restore the previous state.
  • Test restore into an isolated target before relying on the backup. Do not restore over production as a test.

Clean first deployment

No backup is needed only when Compose has no containers and all four persistent mounts are confirmed empty. For a first install with a preloaded corpus and not-yet-created named volumes, use --initial-deployment to create the pre-start backup. This flag is accepted only with no Compose containers and when DATA_ROOT contains exactly minjust-normalized/; it archives absent named volumes as empty components. In every normal backup, an absent volume is an error because it could mean that a previous installation lost data. Missing mount definitions, inaccessible Docker state, or partial service containers are also errors and must stop deployment. Do not create placeholder containers to make backup pass.

For the corpus-only initial case, run this before the first up from the approved checkout. Replace the timestamp with one real UTC timestamp:

cd /opt/akyldash/current
sudo python3 infra/backup/archive.py create \
  --compose-file deploy/production/compose.yaml \
  --env-file /etc/akyldash/production.env \
  --initial-deployment \
  --output /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz

Cold backup procedure

The script requires every Compose container to be stopped. Schedule downtime, then run from the approved checkout on the VPS:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
sudo python3 infra/backup/archive.py create \
  --compose-file deploy/production/compose.yaml \
  --env-file /etc/akyldash/production.env \
  --output /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz

Replace the timestamp placeholder with one real UTC timestamp before running. On a clean first deployment, create exits with a clear message that backup can be skipped. With existing data and all expected volumes present, it archives the storage even when no Compose containers exist. A preloaded corpus with no named volumes requires the explicit --initial-deployment mode; otherwise missing volumes are rejected. Partial service containers or running services are rejected as unsafe. The create command archives the application data, OpenSearch volume, Caddy data/config volumes, production environment file, and review secret. It records the checked-out Git revision, deployment-file hashes, and image references/IDs for each stopped Compose service, writes a SHA-256 sidecar, and refuses to overwrite an existing archive. It also runs SQLite quick_check. Start the stack again only after verification succeeds:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d

Initial post-setup baseline

After the first index has been loaded and validated, the initial admin account has been created, and /review responds over HTTPS, create a second backup. This one captures the initialized index, account database, and newly created Caddy/OpenSearch volumes. Stop all services, create and verify the archive, then start the stack again:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
sudo python3 infra/backup/archive.py create \
  --compose-file deploy/production/compose.yaml \
  --env-file /etc/akyldash/production.env \
  --output /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d

Use one real UTC timestamp in both archive paths. Do not treat initial production setup as complete until this baseline archive verifies. Keep the pre-start corpus archive as well until the baseline is verified and copied to separately managed encrypted storage.

Keep the local archive root-only, then copy it to separately managed encrypted storage. A local archive alone does not protect against loss of the VPS disk. Restore extracts only to a new akyldash-restore-* directory under /tmp or /var/tmp and validates the checksum and SQLite databases. Applying the staged contents requires the separate recovery-target tool; it refuses the inventoried production host, existing data/volumes, and an unmarked host. Example isolated restore:

sudo python3 infra/backup/archive.py restore /var/backups/akyldash/backup.tar.gz \
  --target-dir /var/tmp/akyldash-restore-test

This validates and stages the archive. On the replacement host only, after running bootstrap with --recovery-target and checking out the matching Git revision, apply the staged data:

sudo python3 infra/backup/apply_restore.py \
  --staging-dir /var/tmp/akyldash-restore-test

The apply command restores data and named volumes but leaves services stopped. Test the full procedure on a separate recovery host before relying on it. Do not run an unreviewed restore over existing data.