3.3 KiB
Backup and restore
Current status
No production data or backup existed during the 2026-10-11 audit. The repository
now contains infra/backup/archive.py; it is not installed or run on the VPS,
and /var/backups/akyldash remains a planned root-only location.
Required release policy
- Create a backup before every production change, including any future schema or index migration.
- Include the normalized corpus,
search-users.sqlite3,search-reviews.sqlite3, the OpenSearch index/snapshot, deployment digest, Caddy's certificate data, and required secret file in the protected backup set. Store secrets and Caddy private keys with equivalent access controls. - Verify archive readability, checksums, and SQLite integrity before replacing the previous known-good backup.
- Keep one current pre-release backup. If deployment fails, retain the backup required to restore the previous state.
- Test restore into an isolated target before relying on the backup. Do not restore over production as a test.
Cold backup procedure
The script requires every Compose container to be stopped. Schedule downtime, then run from the approved checkout on the VPS:
cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
sudo python3 infra/backup/archive.py create \
--compose-file deploy/production/compose.yaml \
--env-file /etc/akyldash/production.env \
--output /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
Replace the timestamp placeholder with one real UTC timestamp before running.
The create command refuses running services and archives the application data,
OpenSearch volume, Caddy data/config volumes, production environment file, and
review secret. It records the checked-out Git revision, deployment-file hashes,
and image references/IDs for each stopped Compose service, writes a SHA-256
sidecar, and refuses to overwrite an existing archive. It also runs SQLite
quick_check. Start the stack again only after verification succeeds:
cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
Keep the local archive root-only, then copy it to separately managed encrypted
storage. A local archive alone does not protect against loss of the VPS disk.
Restore extracts only to a new akyldash-restore-* directory under /tmp or
/var/tmp and validates the checksum and SQLite databases. Applying the staged
contents requires the separate recovery-target tool; it refuses the inventoried
production host, existing data/volumes, and an unmarked host. Example isolated
restore:
sudo python3 infra/backup/archive.py restore /var/backups/akyldash/backup.tar.gz \
--target-dir /var/tmp/akyldash-restore-test
This validates and stages the archive. On the replacement host only, after
running bootstrap with --recovery-target and checking out the matching Git
revision, apply the staged data:
sudo python3 infra/backup/apply_restore.py \
--staging-dir /var/tmp/akyldash-restore-test
The apply command restores data and named volumes but leaves services stopped. Test the full procedure on a separate recovery host before relying on it. Do not run an unreviewed restore over existing data.