# Backup and restore ## Current status No production data or backup existed during the 2026-10-11 audit. The repository now contains `infra/backup/archive.py`; it is not installed or run on the VPS, and `/var/backups/akyldash` remains a planned root-only location. ## Required release policy - Create a backup before every production change, including any future schema or index migration. - Include the normalized corpus, `search-users.sqlite3`, `search-reviews.sqlite3`, the OpenSearch index/snapshot, deployment digest, Caddy's certificate data, and required secret file in the protected backup set. Store secrets and Caddy private keys with equivalent access controls. - Verify archive readability, checksums, and SQLite integrity before relying on it. Never overwrite or delete the only usable backup. - Keep one current pre-release backup. If deployment fails, retain the backup required to restore the previous state. - Test restore into an isolated target before relying on the backup. Do not restore over production as a test. ## Clean first deployment No backup is needed only when Compose has no containers and all four persistent mounts are confirmed empty. For a first install with a preloaded corpus and not-yet-created named volumes, use `--initial-deployment` to create the pre-start backup. This flag is accepted only with no Compose containers and when `DATA_ROOT` contains exactly `minjust-normalized/`; it archives absent named volumes as empty components. In every normal backup, an absent volume is an error because it could mean that a previous installation lost data. Missing mount definitions, inaccessible Docker state, or partial service containers are also errors and must stop deployment. Do not create placeholder containers to make backup pass. For the corpus-only initial case, run this before the first `up` from the approved checkout. Replace the timestamp with one real UTC timestamp: ```bash cd /opt/akyldash/current sudo python3 infra/backup/archive.py create \ --compose-file deploy/production/compose.yaml \ --env-file /etc/akyldash/production.env \ --initial-deployment \ --output /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz ``` ## Cold backup procedure The script requires every Compose container to be stopped. Schedule downtime, then run from the approved checkout on the VPS: ```bash cd /opt/akyldash/current sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop sudo python3 infra/backup/archive.py create \ --compose-file deploy/production/compose.yaml \ --env-file /etc/akyldash/production.env \ --output /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz ``` Replace the timestamp placeholder with one real UTC timestamp before running. On a clean first deployment, `create` exits with a clear message that backup can be skipped. With existing data and all expected volumes present, it archives the storage even when no Compose containers exist. A preloaded corpus with no named volumes requires the explicit `--initial-deployment` mode; otherwise missing volumes are rejected. Partial service containers or running services are rejected as unsafe. The create command archives the application data, OpenSearch volume, Caddy data/config volumes, production environment file, and review secret. It records the checked-out Git revision, deployment-file hashes, and image references/IDs for each stopped Compose service, writes a SHA-256 sidecar, and refuses to overwrite an existing archive. It also runs SQLite `quick_check`. Start the stack again only after verification succeeds: ```bash cd /opt/akyldash/current sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d ``` ## Initial post-setup baseline After the first index has been loaded and validated, the initial `admin` account has been created, and `/review` responds over HTTPS, create a second backup. This one captures the initialized index, account database, and newly created Caddy/OpenSearch volumes. Stop all services, create and verify the archive, then start the stack again: ```bash cd /opt/akyldash/current sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop sudo python3 infra/backup/archive.py create \ --compose-file deploy/production/compose.yaml \ --env-file /etc/akyldash/production.env \ --output /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d ``` Use one real UTC timestamp in both archive paths. Do not treat initial production setup as complete until this baseline archive verifies. Keep the pre-start corpus archive as well until the baseline is verified and copied to separately managed encrypted storage. Keep the local archive root-only, then copy it to separately managed encrypted storage. A local archive alone does not protect against loss of the VPS disk. Restore extracts only to a new `akyldash-restore-*` directory under `/tmp` or `/var/tmp` and validates the checksum and SQLite databases. Applying the staged contents requires the separate recovery-target tool; it refuses the inventoried production host, existing data/volumes, and an unmarked host. Example isolated restore: ```bash sudo python3 infra/backup/archive.py restore /var/backups/akyldash/backup.tar.gz \ --target-dir /var/tmp/akyldash-restore-test ``` This validates and stages the archive. On the replacement host only, after running bootstrap with `--recovery-target` and checking out the matching Git revision, apply the staged data: ```bash sudo python3 infra/backup/apply_restore.py \ --staging-dir /var/tmp/akyldash-restore-test ``` The apply command restores data and named volumes but leaves services stopped. Test the full procedure on a separate recovery host before relying on it. Do not run an unreviewed restore over existing data.