Files
akyldash/docs/operations/BACKUP.md
agent 3b41d1f20f
CI / tests-and-images (pull_request) Has been cancelled
Guard volume gaps and back up initialized production
2026-10-11 23:17:48 +03:00

132 lines
6.2 KiB
Markdown

# Backup and restore
## Current status
No production data or backup existed during the 2026-10-11 audit. The repository
now contains `infra/backup/archive.py`; it is not installed or run on the VPS,
and `/var/backups/akyldash` remains a planned root-only location.
## Required release policy
- Create a backup before every production change, including any future schema
or index migration.
- Include the normalized corpus, `search-users.sqlite3`,
`search-reviews.sqlite3`, the OpenSearch index/snapshot, deployment digest,
Caddy's certificate data, and required secret file in the protected backup
set. Store secrets and Caddy private keys with equivalent access controls.
- Verify archive readability, checksums, and SQLite integrity before relying
on it. Never overwrite or delete the only usable backup.
- Keep one current pre-release backup. If deployment fails, retain the backup
required to restore the previous state.
- Test restore into an isolated target before relying on the backup. Do not
restore over production as a test.
## Clean first deployment
No backup is needed only when Compose has no containers and all four
persistent mounts are confirmed empty. For a first install with a preloaded
corpus and not-yet-created named volumes, use `--initial-deployment` to create
the pre-start backup. This flag is accepted only with no Compose containers and
when `DATA_ROOT` contains exactly `minjust-normalized/`; it archives absent
named volumes as empty components. In every normal backup, an absent volume is
an error because it could mean that a previous installation lost data.
Missing mount definitions, inaccessible Docker state, or partial service
containers are also errors and must stop deployment. Do not create placeholder
containers to make backup pass.
For the corpus-only initial case, run this before the first `up` from the
approved checkout. Replace the timestamp with one real UTC timestamp:
```bash
cd /opt/akyldash/current
sudo python3 infra/backup/archive.py create \
--compose-file deploy/production/compose.yaml \
--env-file /etc/akyldash/production.env \
--initial-deployment \
--output /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
```
## Cold backup procedure
The script requires every Compose container to be stopped. Schedule downtime,
then run from the approved checkout on the VPS:
```bash
cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
sudo python3 infra/backup/archive.py create \
--compose-file deploy/production/compose.yaml \
--env-file /etc/akyldash/production.env \
--output /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
```
Replace the timestamp placeholder with one real UTC timestamp before running.
On a clean first deployment, `create` exits with a clear message that backup
can be skipped. With existing data and all expected volumes present, it
archives the storage even when no Compose containers exist. A preloaded corpus
with no named volumes requires the explicit `--initial-deployment` mode;
otherwise missing volumes are rejected. Partial service containers or running
services are rejected as unsafe. The create command archives the application data,
OpenSearch volume, Caddy data/config volumes, production environment file, and
review secret. It records the checked-out Git revision, deployment-file hashes,
and image references/IDs for each stopped Compose service, writes a SHA-256
sidecar, and refuses to overwrite an existing archive. It also runs SQLite
`quick_check`. Start the stack again only after verification succeeds:
```bash
cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
```
## Initial post-setup baseline
After the first index has been loaded and validated, the initial `admin` account
has been created, and `/review` responds over HTTPS, create a second backup.
This one captures the initialized index, account database, and newly created
Caddy/OpenSearch volumes. Stop all services, create and verify the archive,
then start the stack again:
```bash
cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
sudo python3 infra/backup/archive.py create \
--compose-file deploy/production/compose.yaml \
--env-file /etc/akyldash/production.env \
--output /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
```
Use one real UTC timestamp in both archive paths. Do not treat initial
production setup as complete until this baseline archive verifies. Keep the
pre-start corpus archive as well until the baseline is verified and copied to
separately managed encrypted storage.
Keep the local archive root-only, then copy it to separately managed encrypted
storage. A local archive alone does not protect against loss of the VPS disk.
Restore extracts only to a new `akyldash-restore-*` directory under `/tmp` or
`/var/tmp` and validates the checksum and SQLite databases. Applying the staged
contents requires the separate recovery-target tool; it refuses the inventoried
production host, existing data/volumes, and an unmarked host. Example isolated
restore:
```bash
sudo python3 infra/backup/archive.py restore /var/backups/akyldash/backup.tar.gz \
--target-dir /var/tmp/akyldash-restore-test
```
This validates and stages the archive. On the replacement host only, after
running bootstrap with `--recovery-target` and checking out the matching Git
revision, apply the staged data:
```bash
sudo python3 infra/backup/apply_restore.py \
--staging-dir /var/tmp/akyldash-restore-test
```
The apply command restores data and named volumes but leaves services stopped.
Test the full procedure on a separate recovery host before relying on it. Do
not run an unreviewed restore over existing data.