132 lines
6.2 KiB
Markdown
132 lines
6.2 KiB
Markdown
# Backup and restore
|
|
|
|
## Current status
|
|
|
|
No production data or backup existed during the 2026-10-11 audit. The repository
|
|
now contains `infra/backup/archive.py`; it is not installed or run on the VPS,
|
|
and `/var/backups/akyldash` remains a planned root-only location.
|
|
|
|
## Required release policy
|
|
|
|
- Create a backup before every production change, including any future schema
|
|
or index migration.
|
|
- Include the normalized corpus, `search-users.sqlite3`,
|
|
`search-reviews.sqlite3`, the OpenSearch index/snapshot, deployment digest,
|
|
Caddy's certificate data, and required secret file in the protected backup
|
|
set. Store secrets and Caddy private keys with equivalent access controls.
|
|
- Verify archive readability, checksums, and SQLite integrity before relying
|
|
on it. Never overwrite or delete the only usable backup.
|
|
- Keep one current pre-release backup. If deployment fails, retain the backup
|
|
required to restore the previous state.
|
|
- Test restore into an isolated target before relying on the backup. Do not
|
|
restore over production as a test.
|
|
|
|
## Clean first deployment
|
|
|
|
No backup is needed only when Compose has no containers and all four
|
|
persistent mounts are confirmed empty. For a first install with a preloaded
|
|
corpus and not-yet-created named volumes, use `--initial-deployment` to create
|
|
the pre-start backup. This flag is accepted only with no Compose containers and
|
|
when `DATA_ROOT` contains exactly `minjust-normalized/`; it archives absent
|
|
named volumes as empty components. In every normal backup, an absent volume is
|
|
an error because it could mean that a previous installation lost data.
|
|
Missing mount definitions, inaccessible Docker state, or partial service
|
|
containers are also errors and must stop deployment. Do not create placeholder
|
|
containers to make backup pass.
|
|
|
|
For the corpus-only initial case, run this before the first `up` from the
|
|
approved checkout. Replace the timestamp with one real UTC timestamp:
|
|
|
|
```bash
|
|
cd /opt/akyldash/current
|
|
sudo python3 infra/backup/archive.py create \
|
|
--compose-file deploy/production/compose.yaml \
|
|
--env-file /etc/akyldash/production.env \
|
|
--initial-deployment \
|
|
--output /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
|
|
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
|
|
```
|
|
|
|
## Cold backup procedure
|
|
|
|
The script requires every Compose container to be stopped. Schedule downtime,
|
|
then run from the approved checkout on the VPS:
|
|
|
|
```bash
|
|
cd /opt/akyldash/current
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
|
|
sudo python3 infra/backup/archive.py create \
|
|
--compose-file deploy/production/compose.yaml \
|
|
--env-file /etc/akyldash/production.env \
|
|
--output /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
|
|
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYMMDDTHHMMSSZ.tar.gz
|
|
```
|
|
|
|
Replace the timestamp placeholder with one real UTC timestamp before running.
|
|
On a clean first deployment, `create` exits with a clear message that backup
|
|
can be skipped. With existing data and all expected volumes present, it
|
|
archives the storage even when no Compose containers exist. A preloaded corpus
|
|
with no named volumes requires the explicit `--initial-deployment` mode;
|
|
otherwise missing volumes are rejected. Partial service containers or running
|
|
services are rejected as unsafe. The create command archives the application data,
|
|
OpenSearch volume, Caddy data/config volumes, production environment file, and
|
|
review secret. It records the checked-out Git revision, deployment-file hashes,
|
|
and image references/IDs for each stopped Compose service, writes a SHA-256
|
|
sidecar, and refuses to overwrite an existing archive. It also runs SQLite
|
|
`quick_check`. Start the stack again only after verification succeeds:
|
|
|
|
```bash
|
|
cd /opt/akyldash/current
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
|
|
```
|
|
|
|
## Initial post-setup baseline
|
|
|
|
After the first index has been loaded and validated, the initial `admin` account
|
|
has been created, and `/review` responds over HTTPS, create a second backup.
|
|
This one captures the initialized index, account database, and newly created
|
|
Caddy/OpenSearch volumes. Stop all services, create and verify the archive,
|
|
then start the stack again:
|
|
|
|
```bash
|
|
cd /opt/akyldash/current
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
|
|
sudo python3 infra/backup/archive.py create \
|
|
--compose-file deploy/production/compose.yaml \
|
|
--env-file /etc/akyldash/production.env \
|
|
--output /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
|
|
sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
|
|
```
|
|
|
|
Use one real UTC timestamp in both archive paths. Do not treat initial
|
|
production setup as complete until this baseline archive verifies. Keep the
|
|
pre-start corpus archive as well until the baseline is verified and copied to
|
|
separately managed encrypted storage.
|
|
|
|
Keep the local archive root-only, then copy it to separately managed encrypted
|
|
storage. A local archive alone does not protect against loss of the VPS disk.
|
|
Restore extracts only to a new `akyldash-restore-*` directory under `/tmp` or
|
|
`/var/tmp` and validates the checksum and SQLite databases. Applying the staged
|
|
contents requires the separate recovery-target tool; it refuses the inventoried
|
|
production host, existing data/volumes, and an unmarked host. Example isolated
|
|
restore:
|
|
|
|
```bash
|
|
sudo python3 infra/backup/archive.py restore /var/backups/akyldash/backup.tar.gz \
|
|
--target-dir /var/tmp/akyldash-restore-test
|
|
```
|
|
|
|
This validates and stages the archive. On the replacement host only, after
|
|
running bootstrap with `--recovery-target` and checking out the matching Git
|
|
revision, apply the staged data:
|
|
|
|
```bash
|
|
sudo python3 infra/backup/apply_restore.py \
|
|
--staging-dir /var/tmp/akyldash-restore-test
|
|
```
|
|
|
|
The apply command restores data and named volumes but leaves services stopped.
|
|
Test the full procedure on a separate recovery host before relying on it. Do
|
|
not run an unreviewed restore over existing data.
|