2.9 KiB
Rollback
Current status
On 2026-10-25 the public /review endpoint and direct origin returned HTTP
200, but current containers, data/index state, and a verified backup remain
unknown. The repository rollback helper and installer are not confirmed
installed on the VPS. This document describes code-level procedures only.
Before a release, the owner must verify a cold backup whose recorded backend
and OpenSearch image references and image IDs match the current deployment,
and check data compatibility with the target and prior versions. The
owner-run infra/deploy/deploy_release.py verifies the manifest allowlist,
backup, current image IDs, and registry digests; serializes against rollback
using /run/akyldash-deploy.lock; atomically replaces both image refs;
and starts Compose with --no-build. It is not installed as a codex sudo
command and was not run on the VPS.
After a successful release, the owner can run the restricted rollback helper with the exact verified backup recorded as the previous release:
sudo -n /usr/local/sbin/akyldash-rollback \
/var/backups/akyldash/<pre-release-backup>.tar.gz --data-compatible
The helper takes the same deployment lock, validates the selected archive
against the separately pinned previous refs and image IDs, verifies image
availability, makes and verifies a second cold backup, and switches both
protected image refs atomically. Digest references are never retagged; legacy
tag images are restored by image ID. Compose runs with --no-build, followed
by a check that both running container image IDs equal the selected restore
plan, followed by healthcheck and local HTTPS validation. A mismatch fails the
rollback, restores the prior references when they can be verified, and keeps a
secret-free recovery journal; it never records the expected IDs as if they
were running. A failed cutover or rollback blocks another release until an
owner resolves the journal. A legacy deployment without release state can still roll back
matching legacy tags, but digest rollback is refused until the state is
established by the reviewed release helper.
It refuses if the current Compose/Caddyfile differs from the backup or the backup's image records do not match release state. This does not restore data or automatically undo a release with an unknown schema change. The application has no formal migration framework; compatibility must be checked for every change. If it is uncertain, stop and ask the owner.
The initial owner-run deployment must record its running backend image ID in
/opt/akyldash/deployed-image.txt with mode 0600; the rollback helper updates
that record after a successful rollback, and infra/bootstrap/verify.sh checks
it against the running container.
Any data restore requires the owner, a verified pre-change backup, and a maintenance window. Preserve the failed release and both backups until recovery is confirmed.