Files
akyldash/docs/operations/ROLLBACK.md
T
agent 9d8ce6b67f
CI / tests-and-images (pull_request) Successful in 2m43s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
fix: address production release review findings
2026-10-25 13:44:56 +03:00

2.9 KiB

Rollback

Current status

On 2026-10-25 the public /review endpoint and direct origin returned HTTP 200, but current containers, data/index state, and a verified backup remain unknown. The repository rollback helper and installer are not confirmed installed on the VPS. This document describes code-level procedures only.

Before a release, the owner must verify a cold backup whose recorded backend and OpenSearch image references and image IDs match the current deployment, and check data compatibility with the target and prior versions. The owner-run infra/deploy/deploy_release.py verifies the manifest allowlist, backup, current image IDs, and registry digests; serializes against rollback using /run/akyldash-deploy.lock; atomically replaces both image refs; and starts Compose with --no-build. It is not installed as a codex sudo command and was not run on the VPS.

After a successful release, the owner can run the restricted rollback helper with the exact verified backup recorded as the previous release:

sudo -n /usr/local/sbin/akyldash-rollback \
  /var/backups/akyldash/<pre-release-backup>.tar.gz --data-compatible

The helper takes the same deployment lock, validates the selected archive against the separately pinned previous refs and image IDs, verifies image availability, makes and verifies a second cold backup, and switches both protected image refs atomically. Digest references are never retagged; legacy tag images are restored by image ID. Compose runs with --no-build, followed by a check that both running container image IDs equal the selected restore plan, followed by healthcheck and local HTTPS validation. A mismatch fails the rollback, restores the prior references when they can be verified, and keeps a secret-free recovery journal; it never records the expected IDs as if they were running. A failed cutover or rollback blocks another release until an owner resolves the journal. A legacy deployment without release state can still roll back matching legacy tags, but digest rollback is refused until the state is established by the reviewed release helper.

It refuses if the current Compose/Caddyfile differs from the backup or the backup's image records do not match release state. This does not restore data or automatically undo a release with an unknown schema change. The application has no formal migration framework; compatibility must be checked for every change. If it is uncertain, stop and ask the owner.

The initial owner-run deployment must record its running backend image ID in /opt/akyldash/deployed-image.txt with mode 0600; the rollback helper updates that record after a successful rollback, and infra/bootstrap/verify.sh checks it against the running container.

Any data restore requires the owner, a verified pre-change backup, and a maintenance window. Preserve the failed release and both backups until recovery is confirmed.