Compose runs Caddy, the Search API, and a private OpenSearch node. Only Caddy publishes ports 80 and 443; backend and OpenSearch have no host ports. The OpenSearch security plugin is disabled to match the current API client, so keep it on the private network.

Host access and configuration

The production environment file is /etc/akyldash/production.env; the review secret is /etc/akyldash/review_secret. Both must stay outside the checkout, root-owned, and mode 0600. DATA_ROOT must be an absolute path. The codex account is intentionally outside the sudo and docker groups because access to the Docker socket grants root-equivalent control. Run host operations with the narrowly required sudo command; do not use docker as codex directly.

After reviewing the host audit and confirming these files do not already exist, the owner can create them from the approved checkout with sudo bash infra/deploy/prepare-production-config.sh. The script refuses to overwrite existing configuration, installs the tracked env example with mode 0600, and generates a random review secret with mode 0600. It does not start Compose or create volumes. Review and set DATA_ROOT in the protected env file before deployment.

Set DATA_ROOT to the approved data directory and ensure the normalized corpus is at $DATA_ROOT/minjust-normalized/. The first host setup is documented in docs/operations/DEPLOYMENT.md; the root bootstrap installs packages and creates directories, but does not deploy the application or alter SSH, firewall, user groups, or sudoers. The restricted akyldash-compose helper allows only root-owned Compose config and status; service start/stop remains an owner-run root operation.

The repository Gitea Actions runner is an isolated CI worker described in docs/operations/CI_RUNNER.md. It has no production Docker socket, production env files, or application checkout. CI does not deploy or change production; all deployment remains owner-run.

First deployment and upgrades

Run commands from the root-owned checkout at /opt/akyldash/current. Render and review Compose before changing services:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml config --quiet

On a clean first deployment, infra/backup/archive.py create reports that no backup is required when all persistent mounts are empty and no Compose containers exist. Continue without a backup in that case. If any persistent data exists, create and verify a backup before changing schema, images, or starting a new application version. This also applies when containers are absent, including an initial installation where the normalized corpus was preloaded under DATA_ROOT; create the archive before the first up. When named volumes do not exist, pass --initial-deployment; this is accepted only without Compose containers and when DATA_ROOT contains only minjust-normalized/. During upgrades, missing volumes are an error because they can indicate lost production data. The archive tool also refuses ambiguous storage state, running containers, and incomplete Compose service containers. See docs/operations/BACKUP.md for the exact commands. Do not remove an older usable backup to make room; retain the pre-change archive until the new deployment and a newer verified backup are confirmed.

For an initial clean installation, start the stack only after config review:

sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d --build
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml ps
curl -fsS https://akyldash.kg/review >/dev/null

Before this step, confirm DNS and have the owner review firewall rules for TCP 80 and 443. Caddy manages the origin certificate; confirm HTTPS before setting Cloudflare to Full (strict). On the initial start, create the admin account interactively:

sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --create-admin

The account database is $DATA_ROOT/search-users.sqlite3 and is included in the data backup. Admin password reset is also interactive:

sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --reset-admin-password

After the initial OpenSearch index is loaded and validated, the admin account is created, and /review works over HTTPS, create and verify the required post-setup baseline backup before treating deployment as complete. It captures the initialized index, account database, and Caddy certificate data. Follow the exact stop/create/verify/start sequence in docs/operations/BACKUP.md.

Backup, restore, and rollback

Backups are cold and require downtime. Stop services, create the archive, then verify both its checksum and contents before proceeding. Do not run an upgrade or schema change until verification succeeds. Keep archives root-only under /var/backups/akyldash and copy them to separately managed encrypted storage.

Restore is a staged operation on a marked replacement host; it does not overwrite production. For application-only rollback, use the guarded helper with a verified pre-release backup and explicit data-compatibility review. Follow the procedures in BACKUP.md, ROLLBACK.md, and RECOVERY.md.

Automated production deployment is not enabled. Deployments remain owner-run; record the release and verify service health before considering it complete.

Image release and owner-run cutover

PR CI still builds both production images with Kaniko --no-push. The separate Release production images workflow is manual and runs only when dispatched from protected main. It requires repository secrets named REGISTRY_USERNAME and REGISTRY_PASSWORD; without them it fails before any build. Configure those secrets only after the owner confirms a dedicated Gitea account/token with write access to the two repository packages and no unneeded administration rights. Do not reuse a runner or production host credential. The workflow uses the Gitea host and fixed package namespace git.vakanaut.ru/admin/akyldash/{backend,opensearch}, pushes unique commit/run-ID tags, captures registry digests, validates the release descriptor, and uploads the JSON plus SHA-256 sidecar as a Gitea Actions artifact using a full-SHA-pinned Gitea-compatible v3 action. PR CI also uploads a tiny synthetic marker with one-day retention to verify the runner's runtime artifact path without registry credentials. The runner cache stays disabled; artifact compatibility is based on the runtime API supported by act_runner 0.2.13/Gitea 1.25.5 and will be empirically checked by PR CI. Registry and artifact retention must preserve release digests and descriptors for the full rollback window; the owner must confirm both policies.

The schema-v3 descriptor records source repository/ref/commit, workflow run ID, pinned Kaniko builder and both image digests. It is an unsigned trace record, not a cryptographic attestation; deployment relies on retrieving it from the authenticated Gitea run artifact and verifying its checksum and repository allowlist. Do not treat the sidecar alone as proof of authenticity.

For a future owner-approved cutover, use the root-owned checkout and run infra/deploy/deploy_release.py as root with the downloaded manifest, its matching verified cold backup, and infra/deploy/release-policy.json, plus owner-verified non-empty Russian and Kyrgyz canary queries, for example --canary-ru '<known Russian query>' --canary-ky '<known Kyrgyz query>'. The helper checks both queries against the public Search API through local HTTPS after Compose reports healthy, and requires at least one result with a document code before recording the new release state. Select queries only after corpus/index canaries have been verified; do not use generic terms that can match unrelated test data. The helper requires the backup image references and image IDs to match the live stack, serializes deploy and rollback with /run/akyldash-deploy.lock, atomically switches both protected env image refs, and always runs Compose with --no-build. It records recovery state before changing references. If ref replacement reports an error after an atomic rename, recovery rereads the actual file and restores the prior pair before restarting; an unexpected or unverifiable pair stops recovery and leaves a secret-free journal. A failed cutover refuses a subsequent release until an owner resolves the journal. Rollback accepts a digest-pinned previous release only when that backup matches the separately recorded previous refs and image IDs. Legacy tag rollback remains supported. Neither the helper nor registry secrets are installed on the VPS by this PR; deployment remains an owner-only operation and must wait for host/data/backup review. The exact sanitized evidence required from the owner is listed in DEPLOYMENT_OWNER_PACKET.md.