# Production deployment Compose runs Caddy, the Search API, and a private OpenSearch node. Only Caddy publishes ports `80` and `443`; backend and OpenSearch have no host ports. The OpenSearch security plugin is disabled to match the current API client, so keep it on the private network. ## Host access and configuration The production environment file is `/etc/akyldash/production.env`; the review secret is `/etc/akyldash/review_secret`. Both must stay outside the checkout, root-owned, and mode `0600`. `DATA_ROOT` must be an absolute path. The `codex` account is intentionally outside the `sudo` and `docker` groups because access to the Docker socket grants root-equivalent control. Run host operations with the narrowly required `sudo` command; do not use `docker` as `codex` directly. After reviewing the host audit and confirming these files do not already exist, the owner can create them from the approved checkout with `sudo bash infra/deploy/prepare-production-config.sh`. The script refuses to overwrite existing configuration, installs the tracked env example with mode `0600`, and generates a random review secret with mode `0600`. It does not start Compose or create volumes. Review and set `DATA_ROOT` in the protected env file before deployment. Set `DATA_ROOT` to the approved data directory and ensure the normalized corpus is at `$DATA_ROOT/minjust-normalized/`. The first host setup is documented in [`docs/operations/DEPLOYMENT.md`](../../docs/operations/DEPLOYMENT.md); the root bootstrap installs packages and creates directories, but does not deploy the application or alter SSH, firewall, user groups, or sudoers. The restricted `akyldash-compose` helper allows only root-owned Compose `config` and `status`; service start/stop remains an owner-run root operation. The repository Gitea Actions runner is an isolated CI worker described in [`docs/operations/CI_RUNNER.md`](../../docs/operations/CI_RUNNER.md). It has no production Docker socket, production env files, or application checkout. CI does not deploy or change production; all deployment remains owner-run. ## First deployment and upgrades Run commands from the root-owned checkout at `/opt/akyldash/current`. Render and review Compose before changing services: ```bash cd /opt/akyldash/current sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml config --quiet ``` On a clean first deployment, `infra/backup/archive.py create` reports that no backup is required when all persistent mounts are empty and no Compose containers exist. Continue without a backup in that case. If any persistent data exists, create and verify a backup before changing schema, images, or starting a new application version. This also applies when containers are absent, including an initial installation where the normalized corpus was preloaded under `DATA_ROOT`; create the archive before the first `up`. When named volumes do not exist, pass `--initial-deployment`; this is accepted only without Compose containers and when `DATA_ROOT` contains only `minjust-normalized/`. During upgrades, missing volumes are an error because they can indicate lost production data. The archive tool also refuses ambiguous storage state, running containers, and incomplete Compose service containers. See [`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md) for the exact commands. Do not remove an older usable backup to make room; retain the pre-change archive until the new deployment and a newer verified backup are confirmed. For an initial clean installation, start the stack only after config review: ```bash sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d --build sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml ps curl -fsS https://akyldash.kg/review >/dev/null ``` Before this step, confirm DNS and have the owner review firewall rules for TCP `80` and `443`. Caddy manages the origin certificate; confirm HTTPS before setting Cloudflare to Full (strict). On the initial start, create the `admin` account interactively: ```bash sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --create-admin ``` The account database is `$DATA_ROOT/search-users.sqlite3` and is included in the `data` backup. Admin password reset is also interactive: ```bash sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --reset-admin-password ``` After the initial OpenSearch index is loaded and validated, the `admin` account is created, and `/review` works over HTTPS, create and verify the required post-setup baseline backup before treating deployment as complete. It captures the initialized index, account database, and Caddy certificate data. Follow the exact stop/create/verify/start sequence in [`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md#initial-post-setup-baseline). ## Backup, restore, and rollback Backups are cold and require downtime. Stop services, create the archive, then verify both its checksum and contents before proceeding. Do not run an upgrade or schema change until verification succeeds. Keep archives root-only under `/var/backups/akyldash` and copy them to separately managed encrypted storage. Restore is a staged operation on a marked replacement host; it does not overwrite production. For application-only rollback, use the guarded helper with a verified pre-release backup and explicit data-compatibility review. Follow the procedures in [`BACKUP.md`](../../docs/operations/BACKUP.md), [`ROLLBACK.md`](../../docs/operations/ROLLBACK.md), and [`RECOVERY.md`](../../docs/operations/RECOVERY.md). Automated production deployment is not enabled. Deployments remain owner-run; record the release and verify service health before considering it complete. ## Image release and owner-run cutover PR CI still builds both production images with Kaniko `--no-push`. The separate `Release production images` workflow is manual and runs only when dispatched from protected `main`. It requires repository secrets named `REGISTRY_USERNAME` and `REGISTRY_PASSWORD`; without them it fails before any build. Configure those secrets only after the owner confirms a dedicated Gitea account/token with write access to the two repository packages and no unneeded administration rights. Do not reuse a runner or production host credential. The workflow uses the Gitea host and fixed package namespace `git.vakanaut.ru/admin/akyldash/{backend,opensearch}`, pushes unique commit/run-ID tags, captures registry digests, validates the release descriptor, and uploads the JSON plus SHA-256 sidecar as a Gitea Actions artifact using a full-SHA-pinned Gitea-compatible v3 action. PR CI also uploads a tiny synthetic marker with one-day retention to verify the runner's runtime artifact path without registry credentials. The runner cache stays disabled; artifact compatibility is based on the runtime API supported by act_runner 0.2.13/Gitea 1.25.5 and will be empirically checked by PR CI. Registry and artifact retention must preserve release digests and descriptors for the full rollback window; the owner must confirm both policies. The schema-v3 descriptor records source repository/ref/commit, workflow run ID, pinned Kaniko builder and both image digests. It is an unsigned trace record, not a cryptographic attestation; deployment relies on retrieving it from the authenticated Gitea run artifact and verifying its checksum and repository allowlist. Do not treat the sidecar alone as proof of authenticity. For a future owner-approved cutover, use the root-owned checkout and run `infra/deploy/deploy_release.py` as root with the downloaded manifest, its matching verified cold backup, and `infra/deploy/release-policy.json`, plus owner-verified non-empty Russian and Kyrgyz canary queries, for example `--canary-ru '' --canary-ky ''`. The helper checks both queries against the public Search API through local HTTPS after Compose reports healthy, and requires at least one result with a document code before recording the new release state. Select queries only after corpus/index canaries have been verified; do not use generic terms that can match unrelated test data. The helper requires the backup image references and image IDs to match the live stack, serializes deploy and rollback with `/run/akyldash-deploy.lock`, atomically switches both protected env image refs, and always runs Compose with `--no-build`. It records recovery state before changing references. If ref replacement reports an error after an atomic rename, recovery rereads the actual file and restores the prior pair before restarting; an unexpected or unverifiable pair stops recovery and leaves a secret-free journal. A failed cutover refuses a subsequent release until an owner resolves the journal. Rollback accepts a digest-pinned previous release only when that backup matches the separately recorded previous refs and image IDs. Legacy tag rollback remains supported. Neither the helper nor registry secrets are installed on the VPS by this PR; deployment remains an owner-only operation and must wait for host/data/backup review. The exact sanitized evidence required from the owner is listed in [`DEPLOYMENT_OWNER_PACKET.md`](../../docs/operations/DEPLOYMENT_OWNER_PACKET.md).