Files
agent e034fdb816
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
feat(infra): prepare isolated repository CI runner
2026-10-12 00:35:42 +03:00
..

Production deployment

Compose runs Caddy, the Search API, and a private OpenSearch node. Only Caddy publishes ports 80 and 443; backend and OpenSearch have no host ports. The OpenSearch security plugin is disabled to match the current API client, so keep it on the private network.

Host access and configuration

The production environment file is /etc/akyldash/production.env; the review secret is /etc/akyldash/review_secret. Both must stay outside the checkout, root-owned, and mode 0600. DATA_ROOT must be an absolute path. The codex account is intentionally outside the sudo and docker groups because access to the Docker socket grants root-equivalent control. Run host operations with the narrowly required sudo command; do not use docker as codex directly.

After reviewing the host audit and confirming these files do not already exist, the owner can create them from the approved checkout with sudo bash infra/deploy/prepare-production-config.sh. The script refuses to overwrite existing configuration, installs the tracked env example with mode 0600, and generates a random review secret with mode 0600. It does not start Compose or create volumes. Review and set DATA_ROOT in the protected env file before deployment.

Set DATA_ROOT to the approved data directory and ensure the normalized corpus is at $DATA_ROOT/minjust-normalized/. The first host setup is documented in docs/operations/DEPLOYMENT.md; the root bootstrap installs packages and creates directories, but does not deploy the application or alter SSH, firewall, user groups, or sudoers. The restricted akyldash-compose helper allows only root-owned Compose config and status; service start/stop remains an owner-run root operation.

The repository Gitea Actions runner is an isolated CI worker described in docs/operations/CI_RUNNER.md. It has no production Docker socket, production env files, or application checkout. CI does not deploy or change production; all deployment remains owner-run.

First deployment and upgrades

Run commands from the root-owned checkout at /opt/akyldash/current. Render and review Compose before changing services:

cd /opt/akyldash/current
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml config --quiet

On a clean first deployment, infra/backup/archive.py create reports that no backup is required when all persistent mounts are empty and no Compose containers exist. Continue without a backup in that case. If any persistent data exists, create and verify a backup before changing schema, images, or starting a new application version. This also applies when containers are absent, including an initial installation where the normalized corpus was preloaded under DATA_ROOT; create the archive before the first up. When named volumes do not exist, pass --initial-deployment; this is accepted only without Compose containers and when DATA_ROOT contains only minjust-normalized/. During upgrades, missing volumes are an error because they can indicate lost production data. The archive tool also refuses ambiguous storage state, running containers, and incomplete Compose service containers. See docs/operations/BACKUP.md for the exact commands. Do not remove an older usable backup to make room; retain the pre-change archive until the new deployment and a newer verified backup are confirmed.

For an initial clean installation, start the stack only after config review:

sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d --build
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml ps
curl -fsS https://akyldash.kg/review >/dev/null

Before this step, confirm DNS and have the owner review firewall rules for TCP 80 and 443. Caddy manages the origin certificate; confirm HTTPS before setting Cloudflare to Full (strict). On the initial start, create the admin account interactively:

sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --create-admin

The account database is $DATA_ROOT/search-users.sqlite3 and is included in the data backup. Admin password reset is also interactive:

sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --reset-admin-password

After the initial OpenSearch index is loaded and validated, the admin account is created, and /review works over HTTPS, create and verify the required post-setup baseline backup before treating deployment as complete. It captures the initialized index, account database, and Caddy certificate data. Follow the exact stop/create/verify/start sequence in docs/operations/BACKUP.md.

Backup, restore, and rollback

Backups are cold and require downtime. Stop services, create the archive, then verify both its checksum and contents before proceeding. Do not run an upgrade or schema change until verification succeeds. Keep archives root-only under /var/backups/akyldash and copy them to separately managed encrypted storage.

Restore is a staged operation on a marked replacement host; it does not overwrite production. For application-only rollback, use the guarded helper with a verified pre-release backup and explicit data-compatibility review. Follow the procedures in BACKUP.md, ROLLBACK.md, and RECOVERY.md.

Automated production deployment is not enabled. Deployments remain owner-run; record the release and verify service health before considering it complete.