165 lines
9.3 KiB
Markdown
165 lines
9.3 KiB
Markdown
# Production deployment
|
|
|
|
Compose runs Caddy, the Search API, and a private OpenSearch node. Only Caddy
|
|
publishes ports `80` and `443`; backend and OpenSearch have no host ports. The
|
|
OpenSearch security plugin is disabled to match the current API client, so keep
|
|
it on the private network.
|
|
|
|
## Host access and configuration
|
|
|
|
The production environment file is `/etc/akyldash/production.env`; the review
|
|
secret is `/etc/akyldash/review_secret`. Both must stay outside the checkout,
|
|
root-owned, and mode `0600`. `DATA_ROOT` must be an absolute path. The `codex`
|
|
account is intentionally outside the `sudo` and `docker` groups because access
|
|
to the Docker socket grants root-equivalent control. Run host operations with
|
|
the narrowly required `sudo` command; do not use `docker` as `codex` directly.
|
|
|
|
After reviewing the host audit and confirming these files do not already exist,
|
|
the owner can create them from the approved checkout with
|
|
`sudo bash infra/deploy/prepare-production-config.sh`. The script refuses to
|
|
overwrite existing configuration, installs the tracked env example with mode
|
|
`0600`, and generates a random review secret with mode `0600`. It does not
|
|
start Compose or create volumes. Review and set `DATA_ROOT` in the protected
|
|
env file before deployment.
|
|
|
|
Set `DATA_ROOT` to the approved data directory and ensure the normalized corpus
|
|
is at `$DATA_ROOT/minjust-normalized/`. The first host setup is documented in
|
|
[`docs/operations/DEPLOYMENT.md`](../../docs/operations/DEPLOYMENT.md); the
|
|
root bootstrap installs packages and creates directories, but does not deploy
|
|
the application or alter SSH, firewall, user groups, or sudoers. The restricted
|
|
`akyldash-compose` helper allows only root-owned Compose `config` and `status`;
|
|
service start/stop remains an owner-run root operation.
|
|
|
|
The repository Gitea Actions runner is an isolated CI worker described in
|
|
[`docs/operations/CI_RUNNER.md`](../../docs/operations/CI_RUNNER.md). It has no
|
|
production Docker socket, production env files, or application checkout. CI
|
|
does not deploy or change production; all deployment remains owner-run.
|
|
|
|
## First deployment and upgrades
|
|
|
|
Run commands from the root-owned checkout at `/opt/akyldash/current`. Render
|
|
and review Compose before changing services:
|
|
|
|
```bash
|
|
cd /opt/akyldash/current
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml config --quiet
|
|
```
|
|
|
|
On a clean first deployment, `infra/backup/archive.py create` reports that no
|
|
backup is required when all persistent mounts are empty and no Compose
|
|
containers exist. Continue without a backup in that case. If any persistent
|
|
data exists, create and verify a backup before changing schema, images, or
|
|
starting a new application version. This also applies when containers are
|
|
absent, including an initial installation where the normalized corpus was
|
|
preloaded under `DATA_ROOT`; create the archive before the first `up`. When
|
|
named volumes do not exist, pass `--initial-deployment`; this is accepted only
|
|
without Compose containers and when `DATA_ROOT` contains only
|
|
`minjust-normalized/`. During upgrades, missing volumes are an error because
|
|
they can indicate lost production data. The archive tool also refuses
|
|
ambiguous storage state, running containers, and incomplete Compose service
|
|
containers. See
|
|
[`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md) for the exact
|
|
commands. Do not remove an older usable backup to make room; retain the
|
|
pre-change archive until the new deployment and a newer verified backup are
|
|
confirmed.
|
|
|
|
For an initial clean installation, start the stack only after config review:
|
|
|
|
```bash
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d --build
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml ps
|
|
curl -fsS https://akyldash.kg/review >/dev/null
|
|
```
|
|
|
|
Before this step, confirm DNS and have the owner review firewall rules for TCP
|
|
`80` and `443`. Caddy manages the origin certificate; confirm HTTPS before
|
|
setting Cloudflare to Full (strict). On the initial start, create the `admin`
|
|
account interactively:
|
|
|
|
```bash
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --create-admin
|
|
```
|
|
|
|
The account database is `$DATA_ROOT/search-users.sqlite3` and is included in
|
|
the `data` backup. Admin password reset is also interactive:
|
|
|
|
```bash
|
|
sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --reset-admin-password
|
|
```
|
|
|
|
After the initial OpenSearch index is loaded and validated, the `admin` account
|
|
is created, and `/review` works over HTTPS, create and verify the required
|
|
post-setup baseline backup before treating deployment as complete. It captures
|
|
the initialized index, account database, and Caddy certificate data. Follow
|
|
the exact stop/create/verify/start sequence in
|
|
[`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md#initial-post-setup-baseline).
|
|
|
|
## Backup, restore, and rollback
|
|
|
|
Backups are cold and require downtime. Stop services, create the archive, then
|
|
verify both its checksum and contents before proceeding. Do not run an upgrade
|
|
or schema change until verification succeeds. Keep archives root-only under
|
|
`/var/backups/akyldash` and copy them to separately managed encrypted storage.
|
|
|
|
Restore is a staged operation on a marked replacement host; it does not
|
|
overwrite production. For application-only rollback, use the guarded helper
|
|
with a verified pre-release backup and explicit data-compatibility review.
|
|
Follow the procedures in [`BACKUP.md`](../../docs/operations/BACKUP.md),
|
|
[`ROLLBACK.md`](../../docs/operations/ROLLBACK.md), and
|
|
[`RECOVERY.md`](../../docs/operations/RECOVERY.md).
|
|
|
|
Automated production deployment is not enabled. Deployments remain owner-run;
|
|
record the release and verify service health before considering it complete.
|
|
|
|
## Image release and owner-run cutover
|
|
|
|
PR CI still builds both production images with Kaniko `--no-push`. The separate
|
|
`Release production images` workflow is manual and runs only when dispatched
|
|
from protected `main`. It requires repository secrets named
|
|
`REGISTRY_USERNAME` and `REGISTRY_PASSWORD`; without them it fails before any
|
|
build. Configure those secrets only after the owner confirms a dedicated
|
|
Gitea account/token with write access to the two repository packages and no
|
|
unneeded administration rights. Do not reuse a runner or production host
|
|
credential. The workflow uses the Gitea host and fixed package namespace
|
|
`git.vakanaut.ru/admin/akyldash/{backend,opensearch}`, pushes unique
|
|
commit/run-ID tags, captures registry digests, validates the release
|
|
descriptor, and uploads the JSON plus SHA-256 sidecar as a Gitea Actions
|
|
artifact using a full-SHA-pinned Gitea-compatible v3 action. PR CI also uploads
|
|
a tiny synthetic marker with one-day retention to verify the runner's runtime
|
|
artifact path without registry credentials. The runner cache stays disabled;
|
|
artifact compatibility is based on the runtime API supported by act_runner
|
|
0.2.13/Gitea 1.25.5 and will be empirically checked by PR CI. Registry and
|
|
artifact retention must preserve release digests and descriptors for the full
|
|
rollback window; the owner must confirm both policies.
|
|
|
|
The schema-v3 descriptor records source repository/ref/commit, workflow run ID,
|
|
pinned Kaniko builder and both image digests. It is an unsigned trace record,
|
|
not a cryptographic attestation; deployment relies on retrieving it from the
|
|
authenticated Gitea run artifact and verifying its checksum and repository
|
|
allowlist. Do not treat the sidecar alone as proof of authenticity.
|
|
|
|
For a future owner-approved cutover, use the root-owned checkout and run
|
|
`infra/deploy/deploy_release.py` as root with the downloaded manifest, its
|
|
matching verified cold backup, and `infra/deploy/release-policy.json`, plus
|
|
owner-verified non-empty Russian and Kyrgyz canary queries, for example
|
|
`--canary-ru '<known Russian query>' --canary-ky '<known Kyrgyz query>'`.
|
|
The helper checks both queries against the public Search API through local
|
|
HTTPS after Compose reports healthy, and requires at least one result with a
|
|
document code before recording the new release state. Select queries only
|
|
after corpus/index canaries have been verified; do not use generic terms that
|
|
can match unrelated test data. The helper requires the backup image references and image IDs to match the live
|
|
stack, serializes deploy and rollback with `/run/akyldash-deploy.lock`,
|
|
atomically switches both protected env image refs, and always runs Compose with
|
|
`--no-build`. It records recovery state before changing references. If ref
|
|
replacement reports an error after an atomic rename, recovery rereads the
|
|
actual file and restores the prior pair before restarting; an unexpected or
|
|
unverifiable pair stops recovery and leaves a secret-free journal. A failed
|
|
cutover refuses a subsequent release until an owner resolves the journal.
|
|
Rollback accepts a digest-pinned previous release
|
|
only when that backup matches the separately recorded previous refs and image
|
|
IDs. Legacy tag rollback remains supported. Neither the helper nor registry
|
|
secrets are installed on the VPS by this PR; deployment remains an owner-only
|
|
operation and must wait for host/data/backup review. The exact sanitized
|
|
evidence required from the owner is listed in
|
|
[`DEPLOYMENT_OWNER_PACKET.md`](../../docs/operations/DEPLOYMENT_OWNER_PACKET.md).
|