Files
akyldash/docs/operations/CI_RUNNER.md
agent 74aa167d16
CI / tests-and-images (pull_request) Successful in 2m19s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
ci: chain release and dry-run handoff
2026-10-26 22:43:11 +03:00

257 lines
15 KiB
Markdown

# AkylDash Gitea Actions runner
## Scope and isolation
The deployment follows the official [Gitea 1.25 act_runner guide](https://docs.gitea.com/1.25/usage/actions/act-runner/)
for repository-scoped registration. Gitea warns that access to a Docker daemon
is root-equivalent; the dedicated rootless daemon is used instead of the
production daemon. The pinned binary and checksum are published in the
[official Gitea download directory](https://dl.gitea.com/gitea-runner/0.2.13/).
The runner is reserved to repository `admin/akyldash`, using the token under
`https://git.vakanaut.ru/admin/akyldash/settings/actions/runners`. Do not use
organization or instance runner settings. Pin the official `act_runner`
`0.2.13` x86_64 binary from `dl.gitea.com`; the installer verifies its SHA-256
before placing it on the host.
CI jobs run in `docker.gitea.com/runner-images:ubuntu-24.04` containers with
capacity 1, per-job limits of 2 CPU, 3 GiB RAM and 512 processes, and a 15 GiB
dedicated ext4 loop-backed Docker data filesystem. The runner account's home
directory is root-owned mode `0711`, so CI cannot rename the image or mountpoint
entries. Docker config remains under `.config`; registration state and the
runner workspace live in `.config/act-runner`, which is runner-owned. The
rootless Docker setup and user service use `.config/act-runner` as writable
`HOME`, `.config` as `XDG_CONFIG_HOME`, and
`.config/act-runner/.local/share` as `XDG_DATA_HOME`. This keeps Docker config
and its systemd user unit in the existing `.config` tree without granting
write access to the protected home.
The runner systemd unit uses `ProtectHome=read-only` so `/run/user` remains
visible; `ReadWritePaths` retains the existing `/run/user/$uid` writable
carve-out for the runner's rootless runtime. `ProtectHome=yes` would hide
`/run/user` and break the configured runtime and Docker socket paths. See the
[systemd.exec documentation](https://www.freedesktop.org/software/systemd/man/255/systemd.exec.html#ProtectHome=).
The rootless Docker daemon and runner user are separate from production Docker;
`codex` is not added to
sudo or Docker groups. Jobs do not receive the rootless Docker socket, the
production Docker socket, production environment/secrets, or arbitrary host
bind mounts. The runner account has no access to `/etc/akyldash` or the
root-owned application checkout. These are host permission boundaries, not a
claim that untrusted project code is safe; review workflow changes and actions
before merging them.
The runner service restarts on failure and boot. Systemd caps the runner
account's shared slice to 2.5 CPU, 4 GiB RAM and 2048 tasks; the rootless
daemon is separately capped to 2 CPU, 4 GiB and 1024 tasks, while each job has
the limits above. Runner logs go to a rotated file (10 MiB, 7 copies). A
systemd timer checks the runner service and the rootless Docker API each minute;
on failure it restarts the fixed runner service. No UFW changes or public
metrics ports are needed. Runner cache is disabled.
The CI workflow and `runner-smoke.yaml` target the repository runner label
`akyldash-ci`. CI keeps the backend, Telegram, backup/restore, deployment,
bootstrap, runner, Compose, and Caddy checks. Caddy validation runs in a pinned
container action with explicit `/usr/bin/caddy` entrypoint. The workflow passes
Gitea's `${{ github.workspace }}` value into each container action; Caddy and
Kaniko use that supplied path for repository files instead of assuming a
container workspace location. Both production
Dockerfiles are built in a separate pinned Kaniko container action with explicit
`/kaniko/executor` entrypoint using the runner's rootless container engine; the
job container receives no Docker socket. Kaniko uses `--no-push` and
`--no-push-cache`, so CI validates the full backend and OpenSearch builds but
does not publish images or export build output. The two builds run sequentially
within one job. CI run 69 completed both builds historically; the latest
verified main CI run 175 and smoke run 176 succeeded on `b5adc092`. Actual per-action
resource-limit enforcement and peak memory or disk usage were not independently
measured or verified. No registry credentials, production service secrets,
elevated container options, or arbitrary host mounts are configured in PR CI.
The Kaniko action uses the `linux/amd64` v1.28.5-debug manifest
`sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412`,
listed in the [official GitHub Packages entry](https://github.com/osscontainertools/kaniko/pkgs/container/kaniko/1271366984?tag=v1.28.5-debug).
The pinned Caddy image digest remains the same as the former `docker run`
validation. act_runner starts these container actions; they do not expose its
rootless Docker socket to workflow steps.
Production deployment is not enabled. `release-images.yaml` now starts only
after a successful `CI` workflow run for a push to `main`; it checks the source
workflow, event, branch, conclusion, and SHA and checks that the SHA is still
the current `main` before exposing Registry credentials. It checks out that
exact source SHA and uses the release workflow's own run ID for image tags and
the schema-v3 descriptor. PR CI remains no-push and receives no Registry or
production credentials.
Image publication still requires the existing `REGISTRY_USERNAME` and
`REGISTRY_PASSWORD` Actions secrets, which are not configured or verified.
When configured, they must be package-write-only for the two approved image
names. The release descriptor is unsigned. The SHA-256 is included in the
artifact name so a later workflow can identify the expected digest from
artifact metadata without treating job output as trusted input. The existing
SHA-256 sidecar remains inside the artifact. The workflow has not yet run with
Registry credentials, and the sequential Kaniko build's peak resources have not
been measured beyond the configured 2 CPU, 3 GiB RAM, 512-process job limits.
`deploy-handoff.yaml` runs after a successful release on `main`. It rechecks the
source run and stale-SHA guard. Its current mode is explicitly dry-run: it may
read JSON artifact metadata using the installed `gitea-api` helper when present,
but it never downloads the ZIP or opens an SSH connection. If the helper is not
present in the job image, it reports that artifact metadata and host handoff
remain unverified. No SSH, sudo, production secret, or Docker socket is passed
to the runner.
Gitea 1.25.5 documents a run-artifact metadata endpoint and a ZIP endpoint that
redirects to a blob URL. The currently audited `gitea-api` helper supports JSON
GET requests but has no binary output option and does not follow redirects, so
it cannot yet fetch and independently verify the host-side artifact. Do not
enable a live handoff until an owner-approved binary-safe retrieval capability
validates HTTPS redirects and never forwards the API token across origins. The
host must independently validate the successful `main` release run, artifact
name and ZIP sidecar before any guarded deployment. The owner must also verify
artifact and Registry retention covers the rollback window.
## Installation and registration
Review the root script and host audit output before running it. It installs
`uidmap`, `dbus-user-session`, `curl`, `logrotate`, and Docker's rootless extras;
creates the dedicated system user and selects the first free, shared 65536-ID
range from `/etc/subuid` and `/etc/subgid`; reserves 15 GiB
for runner images/workspaces; and configures systemd services. It starts only
the separate rootless Docker daemon; the runner and health timer stay disabled
until repository registration. It does not change
SSH/UFW, production Docker configuration, application data, or production
secrets. It can be rerun after partial setup: it validates and reuses an
existing runner account, subordinate ID mappings, image, and mount unit. An
existing image must be exactly 15 GiB and contain ext4; a mismatch stops the
installer without resizing or formatting it. The runner account must retain
a non-root system UID below `UID_MIN`, its expected home, primary group, and
sole supplementary group. After validating the account, the installer locks
its home directory to root ownership and mode `0711`; it refuses to rerun while
the runner service is active, so it cannot interrupt a job. The mount
is started only when it is not already active, and an active mount must be the
expected ext4 loop device backed by the runner image with both `nosuid` and
`nodev` active. Runner-home directories and user configuration files are
created in runner-owned subdirectories. Root creates only the first `.config`
entry beneath the already locked home; subsequent directory and config writes
run as `akyldash-runner`. The installer rejects static symlinks and validates
ownership and mode after setup. This also normalizes the `0755` intermediate
directories left by the first installer version. Conflicting files or state
stop the installer for manual inspection. A new image requires at least 25 GiB
free on `/`. Image creation opens each parent component without following
symlinks, creates the image exclusively relative to the opened directory, and
passes the open file descriptor to `mkfs.ext4`; validation checks the same inode
and rejects path replacement. Systemd mounts the image under the root-owned home
entry, which the runner cannot replace after validation.
If the reviewed checkout is not present at the inventoried path, first verify
that the target is absent and clone the reviewed `main` there. Do not overwrite
an existing checkout.
```bash
sudo test ! -e /opt/akyldash/current
sudo git clone --branch main \
--single-branch https://git.vakanaut.ru/admin/akyldash.git \
/opt/akyldash/current
```
From that root-owned reviewed checkout on the VPS:
```bash
sudo bash /opt/akyldash/current/infra/runner/install-runner.sh
```
Then, as a repository owner, open repository **Settings → Actions → Runners**,
create a repository-level runner, and copy its registration token locally. Do
not send the token in chat or save it in the checkout. Register interactively
in a terminal so it is not placed in shell history or process arguments:
```bash
sudo -u akyldash-runner env HOME=/var/lib/akyldash-runner/.config/act-runner \
/usr/local/bin/act_runner \
--config /etc/akyldash-runner/config.yaml register
```
Enter `https://git.vakanaut.ru/`, the token, runner name `akyldash-ci-vps-01`,
and label `akyldash-ci:docker://docker.gitea.com/runner-images:ubuntu-24.04`
when prompted. The config sets the label and capacity; verify them in the
repository runner settings. The token is saved only in the mode-0700 runner
runtime directory, outside Git. Start/enable the runner after registration:
```bash
sudo systemctl enable --now akyldash-act-runner.service
sudo systemctl enable --now akyldash-runner-healthcheck.timer
sudo systemctl status akyldash-act-runner.service
sudo systemctl status akyldash-runner-healthcheck.timer
```
The interactive command's terminal behavior should be confirmed before pasting
the token. If it echoes input, stop and use the repository settings to revoke
that token; do not put it in command-line arguments or environment variables.
## Validation
1. Confirm the runner is **Online** in the repository runner settings, with the
expected name and `akyldash-ci` label.
2. Confirm **Runner smoke test** runs for the open PR (or dispatch it manually).
It checks that production
sockets, files, environment secrets, and writable application paths are not
present in the job container.
3. Confirm the run reports the expected runner, succeeds, and the runner
service and health timer remain active.
4. Inspect the `CI` workflow run and confirm Caddy plus both no-push Kaniko
actions pass on `akyldash-ci`. Image publication is a distinct manual
main-only workflow with package-only credentials; it does not deploy.
Never add production service secrets, Docker socket access, or host mounts
to this runner.
## Upgrade
Update `runner_version` and its expected official SHA-256 together in
`infra/runner/install-runner.sh`, review upstream release compatibility with
the installed Gitea, and open a PR. Do not replace the binary while a job is
active. Stop the runner, install/verify the new binary, then start it and
confirm Online plus a smoke run. Preserve
`/var/lib/akyldash-runner/.config/act-runner/.runner`.
## Recovery and rollback
The runner is disposable: production data and secrets are not on its filesystem.
For a runner process failure, inspect `systemctl status` and
`journalctl -u akyldash-act-runner.service`; restart the fixed runner service.
If installation stops after creating or enabling the mount unit, inspect the
image and mount with `stat`, `blkid`, `findmnt`, and `losetup`, then rerun the
reviewed installer only when the image is 15 GiB ext4 and the active mount is
backed by it. The installer recognizes that state and does not format or
restart the mounted filesystem. If size, filesystem, unit contents, or backing
device differ, stop and resolve the state manually; never format the existing
image as a retry.
For lost/corrupt registration state, stop the service and obtain a fresh
repository-level token from Settings → Actions → Runners. Preserve the 15 GiB
data image and state for diagnosis; re-register only after the repository
owner confirms/revokes the old runner in Gitea. The images/cache can be rebuilt
from the workflow. Never format or remove an existing runner store as an
automatic recovery step.
## Status
The owner reports that the repository-scoped runner is registered, appears
Idle in Gitea, and its rootless Docker daemon works; the report date was not
recorded. Gitea status checked 2026-10-18 showed smoke run 62 successful and CI
run 61 pending on head `69c62466`. That workflow requested `ubuntu-latest`,
while the smoke workflow used `akyldash-ci`; the label difference is a
plausible explanation for the pending status, but the runner inventory API
returned 403 and the scheduler's reason was not independently confirmed.
On head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run
64 succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
with `exec: "validate": executable file not found`; Kaniko was skipped and smoke
run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the Caddy action
looked for `/github/workspace/deploy/production/Caddyfile`, which was not
mounted at that fixed path; Kaniko was skipped and smoke run 68 succeeded. On
head `5d72bb09`, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile
and built the backend and OpenSearch images using `${{ github.workspace }}`.
Actual per-action resource-limit enforcement and peak usage were not
independently measured or verified. Image publishing, host-wide resource
metrics, and production deployment remain unverified. Root-guard tests
use `setpriv` for negative cases when jobs start as UID 0 and separately test
the host-audit guard with simulated UID 0 and UID 65534; the real host audit,
production config setup, and runner installer are never executed as root by
those tests. No VPS changes are part of this PR.