Files
agent 74aa167d16
CI / tests-and-images (pull_request) Successful in 2m19s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
ci: chain release and dry-run handoff
2026-10-26 22:43:11 +03:00

15 KiB

AkylDash Gitea Actions runner

Scope and isolation

The deployment follows the official Gitea 1.25 act_runner guide for repository-scoped registration. Gitea warns that access to a Docker daemon is root-equivalent; the dedicated rootless daemon is used instead of the production daemon. The pinned binary and checksum are published in the official Gitea download directory.

The runner is reserved to repository admin/akyldash, using the token under https://git.vakanaut.ru/admin/akyldash/settings/actions/runners. Do not use organization or instance runner settings. Pin the official act_runner 0.2.13 x86_64 binary from dl.gitea.com; the installer verifies its SHA-256 before placing it on the host.

CI jobs run in docker.gitea.com/runner-images:ubuntu-24.04 containers with capacity 1, per-job limits of 2 CPU, 3 GiB RAM and 512 processes, and a 15 GiB dedicated ext4 loop-backed Docker data filesystem. The runner account's home directory is root-owned mode 0711, so CI cannot rename the image or mountpoint entries. Docker config remains under .config; registration state and the runner workspace live in .config/act-runner, which is runner-owned. The rootless Docker setup and user service use .config/act-runner as writable HOME, .config as XDG_CONFIG_HOME, and .config/act-runner/.local/share as XDG_DATA_HOME. This keeps Docker config and its systemd user unit in the existing .config tree without granting write access to the protected home. The runner systemd unit uses ProtectHome=read-only so /run/user remains visible; ReadWritePaths retains the existing /run/user/$uid writable carve-out for the runner's rootless runtime. ProtectHome=yes would hide /run/user and break the configured runtime and Docker socket paths. See the systemd.exec documentation. The rootless Docker daemon and runner user are separate from production Docker; codex is not added to sudo or Docker groups. Jobs do not receive the rootless Docker socket, the production Docker socket, production environment/secrets, or arbitrary host bind mounts. The runner account has no access to /etc/akyldash or the root-owned application checkout. These are host permission boundaries, not a claim that untrusted project code is safe; review workflow changes and actions before merging them.

The runner service restarts on failure and boot. Systemd caps the runner account's shared slice to 2.5 CPU, 4 GiB RAM and 2048 tasks; the rootless daemon is separately capped to 2 CPU, 4 GiB and 1024 tasks, while each job has the limits above. Runner logs go to a rotated file (10 MiB, 7 copies). A systemd timer checks the runner service and the rootless Docker API each minute; on failure it restarts the fixed runner service. No UFW changes or public metrics ports are needed. Runner cache is disabled.

The CI workflow and runner-smoke.yaml target the repository runner label akyldash-ci. CI keeps the backend, Telegram, backup/restore, deployment, bootstrap, runner, Compose, and Caddy checks. Caddy validation runs in a pinned container action with explicit /usr/bin/caddy entrypoint. The workflow passes Gitea's ${{ github.workspace }} value into each container action; Caddy and Kaniko use that supplied path for repository files instead of assuming a container workspace location. Both production Dockerfiles are built in a separate pinned Kaniko container action with explicit /kaniko/executor entrypoint using the runner's rootless container engine; the job container receives no Docker socket. Kaniko uses --no-push and --no-push-cache, so CI validates the full backend and OpenSearch builds but does not publish images or export build output. The two builds run sequentially within one job. CI run 69 completed both builds historically; the latest verified main CI run 175 and smoke run 176 succeeded on b5adc092. Actual per-action resource-limit enforcement and peak memory or disk usage were not independently measured or verified. No registry credentials, production service secrets, elevated container options, or arbitrary host mounts are configured in PR CI.

The Kaniko action uses the linux/amd64 v1.28.5-debug manifest sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412, listed in the official GitHub Packages entry. The pinned Caddy image digest remains the same as the former docker run validation. act_runner starts these container actions; they do not expose its rootless Docker socket to workflow steps. Production deployment is not enabled. release-images.yaml now starts only after a successful CI workflow run for a push to main; it checks the source workflow, event, branch, conclusion, and SHA and checks that the SHA is still the current main before exposing Registry credentials. It checks out that exact source SHA and uses the release workflow's own run ID for image tags and the schema-v3 descriptor. PR CI remains no-push and receives no Registry or production credentials.

Image publication still requires the existing REGISTRY_USERNAME and REGISTRY_PASSWORD Actions secrets, which are not configured or verified. When configured, they must be package-write-only for the two approved image names. The release descriptor is unsigned. The SHA-256 is included in the artifact name so a later workflow can identify the expected digest from artifact metadata without treating job output as trusted input. The existing SHA-256 sidecar remains inside the artifact. The workflow has not yet run with Registry credentials, and the sequential Kaniko build's peak resources have not been measured beyond the configured 2 CPU, 3 GiB RAM, 512-process job limits.

deploy-handoff.yaml runs after a successful release on main. It rechecks the source run and stale-SHA guard. Its current mode is explicitly dry-run: it may read JSON artifact metadata using the installed gitea-api helper when present, but it never downloads the ZIP or opens an SSH connection. If the helper is not present in the job image, it reports that artifact metadata and host handoff remain unverified. No SSH, sudo, production secret, or Docker socket is passed to the runner.

Gitea 1.25.5 documents a run-artifact metadata endpoint and a ZIP endpoint that redirects to a blob URL. The currently audited gitea-api helper supports JSON GET requests but has no binary output option and does not follow redirects, so it cannot yet fetch and independently verify the host-side artifact. Do not enable a live handoff until an owner-approved binary-safe retrieval capability validates HTTPS redirects and never forwards the API token across origins. The host must independently validate the successful main release run, artifact name and ZIP sidecar before any guarded deployment. The owner must also verify artifact and Registry retention covers the rollback window.

Installation and registration

Review the root script and host audit output before running it. It installs uidmap, dbus-user-session, curl, logrotate, and Docker's rootless extras; creates the dedicated system user and selects the first free, shared 65536-ID range from /etc/subuid and /etc/subgid; reserves 15 GiB for runner images/workspaces; and configures systemd services. It starts only the separate rootless Docker daemon; the runner and health timer stay disabled until repository registration. It does not change SSH/UFW, production Docker configuration, application data, or production secrets. It can be rerun after partial setup: it validates and reuses an existing runner account, subordinate ID mappings, image, and mount unit. An existing image must be exactly 15 GiB and contain ext4; a mismatch stops the installer without resizing or formatting it. The runner account must retain a non-root system UID below UID_MIN, its expected home, primary group, and sole supplementary group. After validating the account, the installer locks its home directory to root ownership and mode 0711; it refuses to rerun while the runner service is active, so it cannot interrupt a job. The mount is started only when it is not already active, and an active mount must be the expected ext4 loop device backed by the runner image with both nosuid and nodev active. Runner-home directories and user configuration files are created in runner-owned subdirectories. Root creates only the first .config entry beneath the already locked home; subsequent directory and config writes run as akyldash-runner. The installer rejects static symlinks and validates ownership and mode after setup. This also normalizes the 0755 intermediate directories left by the first installer version. Conflicting files or state stop the installer for manual inspection. A new image requires at least 25 GiB free on /. Image creation opens each parent component without following symlinks, creates the image exclusively relative to the opened directory, and passes the open file descriptor to mkfs.ext4; validation checks the same inode and rejects path replacement. Systemd mounts the image under the root-owned home entry, which the runner cannot replace after validation.

If the reviewed checkout is not present at the inventoried path, first verify that the target is absent and clone the reviewed main there. Do not overwrite an existing checkout.

sudo test ! -e /opt/akyldash/current
sudo git clone --branch main \
  --single-branch https://git.vakanaut.ru/admin/akyldash.git \
  /opt/akyldash/current

From that root-owned reviewed checkout on the VPS:

sudo bash /opt/akyldash/current/infra/runner/install-runner.sh

Then, as a repository owner, open repository Settings → Actions → Runners, create a repository-level runner, and copy its registration token locally. Do not send the token in chat or save it in the checkout. Register interactively in a terminal so it is not placed in shell history or process arguments:

sudo -u akyldash-runner env HOME=/var/lib/akyldash-runner/.config/act-runner \
  /usr/local/bin/act_runner \
  --config /etc/akyldash-runner/config.yaml register

Enter https://git.vakanaut.ru/, the token, runner name akyldash-ci-vps-01, and label akyldash-ci:docker://docker.gitea.com/runner-images:ubuntu-24.04 when prompted. The config sets the label and capacity; verify them in the repository runner settings. The token is saved only in the mode-0700 runner runtime directory, outside Git. Start/enable the runner after registration:

sudo systemctl enable --now akyldash-act-runner.service
sudo systemctl enable --now akyldash-runner-healthcheck.timer
sudo systemctl status akyldash-act-runner.service
sudo systemctl status akyldash-runner-healthcheck.timer

The interactive command's terminal behavior should be confirmed before pasting the token. If it echoes input, stop and use the repository settings to revoke that token; do not put it in command-line arguments or environment variables.

Validation

  1. Confirm the runner is Online in the repository runner settings, with the expected name and akyldash-ci label.
  2. Confirm Runner smoke test runs for the open PR (or dispatch it manually). It checks that production sockets, files, environment secrets, and writable application paths are not present in the job container.
  3. Confirm the run reports the expected runner, succeeds, and the runner service and health timer remain active.
  4. Inspect the CI workflow run and confirm Caddy plus both no-push Kaniko actions pass on akyldash-ci. Image publication is a distinct manual main-only workflow with package-only credentials; it does not deploy. Never add production service secrets, Docker socket access, or host mounts to this runner.

Upgrade

Update runner_version and its expected official SHA-256 together in infra/runner/install-runner.sh, review upstream release compatibility with the installed Gitea, and open a PR. Do not replace the binary while a job is active. Stop the runner, install/verify the new binary, then start it and confirm Online plus a smoke run. Preserve /var/lib/akyldash-runner/.config/act-runner/.runner.

Recovery and rollback

The runner is disposable: production data and secrets are not on its filesystem. For a runner process failure, inspect systemctl status and journalctl -u akyldash-act-runner.service; restart the fixed runner service. If installation stops after creating or enabling the mount unit, inspect the image and mount with stat, blkid, findmnt, and losetup, then rerun the reviewed installer only when the image is 15 GiB ext4 and the active mount is backed by it. The installer recognizes that state and does not format or restart the mounted filesystem. If size, filesystem, unit contents, or backing device differ, stop and resolve the state manually; never format the existing image as a retry. For lost/corrupt registration state, stop the service and obtain a fresh repository-level token from Settings → Actions → Runners. Preserve the 15 GiB data image and state for diagnosis; re-register only after the repository owner confirms/revokes the old runner in Gitea. The images/cache can be rebuilt from the workflow. Never format or remove an existing runner store as an automatic recovery step.

Status

The owner reports that the repository-scoped runner is registered, appears Idle in Gitea, and its rootless Docker daemon works; the report date was not recorded. Gitea status checked 2026-10-18 showed smoke run 62 successful and CI run 61 pending on head 69c62466. That workflow requested ubuntu-latest, while the smoke workflow used akyldash-ci; the label difference is a plausible explanation for the pending status, but the runner inventory API returned 403 and the scheduler's reason was not independently confirmed.

On head 3a89dc21, CI run 63 failed at the host audit root guard and smoke run 64 succeeded. On head 436806a8, CI run 65 failed in the Caddy container action with exec: "validate": executable file not found; Kaniko was skipped and smoke run 66 succeeded. On head d96fc4d3, CI run 67 failed because the Caddy action looked for /github/workspace/deploy/production/Caddyfile, which was not mounted at that fixed path; Kaniko was skipped and smoke run 68 succeeded. On head 5d72bb09, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile and built the backend and OpenSearch images using ${{ github.workspace }}. Actual per-action resource-limit enforcement and peak usage were not independently measured or verified. Image publishing, host-wide resource metrics, and production deployment remain unverified. Root-guard tests use setpriv for negative cases when jobs start as UID 0 and separately test the host-audit guard with simulated UID 0 and UID 65534; the real host audit, production config setup, and runner installer are never executed as root by those tests. No VPS changes are part of this PR.