257 lines
15 KiB
Markdown
257 lines
15 KiB
Markdown
# AkylDash Gitea Actions runner
|
|
|
|
## Scope and isolation
|
|
|
|
The deployment follows the official [Gitea 1.25 act_runner guide](https://docs.gitea.com/1.25/usage/actions/act-runner/)
|
|
for repository-scoped registration. Gitea warns that access to a Docker daemon
|
|
is root-equivalent; the dedicated rootless daemon is used instead of the
|
|
production daemon. The pinned binary and checksum are published in the
|
|
[official Gitea download directory](https://dl.gitea.com/gitea-runner/0.2.13/).
|
|
|
|
The runner is reserved to repository `admin/akyldash`, using the token under
|
|
`https://git.vakanaut.ru/admin/akyldash/settings/actions/runners`. Do not use
|
|
organization or instance runner settings. Pin the official `act_runner`
|
|
`0.2.13` x86_64 binary from `dl.gitea.com`; the installer verifies its SHA-256
|
|
before placing it on the host.
|
|
|
|
CI jobs run in `docker.gitea.com/runner-images:ubuntu-24.04` containers with
|
|
capacity 1, per-job limits of 2 CPU, 3 GiB RAM and 512 processes, and a 15 GiB
|
|
dedicated ext4 loop-backed Docker data filesystem. The runner account's home
|
|
directory is root-owned mode `0711`, so CI cannot rename the image or mountpoint
|
|
entries. Docker config remains under `.config`; registration state and the
|
|
runner workspace live in `.config/act-runner`, which is runner-owned. The
|
|
rootless Docker setup and user service use `.config/act-runner` as writable
|
|
`HOME`, `.config` as `XDG_CONFIG_HOME`, and
|
|
`.config/act-runner/.local/share` as `XDG_DATA_HOME`. This keeps Docker config
|
|
and its systemd user unit in the existing `.config` tree without granting
|
|
write access to the protected home.
|
|
The runner systemd unit uses `ProtectHome=read-only` so `/run/user` remains
|
|
visible; `ReadWritePaths` retains the existing `/run/user/$uid` writable
|
|
carve-out for the runner's rootless runtime. `ProtectHome=yes` would hide
|
|
`/run/user` and break the configured runtime and Docker socket paths. See the
|
|
[systemd.exec documentation](https://www.freedesktop.org/software/systemd/man/255/systemd.exec.html#ProtectHome=).
|
|
The rootless Docker daemon and runner user are separate from production Docker;
|
|
`codex` is not added to
|
|
sudo or Docker groups. Jobs do not receive the rootless Docker socket, the
|
|
production Docker socket, production environment/secrets, or arbitrary host
|
|
bind mounts. The runner account has no access to `/etc/akyldash` or the
|
|
root-owned application checkout. These are host permission boundaries, not a
|
|
claim that untrusted project code is safe; review workflow changes and actions
|
|
before merging them.
|
|
|
|
The runner service restarts on failure and boot. Systemd caps the runner
|
|
account's shared slice to 2.5 CPU, 4 GiB RAM and 2048 tasks; the rootless
|
|
daemon is separately capped to 2 CPU, 4 GiB and 1024 tasks, while each job has
|
|
the limits above. Runner logs go to a rotated file (10 MiB, 7 copies). A
|
|
systemd timer checks the runner service and the rootless Docker API each minute;
|
|
on failure it restarts the fixed runner service. No UFW changes or public
|
|
metrics ports are needed. Runner cache is disabled.
|
|
|
|
The CI workflow and `runner-smoke.yaml` target the repository runner label
|
|
`akyldash-ci`. CI keeps the backend, Telegram, backup/restore, deployment,
|
|
bootstrap, runner, Compose, and Caddy checks. Caddy validation runs in a pinned
|
|
container action with explicit `/usr/bin/caddy` entrypoint. The workflow passes
|
|
Gitea's `${{ github.workspace }}` value into each container action; Caddy and
|
|
Kaniko use that supplied path for repository files instead of assuming a
|
|
container workspace location. Both production
|
|
Dockerfiles are built in a separate pinned Kaniko container action with explicit
|
|
`/kaniko/executor` entrypoint using the runner's rootless container engine; the
|
|
job container receives no Docker socket. Kaniko uses `--no-push` and
|
|
`--no-push-cache`, so CI validates the full backend and OpenSearch builds but
|
|
does not publish images or export build output. The two builds run sequentially
|
|
within one job. CI run 69 completed both builds historically; the latest
|
|
verified main CI run 175 and smoke run 176 succeeded on `b5adc092`. Actual per-action
|
|
resource-limit enforcement and peak memory or disk usage were not independently
|
|
measured or verified. No registry credentials, production service secrets,
|
|
elevated container options, or arbitrary host mounts are configured in PR CI.
|
|
|
|
The Kaniko action uses the `linux/amd64` v1.28.5-debug manifest
|
|
`sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412`,
|
|
listed in the [official GitHub Packages entry](https://github.com/osscontainertools/kaniko/pkgs/container/kaniko/1271366984?tag=v1.28.5-debug).
|
|
The pinned Caddy image digest remains the same as the former `docker run`
|
|
validation. act_runner starts these container actions; they do not expose its
|
|
rootless Docker socket to workflow steps.
|
|
Production deployment is not enabled. `release-images.yaml` now starts only
|
|
after a successful `CI` workflow run for a push to `main`; it checks the source
|
|
workflow, event, branch, conclusion, and SHA and checks that the SHA is still
|
|
the current `main` before exposing Registry credentials. It checks out that
|
|
exact source SHA and uses the release workflow's own run ID for image tags and
|
|
the schema-v3 descriptor. PR CI remains no-push and receives no Registry or
|
|
production credentials.
|
|
|
|
Image publication still requires the existing `REGISTRY_USERNAME` and
|
|
`REGISTRY_PASSWORD` Actions secrets, which are not configured or verified.
|
|
When configured, they must be package-write-only for the two approved image
|
|
names. The release descriptor is unsigned. The SHA-256 is included in the
|
|
artifact name so a later workflow can identify the expected digest from
|
|
artifact metadata without treating job output as trusted input. The existing
|
|
SHA-256 sidecar remains inside the artifact. The workflow has not yet run with
|
|
Registry credentials, and the sequential Kaniko build's peak resources have not
|
|
been measured beyond the configured 2 CPU, 3 GiB RAM, 512-process job limits.
|
|
|
|
`deploy-handoff.yaml` runs after a successful release on `main`. It rechecks the
|
|
source run and stale-SHA guard. Its current mode is explicitly dry-run: it may
|
|
read JSON artifact metadata using the installed `gitea-api` helper when present,
|
|
but it never downloads the ZIP or opens an SSH connection. If the helper is not
|
|
present in the job image, it reports that artifact metadata and host handoff
|
|
remain unverified. No SSH, sudo, production secret, or Docker socket is passed
|
|
to the runner.
|
|
|
|
Gitea 1.25.5 documents a run-artifact metadata endpoint and a ZIP endpoint that
|
|
redirects to a blob URL. The currently audited `gitea-api` helper supports JSON
|
|
GET requests but has no binary output option and does not follow redirects, so
|
|
it cannot yet fetch and independently verify the host-side artifact. Do not
|
|
enable a live handoff until an owner-approved binary-safe retrieval capability
|
|
validates HTTPS redirects and never forwards the API token across origins. The
|
|
host must independently validate the successful `main` release run, artifact
|
|
name and ZIP sidecar before any guarded deployment. The owner must also verify
|
|
artifact and Registry retention covers the rollback window.
|
|
|
|
## Installation and registration
|
|
|
|
Review the root script and host audit output before running it. It installs
|
|
`uidmap`, `dbus-user-session`, `curl`, `logrotate`, and Docker's rootless extras;
|
|
creates the dedicated system user and selects the first free, shared 65536-ID
|
|
range from `/etc/subuid` and `/etc/subgid`; reserves 15 GiB
|
|
for runner images/workspaces; and configures systemd services. It starts only
|
|
the separate rootless Docker daemon; the runner and health timer stay disabled
|
|
until repository registration. It does not change
|
|
SSH/UFW, production Docker configuration, application data, or production
|
|
secrets. It can be rerun after partial setup: it validates and reuses an
|
|
existing runner account, subordinate ID mappings, image, and mount unit. An
|
|
existing image must be exactly 15 GiB and contain ext4; a mismatch stops the
|
|
installer without resizing or formatting it. The runner account must retain
|
|
a non-root system UID below `UID_MIN`, its expected home, primary group, and
|
|
sole supplementary group. After validating the account, the installer locks
|
|
its home directory to root ownership and mode `0711`; it refuses to rerun while
|
|
the runner service is active, so it cannot interrupt a job. The mount
|
|
is started only when it is not already active, and an active mount must be the
|
|
expected ext4 loop device backed by the runner image with both `nosuid` and
|
|
`nodev` active. Runner-home directories and user configuration files are
|
|
created in runner-owned subdirectories. Root creates only the first `.config`
|
|
entry beneath the already locked home; subsequent directory and config writes
|
|
run as `akyldash-runner`. The installer rejects static symlinks and validates
|
|
ownership and mode after setup. This also normalizes the `0755` intermediate
|
|
directories left by the first installer version. Conflicting files or state
|
|
stop the installer for manual inspection. A new image requires at least 25 GiB
|
|
free on `/`. Image creation opens each parent component without following
|
|
symlinks, creates the image exclusively relative to the opened directory, and
|
|
passes the open file descriptor to `mkfs.ext4`; validation checks the same inode
|
|
and rejects path replacement. Systemd mounts the image under the root-owned home
|
|
entry, which the runner cannot replace after validation.
|
|
|
|
If the reviewed checkout is not present at the inventoried path, first verify
|
|
that the target is absent and clone the reviewed `main` there. Do not overwrite
|
|
an existing checkout.
|
|
|
|
```bash
|
|
sudo test ! -e /opt/akyldash/current
|
|
sudo git clone --branch main \
|
|
--single-branch https://git.vakanaut.ru/admin/akyldash.git \
|
|
/opt/akyldash/current
|
|
```
|
|
|
|
From that root-owned reviewed checkout on the VPS:
|
|
|
|
```bash
|
|
sudo bash /opt/akyldash/current/infra/runner/install-runner.sh
|
|
```
|
|
|
|
Then, as a repository owner, open repository **Settings → Actions → Runners**,
|
|
create a repository-level runner, and copy its registration token locally. Do
|
|
not send the token in chat or save it in the checkout. Register interactively
|
|
in a terminal so it is not placed in shell history or process arguments:
|
|
|
|
```bash
|
|
sudo -u akyldash-runner env HOME=/var/lib/akyldash-runner/.config/act-runner \
|
|
/usr/local/bin/act_runner \
|
|
--config /etc/akyldash-runner/config.yaml register
|
|
```
|
|
|
|
Enter `https://git.vakanaut.ru/`, the token, runner name `akyldash-ci-vps-01`,
|
|
and label `akyldash-ci:docker://docker.gitea.com/runner-images:ubuntu-24.04`
|
|
when prompted. The config sets the label and capacity; verify them in the
|
|
repository runner settings. The token is saved only in the mode-0700 runner
|
|
runtime directory, outside Git. Start/enable the runner after registration:
|
|
|
|
```bash
|
|
sudo systemctl enable --now akyldash-act-runner.service
|
|
sudo systemctl enable --now akyldash-runner-healthcheck.timer
|
|
sudo systemctl status akyldash-act-runner.service
|
|
sudo systemctl status akyldash-runner-healthcheck.timer
|
|
```
|
|
|
|
The interactive command's terminal behavior should be confirmed before pasting
|
|
the token. If it echoes input, stop and use the repository settings to revoke
|
|
that token; do not put it in command-line arguments or environment variables.
|
|
|
|
## Validation
|
|
|
|
1. Confirm the runner is **Online** in the repository runner settings, with the
|
|
expected name and `akyldash-ci` label.
|
|
2. Confirm **Runner smoke test** runs for the open PR (or dispatch it manually).
|
|
It checks that production
|
|
sockets, files, environment secrets, and writable application paths are not
|
|
present in the job container.
|
|
3. Confirm the run reports the expected runner, succeeds, and the runner
|
|
service and health timer remain active.
|
|
4. Inspect the `CI` workflow run and confirm Caddy plus both no-push Kaniko
|
|
actions pass on `akyldash-ci`. Image publication is a distinct manual
|
|
main-only workflow with package-only credentials; it does not deploy.
|
|
Never add production service secrets, Docker socket access, or host mounts
|
|
to this runner.
|
|
|
|
## Upgrade
|
|
|
|
Update `runner_version` and its expected official SHA-256 together in
|
|
`infra/runner/install-runner.sh`, review upstream release compatibility with
|
|
the installed Gitea, and open a PR. Do not replace the binary while a job is
|
|
active. Stop the runner, install/verify the new binary, then start it and
|
|
confirm Online plus a smoke run. Preserve
|
|
`/var/lib/akyldash-runner/.config/act-runner/.runner`.
|
|
|
|
## Recovery and rollback
|
|
|
|
The runner is disposable: production data and secrets are not on its filesystem.
|
|
For a runner process failure, inspect `systemctl status` and
|
|
`journalctl -u akyldash-act-runner.service`; restart the fixed runner service.
|
|
If installation stops after creating or enabling the mount unit, inspect the
|
|
image and mount with `stat`, `blkid`, `findmnt`, and `losetup`, then rerun the
|
|
reviewed installer only when the image is 15 GiB ext4 and the active mount is
|
|
backed by it. The installer recognizes that state and does not format or
|
|
restart the mounted filesystem. If size, filesystem, unit contents, or backing
|
|
device differ, stop and resolve the state manually; never format the existing
|
|
image as a retry.
|
|
For lost/corrupt registration state, stop the service and obtain a fresh
|
|
repository-level token from Settings → Actions → Runners. Preserve the 15 GiB
|
|
data image and state for diagnosis; re-register only after the repository
|
|
owner confirms/revokes the old runner in Gitea. The images/cache can be rebuilt
|
|
from the workflow. Never format or remove an existing runner store as an
|
|
automatic recovery step.
|
|
|
|
## Status
|
|
|
|
The owner reports that the repository-scoped runner is registered, appears
|
|
Idle in Gitea, and its rootless Docker daemon works; the report date was not
|
|
recorded. Gitea status checked 2026-10-18 showed smoke run 62 successful and CI
|
|
run 61 pending on head `69c62466`. That workflow requested `ubuntu-latest`,
|
|
while the smoke workflow used `akyldash-ci`; the label difference is a
|
|
plausible explanation for the pending status, but the runner inventory API
|
|
returned 403 and the scheduler's reason was not independently confirmed.
|
|
|
|
On head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run
|
|
64 succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
|
|
with `exec: "validate": executable file not found`; Kaniko was skipped and smoke
|
|
run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the Caddy action
|
|
looked for `/github/workspace/deploy/production/Caddyfile`, which was not
|
|
mounted at that fixed path; Kaniko was skipped and smoke run 68 succeeded. On
|
|
head `5d72bb09`, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile
|
|
and built the backend and OpenSearch images using `${{ github.workspace }}`.
|
|
Actual per-action resource-limit enforcement and peak usage were not
|
|
independently measured or verified. Image publishing, host-wide resource
|
|
metrics, and production deployment remain unverified. Root-guard tests
|
|
use `setpriv` for negative cases when jobs start as UID 0 and separately test
|
|
the host-audit guard with simulated UID 0 and UID 65534; the real host audit,
|
|
production config setup, and runner installer are never executed as root by
|
|
those tests. No VPS changes are part of this PR.
|