Files
akyldash/docs/operations/CURRENT_STATE.md
T
agent 7701e77b3d
CI / tests-and-images (pull_request) Failing after 12s
Runner smoke test / runner-smoke (pull_request) Successful in 4s
fix: restrict runner log ACL across rotation
2026-10-14 23:47:57 +03:00

12 KiB

Infrastructure current state

Last audit: 2026-10-14. Source: read-only SSH session as codex, local repository inspection, Gitea API, and owner-reported workflow results. Root-only Docker state, UFW rule details, and protected file metadata remain unverified.

VPS facts

  • Host: avprogs.hlab.kz, 206.223.249.118, SSH port 22, user codex.
  • OS: Ubuntu Server 24.04.5 LTS; kernel 6.8.0-142-generic from the prior inventory audit.
  • Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB disk with 92 GiB free at this audit.
  • Network: interface ens3, IPv4 206.223.249.118/32.
  • codex is UID/GID 1000, groups codex and users; sudo -n true requires a password. Docker CLI cannot access /var/run/docker.sock as codex.
  • SSH service is running. The inspected sshd configuration has public-key and password authentication enabled and PermitRootLogin yes; no SSH changes were made.
  • Bootstrap completed. Docker CE/CLI 29.9.0, Compose 5.6.0, containerd 2.4.1, and runc 1.5.2 are installed; the system Docker daemon API remains inaccessible to codex.
  • /etc/docker/daemon.json has the expected local log driver and 10m / 3 rotation limits. Bootstrap paths exist with root ownership: /opt/akyldash mode 0755, /etc/akyldash and /var/backups/akyldash mode 0700, and /var/lib/akyldash/data mode 0750.
  • /opt/akyldash/current, verifier/rollback binaries, and the sudoers file are absent. codex has not received sudo or Docker-group access. Metadata for /etc/akyldash/production.env and /etc/akyldash/review_secret could not be checked because /etc/akyldash is root-only.
  • TCP listeners observed: SSH 22 and loopback DNS only; no listener on 1500, 80, or 443. No ISPmanager package or service unit was found through the unprivileged checks; root can confirm with the audit script below.
  • Owner reports UFW is active and allows TCP 22, 80, 443, and 1500. Rule details were not independently read; no firewall changes were made.
  • Read-only runner checks on 2026-10-14 confirmed akyldash-runner is UID 999, GID 988, with no supplementary groups; the runner service and health timer are active. Its systemd environment points Docker CLI to unix:///run/user/999/docker.sock; the service's ExecStartPre Docker health check succeeded. The service has ProtectHome=read-only, ProtectSystem=strict, NoNewPrivileges=yes, PrivateTmp=yes, kernel protections, and RestrictSUIDSGID=yes. Its only writable paths are runner runtime/config, logs, and /run/user/999. The temporary live ProtectHome change is not yet in main and this PR pins it in the installer.
  • systemd-analyze verify accepted the active runner unit. systemd-analyze security reports exposure 7.4 (medium) because several optional sandbox filters are unset; the live process itself runs as UID/GID 999:988, has effective and permitted capabilities 0, and NoNewPrivs=1. No supplementary groups are present.
  • The owner granted codex ACL access to /var/log/akyldash-runner/act_runner.log. Read-only SSH confirmed the directory is akyldash-runner:akyldash-runner mode 0750 with codex:r-x, while the log is root:root mode 0644 with codex:r-- and other::r--. The inspected log contains runner task metadata only, no workflow step output. The active log is therefore currently world-readable; PR #46 changes IaC to a root-owned directory, runner-owned private log, codex search/read ACLs only, and ACL reapplication after service startup and log rotation. No VPS changes were made.
  • The unprivileged check did not find /var/run/docker.sock; the runner user is not in sudo or docker groups. Production secret file metadata remains root-only and was not independently checked. The Gitea runner inventory API returns 403 to this non-owner account. The owner reports the registered runner is Idle and runner-smoke.yaml succeeds with label akyldash-ci.
  • akyldash.kg resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS, HTTP, HTTPS, and /review still return Cloudflare 521; origin TLS and Cloudflare Full (strict) are not verified.

Application and Gitea facts

  • Repository: admin/akyldash, default branch main; Gitea version 1.25.5.
  • The repository API reports push permission, Actions and packages enabled, and PRs enabled. main is protected and requires one approval; the current API principal cannot push or merge directly to main.
  • Both repository workflows are active in Gitea. runner-smoke.yaml targets akyldash-ci; ci.yaml still targets ubuntu-latest in main, which has no matching online runner. This PR changes it to the project label.
  • The Gitea runner inventory API returns 403 because this API principal is not the repository owner. The registered runner's label list cannot be read directly; the owner-reported successful smoke run confirms the custom label is accepted. Package publishing permissions remain unverified.
  • No public application listener is observed on TCP 80/443. Docker container/volume state still requires the root audit. Repository Compose describes Caddy 2.11.6 pinned by digest, Python Search API 0.11.3, SQLite account/review data, and OpenSearch 3.7.0 with analysis-icu.
  • The target publishes only Caddy on TCP 80/443. Backend and Caddy share a proxy network; OpenSearch and backend share a separate internal network, and backend has no host port. Caddy certificate storage, OpenSearch, and application data use persistent volumes/bind mounts.
  • The root-owned repository checkout, production env/secret files, Compose volumes/containers, backup, verifier/rollback helpers, and application are not confirmed installed. TLS and registry digest deployment for backend/OpenSearch are not configured.
  • The repository has a cold-backup archive/verification tool and a one-shot restore apply tool that requires a marked replacement host. Backup/restore tests use synthetic data; neither tool has run against production. The archive preflight permits a clean first deployment to skip backup, requires backup for existing persistent data even without containers, allows absent named volumes only in explicit corpus-only initial-deployment mode, and rejects inaccessible/ambiguous mounts and incomplete/running service containers. Initial deployment docs require a second verified backup after index and admin initialization.
  • Recovery bootstrap has an explicit --recovery-target mode that refuses the inventoried production hostname/IP; its host-identity guard has shell tests. The mode has not been run on a VPS.
  • The repository defines a root-owned read-only verifier, Compose config and status helper, guarded rollback helper, and installer with exact sudoers command rules. None is installed on the VPS. CI contains no production deployment job; starting/stopping services remains owner-run.
  • A repository-scoped official act_runner 0.2.13 installation is managed in infra/runner/: rootless Docker under akyldash-runner, 15 GiB ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU, 3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048 tasks), rotated file logs, and a systemd health timer. Jobs use the akyldash-ci label and do not receive the rootless or production Docker socket. The owner reports a partial runner installation: the 15 GiB ext4 loop-backed store is active on /dev/loop0. The owner reports the runner is registered and Idle, and the smoke workflow succeeds. This PR makes no VPS changes. The installer locks the runner home to root ownership (0711) so CI cannot replace the image or mountpoint, while writable config and runtime state remain in runner-owned subdirectories. It reuses legacy 0755 config directories, creates the image through a pinned file descriptor, rejects path replacement, and requires mount options nosuid,nodev. The reported installation blocker is that rootless setup receives this protected directory as HOME; the installer assigns writable runner-owned HOME/XDG paths, persists them in the docker.service user-unit drop-in, and restarts an active daemon and checks the running process environment. The runner service's temporary ProtectHome=read-only workaround is also committed to the installer with an exact known-drift migration guard.
  • Runner jobs do not receive either Docker socket (container.docker_host: "-") or production secrets. PR #46 changes Caddyfile validation to use the pinned Caddy binary directly; docker compose config --quiet remains a client-only static check. The two production image builds still require a builder. The PR audit rejects nested rootless BuildKit in the job container because it requires weaker sandbox settings, and recommends a disposable, separately isolated BuildKit VM reached through a short-lived, TLS-authenticated buildctl connection. No VPS changes were made; image builds remain unverified until the owner provisions that builder.
  • Runner log lines for run 49 show act_runner 0.2.13 with registered label [akyldash-ci] accepted task 301 at 2026-10-15 01:16:11 +05. The main CI task failed at 01:16:24 +05, and smoke task 302 started immediately after. These daemon logs do not record the failed step. Gitea 1.25.5's Actions jobs endpoint returns an empty list for completed run 49; the current API account cannot use the owner-only jobs endpoint to obtain the log download ID. The exact failing step is undetermined.
  • An isolated local Compose smoke test built backend/OpenSearch images, reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP 200 for /review through Caddy. Its containers, networks, and volumes were removed afterward. This does not verify VPS startup, DNS, or public TLS.
  • The app has no migration framework or SMTP setup. Current app versions: backend 0.11.3, frontend 0.1.2, Telegram bot 0.2.2 (the bot is not deployed on this VPS).

Checks not completed

  • Docker Engine version from the server API, container and volume inventory, protected env/secret metadata, sudoers, and detailed UFW rules require root. infra/bootstrap/audit-host.sh is provided for an owner-run read-only audit.
  • The backup archive/restore tool was tested with synthetic local data; a backup and restore of real production data remain unverified because production is not deployed.
  • Cloudflare edge response was checked; origin DNS target, origin certificate, Full (strict), direct runner-label inventory, registry publishing, and production deployment have not been verified.
  • This audit made no host changes. No package or service was installed by this audit; the owner performed the initial bootstrap beforehand.

Next owner actions

  1. Install the approved root-owned checkout after its PR is reviewed, then run sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh and share the output. The script prints no secret values and does not mutate the host.
  2. After reviewing the audit output and confirming the target files are absent, run sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh; it refuses to overwrite existing files. Set the approved DATA_ROOT and place the approved corpus/index and Cloudflare origin configuration. Do not deploy until data/service state and the required backup path are reviewed.
  3. Run sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh after reviewing the checkout; it grants only fixed verification, Compose config/status, and guarded rollback commands.
  4. After PR review and merge, install the reviewed checkout and rerun sudo bash /opt/akyldash/current/infra/runner/install-runner.sh to pin the effective ProtectHome=read-only setting. Then verify both workflows on akyldash-ci; resolve the documented Docker-build isolation gap before treating image-build CI as complete. Do not re-register the existing runner.
  5. Test backup/restore and rollback only with synthetic data in an isolated environment before any production deployment.