Files
akyldash/docs/operations/CURRENT_STATE.md
T
agent 9ec03200fb
Runner smoke test / runner-smoke (pull_request) Successful in 4s
CI / tests-and-images (pull_request) Has been cancelled
fix: restart rootless Docker with updated environment
2026-10-14 17:41:24 +03:00

9.5 KiB

Infrastructure current state

Last audit: 2026-10-12. Source: read-only SSH session as codex, owner-reported UFW state, local repository inspection, and Gitea API. Root-only Docker state, UFW rule details, and protected file metadata remain unverified.

VPS facts

  • Host: avprogs.hlab.kz, 206.223.249.118, SSH port 22, user codex.
  • OS: Ubuntu Server 24.04.5 LTS; kernel 6.8.0-142-generic from the prior inventory audit.
  • Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB disk with 92 GiB free at this audit.
  • Network: interface ens3, IPv4 206.223.249.118/32.
  • codex is UID/GID 1000, groups codex and users; sudo -n true requires a password. Docker CLI cannot access /var/run/docker.sock as codex.
  • SSH service is running. The inspected sshd configuration has public-key and password authentication enabled and PermitRootLogin yes; no SSH changes were made.
  • Bootstrap completed. Docker CE/CLI 29.9.0, Compose 5.6.0, containerd 2.4.1, and runc 1.5.2 are installed; package versions and active Docker service were independently checked. The Docker daemon API remains root-only.
  • /etc/docker/daemon.json has the expected local log driver and 10m / 3 rotation limits. Bootstrap paths exist with root ownership: /opt/akyldash mode 0755, /etc/akyldash and /var/backups/akyldash mode 0700, and /var/lib/akyldash/data mode 0750.
  • /opt/akyldash/current, verifier/rollback binaries, and the sudoers file are absent. codex has not received sudo or Docker-group access. Metadata for /etc/akyldash/production.env and /etc/akyldash/review_secret could not be checked because /etc/akyldash is root-only.
  • TCP listeners observed: SSH 22 and loopback DNS only; no listener on 1500, 80, or 443. No ISPmanager package or service unit was found through the unprivileged checks; root can confirm with the audit script below.
  • Owner reports UFW is active and allows TCP 22, 80, 443, and 1500. Rule details were not independently read; no firewall changes were made.
  • Additional read-only runner checks on 2026-10-12 confirmed / is ext4 with about 92 GiB free at the prior audit, /usr/bin/dockerd-rootless.sh exists, and git.vakanaut.ru resolves to 195.209.48.28. From the VPS, https://git.vakanaut.ru/api/v1/version returned Gitea 1.25.5; runner registration and Gitea Actions protocol connectivity are not confirmed.
  • akyldash.kg resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS, HTTP, HTTPS, and /review still return Cloudflare 521; origin TLS and Cloudflare Full (strict) are not verified.

Application and Gitea facts

  • Repository: admin/akyldash, default branch main; Gitea version 1.25.5.
  • The repository API reports push permission, Actions and packages enabled, and PRs enabled. main is protected and requires one approval; the current API principal cannot push or merge directly to main.
  • The repository runner-list API returns 403 because this API principal is not the repository owner. The Actions task API currently reports no workflow runs; this does not establish runner availability. Runner inventory and package publishing permissions remain unverified.
  • No public application listener is observed on TCP 80/443. Docker container/volume state still requires the root audit. Repository Compose describes Caddy 2.11.6 pinned by digest, Python Search API 0.11.3, SQLite account/review data, and OpenSearch 3.7.0 with analysis-icu.
  • The target publishes only Caddy on TCP 80/443. Backend and Caddy share a proxy network; OpenSearch and backend share a separate internal network, and backend has no host port. Caddy certificate storage, OpenSearch, and application data use persistent volumes/bind mounts.
  • The root-owned repository checkout, production env/secret files, Compose volumes/containers, backup, verifier/rollback helpers, and application are not confirmed installed. TLS and registry digest deployment for backend/OpenSearch are not configured.
  • The repository has a cold-backup archive/verification tool and a one-shot restore apply tool that requires a marked replacement host. Backup/restore tests use synthetic data; neither tool has run against production. The archive preflight permits a clean first deployment to skip backup, requires backup for existing persistent data even without containers, allows absent named volumes only in explicit corpus-only initial-deployment mode, and rejects inaccessible/ambiguous mounts and incomplete/running service containers. Initial deployment docs require a second verified backup after index and admin initialization.
  • Recovery bootstrap has an explicit --recovery-target mode that refuses the inventoried production hostname/IP; its host-identity guard has shell tests. The mode has not been run on a VPS.
  • The repository defines a root-owned read-only verifier, Compose config and status helper, guarded rollback helper, and installer with exact sudoers command rules. None is installed on the VPS. CI contains no production deployment job; starting/stopping services remains owner-run.
  • A repository-scoped official act_runner 0.2.13 installation is prepared in infra/runner/: rootless Docker under akyldash-runner, 15 GiB ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU, 3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048 tasks), rotated file logs, and a systemd health timer. Jobs use the akyldash-ci label and do not receive the rootless or production Docker socket. The owner reports a partial runner installation: the 15 GiB ext4 loop-backed store is active on /dev/loop0 after manually enabling and starting its mount unit when the installer stopped. Remaining components, repository registration, and Online state are unverified; this PR makes no VPS changes. The reviewed installer locks the runner home to root ownership (0711) so CI cannot replace the image or mountpoint, while writable config and runtime state remain in runner-owned subdirectories. It reuses legacy 0755 config directories, creates the image through a pinned file descriptor, rejects path replacement, and requires mount options nosuid,nodev. The reported installation blocker is that rootless setup receives this protected directory as HOME; the installer assigns writable runner-owned HOME/XDG paths, persists them in the docker.service user-unit drop-in, and restarts an active daemon and checks the running process environment. Local runner guards pass; the change has not been run on the VPS.
  • Gitea repository runner inventory could not be read by the current API principal (403). The owner has not supplied runner state, and no runner Online status or runner-hosted workflow is claimed. The manual runner-smoke.yaml workflow is prepared but has not run.
  • An isolated local Compose smoke test built backend/OpenSearch images, reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP 200 for /review through Caddy. Its containers, networks, and volumes were removed afterward. This does not verify VPS startup, DNS, or public TLS.
  • The app has no migration framework or SMTP setup. Current app versions: backend 0.11.3, frontend 0.1.2, Telegram bot 0.2.2 (the bot is not deployed on this VPS).

Checks not completed

  • Docker Engine version from the server API, container and volume inventory, protected env/secret metadata, sudoers, and detailed UFW rules require root. infra/bootstrap/audit-host.sh is provided for an owner-run read-only audit.
  • The backup archive/restore tool was tested with synthetic local data; a backup and restore of real production data remain unverified because production is not deployed.
  • Cloudflare edge response was checked; origin DNS target, origin certificate, Full (strict), runner registration/Online state, registry publishing, and production deployment have not been verified.
  • This audit made no host changes. No package or service was installed by this audit; the owner performed the initial bootstrap beforehand.

Next owner actions

  1. Install the approved root-owned checkout after its PR is reviewed, then run sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh and share the output. The script prints no secret values and does not mutate the host.
  2. After reviewing the audit output and confirming the target files are absent, run sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh; it refuses to overwrite existing files. Set the approved DATA_ROOT and place the approved corpus/index and Cloudflare origin configuration. Do not deploy until data/service state and the required backup path are reviewed.
  3. Run sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh after reviewing the checkout; it grants only fixed verification, Compose config/status, and guarded rollback commands.
  4. After PR review and root audit, run sudo bash /opt/akyldash/current/infra/runner/install-runner.sh. In the repository Settings → Actions → Runners create a repository-level runner; enter the token only at the interactive registration prompt as documented in CI_RUNNER.md. Then enable the runner and health timer, confirm Online, and dispatch the smoke workflow. Production deployment remains manual and has no CI job.
  5. Test backup/restore and rollback only with synthetic data in an isolated environment before any production deployment.