# Infrastructure current state Read-only host recheck: 2026-10-18. Source: SSH as `codex`, local repository inspection, and Gitea API. Root-only Docker state, UFW rule details, and protected file metadata remain unverified. ## VPS facts - Host: `avprogs.hlab.kz`, `206.223.249.118`, SSH port `22`, user `codex`. - OS: Ubuntu Server `24.04.5 LTS`; kernel `6.8.0-142-generic` from the prior inventory audit. - Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with 84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop mount with 12 GiB free at that check. - Network: interface `ens3`, IPv4 `206.223.249.118/32`. - `codex` is UID/GID 1000, groups `codex` and `users`; `sudo -n true` requires a password. Docker CLI cannot access `/var/run/docker.sock` as `codex`. - SSH service is running. The inspected sshd configuration has public-key and password authentication enabled and `PermitRootLogin yes`; no SSH changes were made. - Bootstrap completed. Docker CLI `29.9.0` and Compose `5.6.0` were rechecked over SSH on 2026-10-18; containerd `2.4.1` and runc `1.5.2` are from the prior package audit. The Docker daemon API remains root-only. - `/etc/docker/daemon.json` has the expected `local` log driver and `10m` / `3` rotation limits. Bootstrap paths exist with root ownership: `/opt/akyldash` mode `0755`, `/etc/akyldash` and `/var/backups/akyldash` mode `0700`, and `/var/lib/akyldash/data` mode `0750`. - `/opt/akyldash/current`, verifier/rollback binaries, and the sudoers file are absent. `codex` has not received sudo or Docker-group access. Metadata for `/etc/akyldash/production.env` and `/etc/akyldash/review_secret` could not be checked because access is denied. The target `/var/lib/akyldash/data/minjust-normalized` and its manifest metadata were also denied; their presence, ownership, and contents are unknown. No sudo was used. - TCP listeners observed: SSH `22` and loopback DNS only; no listener on `1500`, `80`, or `443`. No ISPmanager package or service unit was found through the unprivileged checks; root can confirm with the audit script below. - Owner reports UFW is active and allows TCP `22`, `80`, `443`, and `1500`. Rule details were not independently read; no firewall changes were made. - Additional read-only runner checks on 2026-10-12 confirmed `/` is ext4 with about 92 GiB free at the prior audit, `/usr/bin/dockerd-rootless.sh` exists, and `git.vakanaut.ru` resolves to `195.209.48.28`. From the VPS, `https://git.vakanaut.ru/api/v1/version` returned Gitea `1.25.5`; runner registration and Gitea Actions protocol connectivity are not confirmed. - `akyldash.kg` resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS, HTTPS returned Cloudflare `521` with TLS verification result `0` on 2026-10-18. TCP listeners were SSH `22` and local DNS only, so no origin listener on `80`/`443` was observed. Origin TLS and Cloudflare Full (strict) are not verified. ## Application and Gitea facts - Repository: `admin/akyldash`, default branch `main`; Gitea version `1.25.5`. - The repository API reports push permission, Actions and packages enabled, and PRs enabled. `main` is protected and requires one approval; the current API principal cannot push or merge directly to `main`. - The repository runner-list API returns `403` because this API principal is not the repository owner. The latest verified CI run 75 and smoke run 76 both succeeded on `main` head `0e276ae6`. The Actions run-list endpoint returned `403` for this API principal during this recheck, so no newer run state was independently retrieved. The owner reports the runner is registered and Idle; repository runner inventory and package publishing permissions remain unverified. - No public application listener is observed on TCP `80`/`443`. Docker container/volume state still requires the root audit. Repository Compose describes Caddy `2.11.6` pinned by digest, Python Search API `0.12.2` in this branch, SQLite account/review data, and OpenSearch `3.7.0` with `analysis-icu`. - The target publishes only Caddy on TCP `80`/`443`. Backend and Caddy share a proxy network; OpenSearch and backend share a separate internal network, and backend has no host port. Caddy certificate storage, OpenSearch, and application data use persistent volumes/bind mounts. - The root-owned repository checkout, production env/secret files, Compose volumes/containers, backup, verifier/rollback helpers, and application are not confirmed installed. TLS and registry digest deployment for backend/OpenSearch are not configured. - The repository has a cold-backup archive/verification tool and a one-shot restore apply tool that requires a marked replacement host. Backup/restore tests use synthetic data; neither tool has run against production. The archive preflight permits a clean first deployment to skip backup, requires backup for existing persistent data even without containers, allows absent named volumes only in explicit corpus-only initial-deployment mode, and rejects inaccessible/ambiguous mounts and incomplete/running service containers. Initial deployment docs require a second verified backup after index and admin initialization. - Recovery bootstrap has an explicit `--recovery-target` mode that refuses the inventoried production hostname/IP; its host-identity guard has shell tests. The mode has not been run on a VPS. - The repository defines a root-owned read-only verifier, Compose `config` and `status` helper, guarded rollback helper, and installer with exact sudoers command rules. None is installed on the VPS. CI contains no production deployment job; starting/stopping services remains owner-run. - Rollback preflight validates digest-pinned image references against archived image IDs and local availability, then refuses before stopping services or creating a backup because the current flow cannot safely switch immutable digest references. This behavior is covered by synthetic local tests; it has not been exercised on the VPS. - This branch adds an offline release-manifest validator and tests for the repository allowlist intended for a future root-owned host integration. The manifest format does not prove source provenance. No registry, host policy, publish credential, or deployment integration is configured or installed. - A repository-scoped official `act_runner` `0.2.13` installation is configured in `infra/runner/`: rootless Docker under `akyldash-runner`, 15 GiB ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU, 3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048 tasks), rotated file logs, and a systemd health timer. Jobs target the `akyldash-ci` label and do not receive either Docker socket. On 2026-10-18, SSH showed `akyldash-act-runner.service` active under `akyldash-runner`, with zero restarts and about 115 MiB current memory. Its rootless Docker store is mounted from `/dev/loop0` as ext4 with `rw,nosuid,nodev`; the 15 GiB mount had 12 GiB free. The separate `akyldash-runner-restart.service` remains failed with exit status 1 since 2026-10-14; its journal had no entries, so the cause is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on head `0e276ae6`. The Actions API returned 403 during this recheck, so no newer run state was independently retrieved. A read-only SSH recheck on 2026-10-21 confirmed the active `akyldash-act-runner.service` runs as `akyldash-runner` with `ProtectHome=read-only`; `ReadWritePaths` includes the runner runtime/config/log paths and `/run/user/999`. Its `ExecStart` uses `/etc/akyldash-runner/config.yaml`. The live unit retains a writable carveout for `/run/user/999`; the installer keeps that existing carveout and changes `ProtectHome` from `yes` to `read-only` so the runtime path remains visible. No VPS unit was changed. Main commit `d2fb4b81` passed CI run 115 and smoke run 116. On head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run 64 succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action with `exec: "validate": executable file not found`; Kaniko was skipped and smoke run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the Caddy action's fixed `/github/workspace/...` path did not exist; Kaniko was skipped and smoke run 68 succeeded. On head `5d72bb09`, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile and built the backend and OpenSearch images using `${{ github.workspace }}`. Actual per-action resource-limit enforcement and peak usage were not independently measured or verified. Image publishing, host-wide resource metrics, and production deployment remain unverified. No VPS changes are made. - An isolated local Compose smoke test built backend/OpenSearch images, reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP 200 for `/review` through Caddy. Its containers, networks, and volumes were removed afterward. This does not verify VPS startup, DNS, or public TLS. - The app has no migration framework or SMTP setup. Repository versions on this branch are: backend `0.13.0`, frontend `0.2.0`, Telegram bot `0.2.2` (the bot is not deployed on this VPS). ## Corpus readiness (2026-10-18) - The local raw archive and normalized corpus are ignored under `data/`; no corpus files were changed. Their read-only SQLite manifests both pass `quick_check` and each report 209,958 documents. The raw manifest is 28,295,168 bytes with SHA-256 `f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e`; it has one fetch error, code `6`, absent from `documents`. The normalized manifest is 31,571,968 bytes with SHA-256 `1679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf`; all entries are `success`, schema `1`, normalizer `1.0.0`. - The read-only full-file inventory in `/tmp/akyldash-corpus-audit.Mjr6aa` stopped before validation in run `b4a4556e30cc464d90bc8d75ef1673e9` with `Python int too large to convert to SQLite INTEGER`. At stop it had processed 2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856 normalized entries (246,321 files, 151,535 directories). This is a verifier metadata storage failure, not a corpus finding. A full rehash with the corrected, versioned verifier is required before readiness can be stated. - A fresh full rehash is running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`, run `f4dd5e53020f442bbbecee2d0e4986a6`, with verifier SHA-256 `83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`. `result.json` currently reports `scan_status: running`; its live counters must be recorded after completion. Sources remain read-only. - Through the repository client, bounded MinJust OpenData requests for `GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and `CheckAvailable` each returned HTTP `404`. The list API returned no code list; the direct requests do not establish whether code `6` exists. The public listing page is reachable, but it does not map the numeric API code. A metadata-only `CheckAvailable` request from the VPS failed in HTTP/2 with `curl` exit 92 (`INTERNAL_ERROR`), HTTP status `000`, and TLS verification 0; the HTTP/1.1 retry was not completed. Treat the code `6` disposition and full corpus readiness as unresolved. - On the VPS, read-only `stat` of `/var/lib/akyldash/data/minjust-normalized` and its manifest was denied. Production corpus presence and metadata remain unknown. No corpus was transferred and no production service, Docker socket, runner policy, or VPS configuration was changed. ## Current MinJust API staging (2026-10-20) - The public site's current API uses `POST /api/v1/GetDocuments` with page size 12, `GET /api/v1/GetDocument?DocumentCode=...`, and `GET /api/v1/GetEdition?editionId=...&lang=ru|kg`. A bounded page-1 probe reported 210,639 filtered records; no full listing was enumerated for this recovery. Official alias `1-4` resolves to canonical document code `205064`. - Document code `6` has 201 official editions. The API's `lang=kg` response places Kyrgyz HTML in `contentRu` while returning `contentKg=null`; staging maps by requested language (`kg` to archive `ky`) and preserves the edition order from the document response. Normalization and search export now retain and use that order instead of sorting IDs. - Without `exact=true`, one probe for edition `920292` returned response field `id=920492` despite a different text length/hash. Two bounded `exact=true` probes returned matching IDs and names/dates from `GetDocument`: edition `920292` / `04.08.1998` (485,487 RU characters, SHA-256 `2b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718`) and edition `920492` / `13.04.2017` (1,139,908 RU characters, SHA-256 `e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8`). Every edition request now uses `exact=true`, requires returned ID equality, and checks RU/KY names against `GetDocument` metadata. - `backend/ingestion/minjust_current.py` stages one document into a separate generation, at no more than one request per second, with a 16 MiB response limit, a 4 GiB text budget per document, contract-bound checkpoints, and atomic publication. Staging requires a private 0700 directory owned by the current user and rejects symlinks under checkpoints/document paths; legacy corpus exclusions are anchored to the repository root. The corpus verifier continues to validate unchanged normalized manifests from versions 1.0.0 and 1.1.0. The earlier non-exact pass is quarantined at `/tmp/akyldash-current-code6.ZAMC37u4ex/qualification.json`: its 240 language responses (304,493,520 bytes) remain unverified and are not reusable. The retained work-tree SHA-256 is `1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a`. - Exact-only code-6 staging is complete at `/tmp/akyldash-current-code6-exact.L4k3AvnOlP`: 201 editions and 402 complete language checkpoints, 604,377,885 raw bytes. The raw manifest passes `quick_check`; its document checksum matches the staged directory at `81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea`. Separate normalization at `/tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalized` contains 201 editions in official response order, passes `quick_check`, and records normalizer `1.1.0` (2,048,797,024 bytes). The prior `data/minjust-cbd` and `data/minjust-normalized` trees and production were not used as input or modified. The raw staging manifest still records attachments and embedded media as unverified; this single document does not establish corpus completeness or production readiness. ## Checks not completed - Docker Engine version from the server API, container and volume inventory, protected env/secret metadata, sudoers, and detailed UFW rules require root. `infra/bootstrap/audit-host.sh` is provided for an owner-run read-only audit. - The backup archive/restore tool was tested with synthetic local data; a backup and restore of real production data remain unverified because production is not deployed. - Cloudflare edge response was checked; origin DNS target, origin certificate, Full (strict), registry publishing, host-wide runner metrics, and production deployment have not been verified. The owner-reported Idle state has not been independently checked through runner inventory because the API returned 403. - This audit made no host changes. No package or service was installed by this audit; the owner performed the initial bootstrap beforehand. ## Next owner actions 1. Install the approved root-owned checkout after its PR is reviewed, then run `sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh` and share the output. The script prints no secret values and does not mutate the host. 2. After reviewing the audit output and confirming the target files are absent, run `sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh`; it refuses to overwrite existing files. Set the approved `DATA_ROOT` and place the approved corpus/index and Cloudflare origin configuration. Do not deploy until data/service state and the required backup path are reviewed. 3. Run `sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh` after reviewing the checkout; it grants only fixed verification, Compose `config`/`status`, and guarded rollback commands. 4. After merging the reviewed PR, confirm a subsequent CI and smoke run on `akyldash-ci`. Image publishing and production deployment remain manual and are not part of CI. 5. Test backup/restore and rollback only with synthetic data in an isolated environment before any production deployment. ## Change in this PR Adds a read-only, resumable full-file corpus inventory and normalized-data cross-check, a source/staging comparator requiring nonzero documents and fragments plus held shared artifact locks, synthetic regression tests, and an operator handoff runbook. CI's existing `infra/deploy` unittest discovery includes the new tests. No workflow, runner, VPS, production data, or Docker settings are changed. The first real corpus scan stopped on an integer-storage bug; full validation with the corrected verifier is still required. The source API's code `6` outcome remains unconfirmed.