Files
akyldash/docs/operations/CURRENT_STATE.md
agent d01fdabd68
Runner smoke test / runner-smoke (pull_request) Failing after 13m50s
CI / tests-and-images (pull_request) Failing after 13m55s
feat(frontend): implement legal search interface
2026-10-23 00:00:41 +03:00

17 KiB

Infrastructure current state

Read-only host recheck: 2026-10-18. Source: SSH as codex, local repository inspection, and Gitea API. Root-only Docker state, UFW rule details, and protected file metadata remain unverified.

VPS facts

  • Host: avprogs.hlab.kz, 206.223.249.118, SSH port 22, user codex.
  • OS: Ubuntu Server 24.04.5 LTS; kernel 6.8.0-142-generic from the prior inventory audit.
  • Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with 84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop mount with 12 GiB free at that check.
  • Network: interface ens3, IPv4 206.223.249.118/32.
  • codex is UID/GID 1000, groups codex and users; sudo -n true requires a password. Docker CLI cannot access /var/run/docker.sock as codex.
  • SSH service is running. The inspected sshd configuration has public-key and password authentication enabled and PermitRootLogin yes; no SSH changes were made.
  • Bootstrap completed. Docker CLI 29.9.0 and Compose 5.6.0 were rechecked over SSH on 2026-10-18; containerd 2.4.1 and runc 1.5.2 are from the prior package audit. The Docker daemon API remains root-only.
  • /etc/docker/daemon.json has the expected local log driver and 10m / 3 rotation limits. Bootstrap paths exist with root ownership: /opt/akyldash mode 0755, /etc/akyldash and /var/backups/akyldash mode 0700, and /var/lib/akyldash/data mode 0750.
  • /opt/akyldash/current, verifier/rollback binaries, and the sudoers file are absent. codex has not received sudo or Docker-group access. Metadata for /etc/akyldash/production.env and /etc/akyldash/review_secret could not be checked because access is denied. The target /var/lib/akyldash/data/minjust-normalized and its manifest metadata were also denied; their presence, ownership, and contents are unknown. No sudo was used.
  • TCP listeners observed: SSH 22 and loopback DNS only; no listener on 1500, 80, or 443. No ISPmanager package or service unit was found through the unprivileged checks; root can confirm with the audit script below.
  • Owner reports UFW is active and allows TCP 22, 80, 443, and 1500. Rule details were not independently read; no firewall changes were made.
  • Additional read-only runner checks on 2026-10-12 confirmed / is ext4 with about 92 GiB free at the prior audit, /usr/bin/dockerd-rootless.sh exists, and git.vakanaut.ru resolves to 195.209.48.28. From the VPS, https://git.vakanaut.ru/api/v1/version returned Gitea 1.25.5; runner registration and Gitea Actions protocol connectivity are not confirmed.
  • akyldash.kg resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS, HTTPS returned Cloudflare 521 with TLS verification result 0 on 2026-10-18. TCP listeners were SSH 22 and local DNS only, so no origin listener on 80/443 was observed. Origin TLS and Cloudflare Full (strict) are not verified.

Application and Gitea facts

  • Repository: admin/akyldash, default branch main; Gitea version 1.25.5.
  • The repository API reports push permission, Actions and packages enabled, and PRs enabled. main is protected and requires one approval; the current API principal cannot push or merge directly to main.
  • The repository runner-list API returns 403 because this API principal is not the repository owner. The latest verified CI run 75 and smoke run 76 both succeeded on main head 0e276ae6. The Actions run-list endpoint returned 403 for this API principal during this recheck, so no newer run state was independently retrieved. The owner reports the runner is registered and Idle; repository runner inventory and package publishing permissions remain unverified.
  • No public application listener is observed on TCP 80/443. Docker container/volume state still requires the root audit. Repository Compose describes Caddy 2.11.6 pinned by digest, Python Search API 0.12.2 in this branch, SQLite account/review data, and OpenSearch 3.7.0 with analysis-icu.
  • The target publishes only Caddy on TCP 80/443. Backend and Caddy share a proxy network; OpenSearch and backend share a separate internal network, and backend has no host port. Caddy certificate storage, OpenSearch, and application data use persistent volumes/bind mounts.
  • The root-owned repository checkout, production env/secret files, Compose volumes/containers, backup, verifier/rollback helpers, and application are not confirmed installed. TLS and registry digest deployment for backend/OpenSearch are not configured.
  • The repository has a cold-backup archive/verification tool and a one-shot restore apply tool that requires a marked replacement host. Backup/restore tests use synthetic data; neither tool has run against production. The archive preflight permits a clean first deployment to skip backup, requires backup for existing persistent data even without containers, allows absent named volumes only in explicit corpus-only initial-deployment mode, and rejects inaccessible/ambiguous mounts and incomplete/running service containers. Initial deployment docs require a second verified backup after index and admin initialization.
  • Recovery bootstrap has an explicit --recovery-target mode that refuses the inventoried production hostname/IP; its host-identity guard has shell tests. The mode has not been run on a VPS.
  • The repository defines a root-owned read-only verifier, Compose config and status helper, guarded rollback helper, and installer with exact sudoers command rules. None is installed on the VPS. CI contains no production deployment job; starting/stopping services remains owner-run.
  • Rollback preflight validates digest-pinned image references against archived image IDs and local availability, then refuses before stopping services or creating a backup because the current flow cannot safely switch immutable digest references. This behavior is covered by synthetic local tests; it has not been exercised on the VPS.
  • This branch adds an offline release-manifest validator and tests for the repository allowlist intended for a future root-owned host integration. The manifest format does not prove source provenance. No registry, host policy, publish credential, or deployment integration is configured or installed.
  • A repository-scoped official act_runner 0.2.13 installation is configured in infra/runner/: rootless Docker under akyldash-runner, 15 GiB ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU, 3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048 tasks), rotated file logs, and a systemd health timer. Jobs target the akyldash-ci label and do not receive either Docker socket. On 2026-10-18, SSH showed akyldash-act-runner.service active under akyldash-runner, with zero restarts and about 115 MiB current memory. Its rootless Docker store is mounted from /dev/loop0 as ext4 with rw,nosuid,nodev; the 15 GiB mount had 12 GiB free. The separate akyldash-runner-restart.service remains failed with exit status 1 since 2026-10-14; its journal had no entries, so the cause is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on head 0e276ae6. The Actions API returned 403 during this recheck, so no newer run state was independently retrieved. A read-only SSH recheck on 2026-10-21 confirmed the active akyldash-act-runner.service runs as akyldash-runner with ProtectHome=read-only; ReadWritePaths includes the runner runtime/config/log paths and /run/user/999. Its ExecStart uses /etc/akyldash-runner/config.yaml. The live unit retains a writable carveout for /run/user/999; the installer keeps that existing carveout and changes ProtectHome from yes to read-only so the runtime path remains visible. No VPS unit was changed. Main commit d2fb4b81 passed CI run 115 and smoke run 116. On head 3a89dc21, CI run 63 failed at the host audit root guard and smoke run 64 succeeded. On head 436806a8, CI run 65 failed in the Caddy container action with exec: "validate": executable file not found; Kaniko was skipped and smoke run 66 succeeded. On head d96fc4d3, CI run 67 failed because the Caddy action's fixed /github/workspace/... path did not exist; Kaniko was skipped and smoke run 68 succeeded. On head 5d72bb09, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile and built the backend and OpenSearch images using ${{ github.workspace }}. Actual per-action resource-limit enforcement and peak usage were not independently measured or verified. Image publishing, host-wide resource metrics, and production deployment remain unverified. No VPS changes are made.
  • An isolated local Compose smoke test built backend/OpenSearch images, reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP 200 for /review through Caddy. Its containers, networks, and volumes were removed afterward. This does not verify VPS startup, DNS, or public TLS.
  • The app has no migration framework or SMTP setup. Repository versions on this branch are: backend 0.13.0, frontend 0.2.0, Telegram bot 0.2.2 (the bot is not deployed on this VPS).

Corpus readiness (2026-10-18)

  • The local raw archive and normalized corpus are ignored under data/; no corpus files were changed. Their read-only SQLite manifests both pass quick_check and each report 209,958 documents. The raw manifest is 28,295,168 bytes with SHA-256 f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e; it has one fetch error, code 6, absent from documents. The normalized manifest is 31,571,968 bytes with SHA-256 1679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf; all entries are success, schema 1, normalizer 1.0.0.
  • The read-only full-file inventory in /tmp/akyldash-corpus-audit.Mjr6aa stopped before validation in run b4a4556e30cc464d90bc8d75ef1673e9 with Python int too large to convert to SQLite INTEGER. At stop it had processed 2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856 normalized entries (246,321 files, 151,535 directories). This is a verifier metadata storage failure, not a corpus finding. A full rehash with the corrected, versioned verifier is required before readiness can be stated.
  • A fresh full rehash is running in /tmp/akyldash-corpus-audit-final.gtsD87vc, run f4dd5e53020f442bbbecee2d0e4986a6, with verifier SHA-256 83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5. result.json currently reports scan_status: running; its live counters must be recorded after completion. Sources remain read-only.
  • Through the repository client, bounded MinJust OpenData requests for GetDocumentListByQuery, GetDocument(6), control GetDocument(1096), and CheckAvailable each returned HTTP 404. The list API returned no code list; the direct requests do not establish whether code 6 exists. The public listing page is reachable, but it does not map the numeric API code. A metadata-only CheckAvailable request from the VPS failed in HTTP/2 with curl exit 92 (INTERNAL_ERROR), HTTP status 000, and TLS verification 0; the HTTP/1.1 retry was not completed. Treat the code 6 disposition and full corpus readiness as unresolved.
  • On the VPS, read-only stat of /var/lib/akyldash/data/minjust-normalized and its manifest was denied. Production corpus presence and metadata remain unknown. No corpus was transferred and no production service, Docker socket, runner policy, or VPS configuration was changed.

Current MinJust API staging (2026-10-20)

  • The public site's current API uses POST /api/v1/GetDocuments with page size 12, GET /api/v1/GetDocument?DocumentCode=..., and GET /api/v1/GetEdition?editionId=...&lang=ru|kg. A bounded page-1 probe reported 210,639 filtered records; no full listing was enumerated for this recovery. Official alias 1-4 resolves to canonical document code 205064.
  • Document code 6 has 201 official editions. The API's lang=kg response places Kyrgyz HTML in contentRu while returning contentKg=null; staging maps by requested language (kg to archive ky) and preserves the edition order from the document response. Normalization and search export now retain and use that order instead of sorting IDs.
  • Without exact=true, one probe for edition 920292 returned response field id=920492 despite a different text length/hash. Two bounded exact=true probes returned matching IDs and names/dates from GetDocument: edition 920292 / 04.08.1998 (485,487 RU characters, SHA-256 2b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718) and edition 920492 / 13.04.2017 (1,139,908 RU characters, SHA-256 e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8). Every edition request now uses exact=true, requires returned ID equality, and checks RU/KY names against GetDocument metadata.
  • backend/ingestion/minjust_current.py stages one document into a separate generation, at no more than one request per second, with a 16 MiB response limit, a 4 GiB text budget per document, contract-bound checkpoints, and atomic publication. Staging requires a private 0700 directory owned by the current user and rejects symlinks under checkpoints/document paths; legacy corpus exclusions are anchored to the repository root. The corpus verifier continues to validate unchanged normalized manifests from versions 1.0.0 and 1.1.0. The earlier non-exact pass is quarantined at /tmp/akyldash-current-code6.ZAMC37u4ex/qualification.json: its 240 language responses (304,493,520 bytes) remain unverified and are not reusable. The retained work-tree SHA-256 is 1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a.
  • Exact-only code-6 staging is complete at /tmp/akyldash-current-code6-exact.L4k3AvnOlP: 201 editions and 402 complete language checkpoints, 604,377,885 raw bytes. The raw manifest passes quick_check; its document checksum matches the staged directory at 81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea. Separate normalization at /tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalized contains 201 editions in official response order, passes quick_check, and records normalizer 1.1.0 (2,048,797,024 bytes). The prior data/minjust-cbd and data/minjust-normalized trees and production were not used as input or modified. The raw staging manifest still records attachments and embedded media as unverified; this single document does not establish corpus completeness or production readiness.

Checks not completed

  • Docker Engine version from the server API, container and volume inventory, protected env/secret metadata, sudoers, and detailed UFW rules require root. infra/bootstrap/audit-host.sh is provided for an owner-run read-only audit.
  • The backup archive/restore tool was tested with synthetic local data; a backup and restore of real production data remain unverified because production is not deployed.
  • Cloudflare edge response was checked; origin DNS target, origin certificate, Full (strict), registry publishing, host-wide runner metrics, and production deployment have not been verified. The owner-reported Idle state has not been independently checked through runner inventory because the API returned 403.
  • This audit made no host changes. No package or service was installed by this audit; the owner performed the initial bootstrap beforehand.

Next owner actions

  1. Install the approved root-owned checkout after its PR is reviewed, then run sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh and share the output. The script prints no secret values and does not mutate the host.
  2. After reviewing the audit output and confirming the target files are absent, run sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh; it refuses to overwrite existing files. Set the approved DATA_ROOT and place the approved corpus/index and Cloudflare origin configuration. Do not deploy until data/service state and the required backup path are reviewed.
  3. Run sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh after reviewing the checkout; it grants only fixed verification, Compose config/status, and guarded rollback commands.
  4. After merging the reviewed PR, confirm a subsequent CI and smoke run on akyldash-ci. Image publishing and production deployment remain manual and are not part of CI.
  5. Test backup/restore and rollback only with synthetic data in an isolated environment before any production deployment.

Change in this PR

Adds a read-only, resumable full-file corpus inventory and normalized-data cross-check, a source/staging comparator requiring nonzero documents and fragments plus held shared artifact locks, synthetic regression tests, and an operator handoff runbook. CI's existing infra/deploy unittest discovery includes the new tests. No workflow, runner, VPS, production data, or Docker settings are changed. The first real corpus scan stopped on an integer-storage bug; full validation with the corrected verifier is still required. The source API's code 6 outcome remains unconfirmed.