17 KiB
Infrastructure current state
Read-only host recheck: 2026-10-18. Source: SSH as codex, local repository
inspection, and Gitea API. Root-only Docker state, UFW rule details, and
protected file metadata remain unverified.
VPS facts
- Host:
avprogs.hlab.kz,206.223.249.118, SSH port22, usercodex. - OS: Ubuntu Server
24.04.5 LTS; kernel6.8.0-142-genericfrom the prior inventory audit. - Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with 84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop mount with 12 GiB free at that check.
- Network: interface
ens3, IPv4206.223.249.118/32. codexis UID/GID 1000, groupscodexandusers;sudo -n truerequires a password. Docker CLI cannot access/var/run/docker.sockascodex.- SSH service is running. The inspected sshd configuration has public-key and
password authentication enabled and
PermitRootLogin yes; no SSH changes were made. - Bootstrap completed. Docker CLI
29.9.0and Compose5.6.0were rechecked over SSH on 2026-10-18; containerd2.4.1and runc1.5.2are from the prior package audit. The Docker daemon API remains root-only. /etc/docker/daemon.jsonhas the expectedlocallog driver and10m/3rotation limits. Bootstrap paths exist with root ownership:/opt/akyldashmode0755,/etc/akyldashand/var/backups/akyldashmode0700, and/var/lib/akyldash/datamode0750./opt/akyldash/current, verifier/rollback binaries, and the sudoers file are absent.codexhas not received sudo or Docker-group access. Metadata for/etc/akyldash/production.envand/etc/akyldash/review_secretcould not be checked because access is denied. The target/var/lib/akyldash/data/minjust-normalizedand its manifest metadata were also denied; their presence, ownership, and contents are unknown. No sudo was used.- TCP listeners observed: SSH
22and loopback DNS only; no listener on1500,80, or443. No ISPmanager package or service unit was found through the unprivileged checks; root can confirm with the audit script below. - Owner reports UFW is active and allows TCP
22,80,443, and1500. Rule details were not independently read; no firewall changes were made. - Additional read-only runner checks on 2026-10-12 confirmed
/is ext4 with about 92 GiB free at the prior audit,/usr/bin/dockerd-rootless.shexists, andgit.vakanaut.ruresolves to195.209.48.28. From the VPS,https://git.vakanaut.ru/api/v1/versionreturned Gitea1.25.5; runner registration and Gitea Actions protocol connectivity are not confirmed. akyldash.kgresolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS, HTTPS returned Cloudflare521with TLS verification result0on 2026-10-18. TCP listeners were SSH22and local DNS only, so no origin listener on80/443was observed. Origin TLS and Cloudflare Full (strict) are not verified.
Application and Gitea facts
- Repository:
admin/akyldash, default branchmain; Gitea version1.25.5. - The repository API reports push permission, Actions and packages enabled, and
PRs enabled.
mainis protected and requires one approval; the current API principal cannot push or merge directly tomain. - The repository runner-list API returns
403because this API principal is not the repository owner. The latest verified CI run 75 and smoke run 76 both succeeded onmainhead0e276ae6. The Actions run-list endpoint returned403for this API principal during this recheck, so no newer run state was independently retrieved. The owner reports the runner is registered and Idle; repository runner inventory and package publishing permissions remain unverified. - No public application listener is observed on TCP
80/443. Docker container/volume state still requires the root audit. Repository Compose describes Caddy2.11.6pinned by digest, Python Search API0.12.2in this branch, SQLite account/review data, and OpenSearch3.7.0withanalysis-icu. - The target publishes only Caddy on TCP
80/443. Backend and Caddy share a proxy network; OpenSearch and backend share a separate internal network, and backend has no host port. Caddy certificate storage, OpenSearch, and application data use persistent volumes/bind mounts. - The root-owned repository checkout, production env/secret files, Compose volumes/containers, backup, verifier/rollback helpers, and application are not confirmed installed. TLS and registry digest deployment for backend/OpenSearch are not configured.
- The repository has a cold-backup archive/verification tool and a one-shot restore apply tool that requires a marked replacement host. Backup/restore tests use synthetic data; neither tool has run against production. The archive preflight permits a clean first deployment to skip backup, requires backup for existing persistent data even without containers, allows absent named volumes only in explicit corpus-only initial-deployment mode, and rejects inaccessible/ambiguous mounts and incomplete/running service containers. Initial deployment docs require a second verified backup after index and admin initialization.
- Recovery bootstrap has an explicit
--recovery-targetmode that refuses the inventoried production hostname/IP; its host-identity guard has shell tests. The mode has not been run on a VPS. - The repository defines a root-owned read-only verifier, Compose
configandstatushelper, guarded rollback helper, and installer with exact sudoers command rules. None is installed on the VPS. CI contains no production deployment job; starting/stopping services remains owner-run. - Rollback preflight validates digest-pinned image references against archived image IDs and local availability, then refuses before stopping services or creating a backup because the current flow cannot safely switch immutable digest references. This behavior is covered by synthetic local tests; it has not been exercised on the VPS.
- This branch adds an offline release-manifest validator and tests for the repository allowlist intended for a future root-owned host integration. The manifest format does not prove source provenance. No registry, host policy, publish credential, or deployment integration is configured or installed.
- A repository-scoped official
act_runner0.2.13installation is configured ininfra/runner/: rootless Docker underakyldash-runner, 15 GiB ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU, 3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048 tasks), rotated file logs, and a systemd health timer. Jobs target theakyldash-cilabel and do not receive either Docker socket. On 2026-10-18, SSH showedakyldash-act-runner.serviceactive underakyldash-runner, with zero restarts and about 115 MiB current memory. Its rootless Docker store is mounted from/dev/loop0as ext4 withrw,nosuid,nodev; the 15 GiB mount had 12 GiB free. The separateakyldash-runner-restart.serviceremains failed with exit status 1 since 2026-10-14; its journal had no entries, so the cause is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on head0e276ae6. The Actions API returned 403 during this recheck, so no newer run state was independently retrieved. A read-only SSH recheck on 2026-10-21 confirmed the activeakyldash-act-runner.serviceruns asakyldash-runnerwithProtectHome=read-only;ReadWritePathsincludes the runner runtime/config/log paths and/run/user/999. ItsExecStartuses/etc/akyldash-runner/config.yaml. The live unit retains a writable carveout for/run/user/999; the installer keeps that existing carveout and changesProtectHomefromyestoread-onlyso the runtime path remains visible. No VPS unit was changed. Main commitd2fb4b81passed CI run 115 and smoke run 116. On head3a89dc21, CI run 63 failed at the host audit root guard and smoke run 64 succeeded. On head436806a8, CI run 65 failed in the Caddy container action withexec: "validate": executable file not found; Kaniko was skipped and smoke run 66 succeeded. On headd96fc4d3, CI run 67 failed because the Caddy action's fixed/github/workspace/...path did not exist; Kaniko was skipped and smoke run 68 succeeded. On head5d72bb09, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile and built the backend and OpenSearch images using${{ github.workspace }}. Actual per-action resource-limit enforcement and peak usage were not independently measured or verified. Image publishing, host-wide resource metrics, and production deployment remain unverified. No VPS changes are made. - An isolated local Compose smoke test built backend/OpenSearch images,
reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP
200 for
/reviewthrough Caddy. Its containers, networks, and volumes were removed afterward. This does not verify VPS startup, DNS, or public TLS. - The app has no migration framework or SMTP setup. Repository versions on this
branch are:
backend
0.13.0, frontend0.2.0, Telegram bot0.2.2(the bot is not deployed on this VPS).
Corpus readiness (2026-10-18)
- The local raw archive and normalized corpus are ignored under
data/; no corpus files were changed. Their read-only SQLite manifests both passquick_checkand each report 209,958 documents. The raw manifest is 28,295,168 bytes with SHA-256f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e; it has one fetch error, code6, absent fromdocuments. The normalized manifest is 31,571,968 bytes with SHA-2561679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf; all entries aresuccess, schema1, normalizer1.0.0. - The read-only full-file inventory in
/tmp/akyldash-corpus-audit.Mjr6aastopped before validation in runb4a4556e30cc464d90bc8d75ef1673e9withPython int too large to convert to SQLite INTEGER. At stop it had processed 2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856 normalized entries (246,321 files, 151,535 directories). This is a verifier metadata storage failure, not a corpus finding. A full rehash with the corrected, versioned verifier is required before readiness can be stated. - A fresh full rehash is running in
/tmp/akyldash-corpus-audit-final.gtsD87vc, runf4dd5e53020f442bbbecee2d0e4986a6, with verifier SHA-25683dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5.result.jsoncurrently reportsscan_status: running; its live counters must be recorded after completion. Sources remain read-only. - Through the repository client, bounded MinJust OpenData requests for
GetDocumentListByQuery,GetDocument(6), controlGetDocument(1096), andCheckAvailableeach returned HTTP404. The list API returned no code list; the direct requests do not establish whether code6exists. The public listing page is reachable, but it does not map the numeric API code. A metadata-onlyCheckAvailablerequest from the VPS failed in HTTP/2 withcurlexit 92 (INTERNAL_ERROR), HTTP status000, and TLS verification 0; the HTTP/1.1 retry was not completed. Treat the code6disposition and full corpus readiness as unresolved. - On the VPS, read-only
statof/var/lib/akyldash/data/minjust-normalizedand its manifest was denied. Production corpus presence and metadata remain unknown. No corpus was transferred and no production service, Docker socket, runner policy, or VPS configuration was changed.
Current MinJust API staging (2026-10-20)
- The public site's current API uses
POST /api/v1/GetDocumentswith page size 12,GET /api/v1/GetDocument?DocumentCode=..., andGET /api/v1/GetEdition?editionId=...&lang=ru|kg. A bounded page-1 probe reported 210,639 filtered records; no full listing was enumerated for this recovery. Official alias1-4resolves to canonical document code205064. - Document code
6has 201 official editions. The API'slang=kgresponse places Kyrgyz HTML incontentRuwhile returningcontentKg=null; staging maps by requested language (kgto archiveky) and preserves the edition order from the document response. Normalization and search export now retain and use that order instead of sorting IDs. - Without
exact=true, one probe for edition920292returned response fieldid=920492despite a different text length/hash. Two boundedexact=trueprobes returned matching IDs and names/dates fromGetDocument: edition920292/04.08.1998(485,487 RU characters, SHA-2562b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718) and edition920492/13.04.2017(1,139,908 RU characters, SHA-256e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8). Every edition request now usesexact=true, requires returned ID equality, and checks RU/KY names againstGetDocumentmetadata. backend/ingestion/minjust_current.pystages one document into a separate generation, at no more than one request per second, with a 16 MiB response limit, a 4 GiB text budget per document, contract-bound checkpoints, and atomic publication. Staging requires a private 0700 directory owned by the current user and rejects symlinks under checkpoints/document paths; legacy corpus exclusions are anchored to the repository root. The corpus verifier continues to validate unchanged normalized manifests from versions 1.0.0 and 1.1.0. The earlier non-exact pass is quarantined at/tmp/akyldash-current-code6.ZAMC37u4ex/qualification.json: its 240 language responses (304,493,520 bytes) remain unverified and are not reusable. The retained work-tree SHA-256 is1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a.- Exact-only code-6 staging is complete at
/tmp/akyldash-current-code6-exact.L4k3AvnOlP: 201 editions and 402 complete language checkpoints, 604,377,885 raw bytes. The raw manifest passesquick_check; its document checksum matches the staged directory at81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea. Separate normalization at/tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalizedcontains 201 editions in official response order, passesquick_check, and records normalizer1.1.0(2,048,797,024 bytes). The priordata/minjust-cbdanddata/minjust-normalizedtrees and production were not used as input or modified. The raw staging manifest still records attachments and embedded media as unverified; this single document does not establish corpus completeness or production readiness.
Checks not completed
- Docker Engine version from the server API, container and volume inventory,
protected env/secret metadata, sudoers, and detailed UFW rules require root.
infra/bootstrap/audit-host.shis provided for an owner-run read-only audit. - The backup archive/restore tool was tested with synthetic local data; a backup and restore of real production data remain unverified because production is not deployed.
- Cloudflare edge response was checked; origin DNS target, origin certificate, Full (strict), registry publishing, host-wide runner metrics, and production deployment have not been verified. The owner-reported Idle state has not been independently checked through runner inventory because the API returned 403.
- This audit made no host changes. No package or service was installed by this audit; the owner performed the initial bootstrap beforehand.
Next owner actions
- Install the approved root-owned checkout after its PR is reviewed, then run
sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.shand share the output. The script prints no secret values and does not mutate the host. - After reviewing the audit output and confirming the target files are absent,
run
sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh; it refuses to overwrite existing files. Set the approvedDATA_ROOTand place the approved corpus/index and Cloudflare origin configuration. Do not deploy until data/service state and the required backup path are reviewed. - Run
sudo bash /opt/akyldash/current/infra/deploy/install-helper.shafter reviewing the checkout; it grants only fixed verification, Composeconfig/status, and guarded rollback commands. - After merging the reviewed PR, confirm a subsequent CI and smoke run on
akyldash-ci. Image publishing and production deployment remain manual and are not part of CI. - Test backup/restore and rollback only with synthetic data in an isolated environment before any production deployment.
Change in this PR
Adds a read-only, resumable full-file corpus inventory and normalized-data
cross-check, a source/staging comparator requiring nonzero documents and
fragments plus held shared artifact locks, synthetic regression tests, and an
operator handoff runbook. CI's
existing infra/deploy unittest discovery includes the new tests. No workflow,
runner, VPS, production data, or Docker settings are changed. The first real
corpus scan stopped on an integer-storage bug; full validation with the
corrected verifier is still required. The source API's code 6 outcome remains
unconfirmed.