Files
akyldash/docs/operations/CURRENT_STATE.md
agent d01fdabd68
Runner smoke test / runner-smoke (pull_request) Failing after 13m50s
CI / tests-and-images (pull_request) Failing after 13m55s
feat(frontend): implement legal search interface
2026-10-23 00:00:41 +03:00

274 lines
17 KiB
Markdown

# Infrastructure current state
Read-only host recheck: 2026-10-18. Source: SSH as `codex`, local repository
inspection, and Gitea API. Root-only Docker state, UFW rule details, and
protected file metadata remain unverified.
## VPS facts
- Host: `avprogs.hlab.kz`, `206.223.249.118`, SSH port `22`, user `codex`.
- OS: Ubuntu Server `24.04.5 LTS`; kernel `6.8.0-142-generic` from the prior
inventory audit.
- Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with
84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop
mount with 12 GiB free at that check.
- Network: interface `ens3`, IPv4 `206.223.249.118/32`.
- `codex` is UID/GID 1000, groups `codex` and `users`; `sudo -n true` requires
a password. Docker CLI cannot access `/var/run/docker.sock` as `codex`.
- SSH service is running. The inspected sshd configuration has public-key and
password authentication enabled and `PermitRootLogin yes`; no SSH changes
were made.
- Bootstrap completed. Docker CLI `29.9.0` and Compose `5.6.0` were rechecked
over SSH on 2026-10-18; containerd `2.4.1` and runc `1.5.2` are from the
prior package audit. The Docker daemon API remains root-only.
- `/etc/docker/daemon.json` has the expected `local` log driver and `10m` / `3`
rotation limits. Bootstrap paths exist with root ownership: `/opt/akyldash`
mode `0755`, `/etc/akyldash` and `/var/backups/akyldash` mode `0700`, and
`/var/lib/akyldash/data` mode `0750`.
- `/opt/akyldash/current`, verifier/rollback binaries, and the sudoers file are
absent. `codex` has not received sudo or Docker-group access. Metadata for
`/etc/akyldash/production.env` and `/etc/akyldash/review_secret` could not be
checked because access is denied. The target
`/var/lib/akyldash/data/minjust-normalized` and its manifest metadata were
also denied; their presence, ownership, and contents are unknown. No sudo was
used.
- TCP listeners observed: SSH `22` and loopback DNS only; no listener on `1500`,
`80`, or `443`. No ISPmanager package or service unit was found through the
unprivileged checks; root can confirm with the audit script below.
- Owner reports UFW is active and allows TCP `22`, `80`, `443`, and `1500`.
Rule details were not independently read; no firewall changes were made.
- Additional read-only runner checks on 2026-10-12 confirmed `/` is ext4 with
about 92 GiB free at the prior audit, `/usr/bin/dockerd-rootless.sh` exists,
and `git.vakanaut.ru` resolves to `195.209.48.28`. From the VPS,
`https://git.vakanaut.ru/api/v1/version` returned Gitea `1.25.5`; runner
registration and Gitea Actions protocol connectivity are not confirmed.
- `akyldash.kg` resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS,
HTTPS returned Cloudflare `521` with TLS verification result `0` on
2026-10-18. TCP listeners were SSH `22` and local DNS only, so no origin
listener on `80`/`443` was observed. Origin TLS and Cloudflare Full (strict)
are not verified.
## Application and Gitea facts
- Repository: `admin/akyldash`, default branch `main`; Gitea version `1.25.5`.
- The repository API reports push permission, Actions and packages enabled, and
PRs enabled. `main` is protected and requires one approval; the current API
principal cannot push or merge directly to `main`.
- The repository runner-list API returns `403` because this API principal is
not the repository owner. The latest verified CI run 75 and smoke run 76 both
succeeded on `main` head `0e276ae6`. The Actions run-list endpoint returned
`403` for this API principal during this recheck, so no newer run state was
independently retrieved. The owner reports the runner is registered and
Idle; repository runner inventory and package publishing permissions remain
unverified.
- No public application listener is observed on TCP `80`/`443`. Docker
container/volume state still requires the root audit. Repository Compose describes
Caddy `2.11.6` pinned by digest, Python Search API `0.12.2` in this branch,
SQLite account/review data, and OpenSearch `3.7.0` with `analysis-icu`.
- The target publishes only Caddy on TCP `80`/`443`. Backend and Caddy share a
proxy network; OpenSearch and backend share a separate internal network, and
backend has no host port. Caddy certificate storage, OpenSearch, and
application data use persistent volumes/bind mounts.
- The root-owned repository checkout, production env/secret files, Compose
volumes/containers, backup, verifier/rollback helpers, and application are
not confirmed installed. TLS and registry digest deployment for
backend/OpenSearch are not configured.
- The repository has a cold-backup archive/verification tool and a one-shot
restore apply tool that requires a marked replacement host. Backup/restore
tests use synthetic data; neither tool has run against production. The
archive preflight permits a clean first deployment to skip backup, requires
backup for existing persistent data even without containers, allows absent
named volumes only in explicit corpus-only initial-deployment mode, and
rejects inaccessible/ambiguous mounts and incomplete/running service
containers. Initial deployment docs require a second verified backup after
index and admin initialization.
- Recovery bootstrap has an explicit `--recovery-target` mode that refuses the
inventoried production hostname/IP; its host-identity guard has shell tests.
The mode has not been run on a VPS.
- The repository defines a root-owned read-only verifier, Compose `config` and
`status` helper, guarded rollback helper, and installer with exact sudoers
command rules. None is installed on the VPS. CI contains no production
deployment job; starting/stopping services remains owner-run.
- Rollback preflight validates digest-pinned image references against archived
image IDs and local availability, then refuses before stopping services or
creating a backup because the current flow cannot safely switch immutable
digest references. This behavior is covered by synthetic local tests; it has
not been exercised on the VPS.
- This branch adds an offline release-manifest validator and tests for the
repository allowlist intended for a future root-owned host integration. The
manifest format does not prove source provenance. No registry, host policy,
publish credential, or deployment integration is configured or installed.
- A repository-scoped official `act_runner` `0.2.13` installation is
configured in `infra/runner/`: rootless Docker under `akyldash-runner`, 15 GiB
ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU,
3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048
tasks), rotated file logs, and a systemd health timer. Jobs target the
`akyldash-ci` label and do not receive either Docker socket. On 2026-10-18,
SSH showed `akyldash-act-runner.service` active under `akyldash-runner`, with
zero restarts and about 115 MiB current memory. Its rootless Docker store is
mounted from `/dev/loop0` as ext4 with `rw,nosuid,nodev`; the 15 GiB mount had
12 GiB free. The separate `akyldash-runner-restart.service` remains failed
with exit status 1 since 2026-10-14; its journal had no entries, so the cause
is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on
head `0e276ae6`. The Actions API returned 403 during this recheck, so no newer
run state was independently retrieved. A read-only SSH recheck on 2026-10-21
confirmed the active `akyldash-act-runner.service` runs as
`akyldash-runner` with `ProtectHome=read-only`; `ReadWritePaths` includes the
runner runtime/config/log paths and `/run/user/999`. Its `ExecStart` uses
`/etc/akyldash-runner/config.yaml`. The live unit retains a writable carveout
for `/run/user/999`; the installer keeps that existing carveout and changes
`ProtectHome` from `yes` to `read-only` so the runtime path remains visible.
No VPS unit was changed. Main commit
`d2fb4b81` passed CI run 115 and smoke run 116. On
head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run 64
succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
with `exec: "validate": executable file not found`; Kaniko was skipped and
smoke run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the
Caddy action's fixed `/github/workspace/...` path did not exist; Kaniko was
skipped and smoke run 68 succeeded. On head `5d72bb09`, CI run 69 and smoke
run 70 succeeded. CI validated the Caddyfile and built the backend and
OpenSearch images using `${{ github.workspace }}`. Actual per-action
resource-limit enforcement and peak usage were not independently measured or
verified. Image publishing, host-wide resource metrics, and production
deployment remain unverified. No VPS changes are made.
- An isolated local Compose smoke test built backend/OpenSearch images,
reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP
200 for `/review` through Caddy. Its containers, networks, and volumes were
removed afterward. This does not verify VPS startup, DNS, or public TLS.
- The app has no migration framework or SMTP setup. Repository versions on this
branch are:
backend `0.13.0`, frontend `0.2.0`, Telegram bot `0.2.2` (the bot is not
deployed on this VPS).
## Corpus readiness (2026-10-18)
- The local raw archive and normalized corpus are ignored under `data/`; no
corpus files were changed. Their read-only SQLite manifests both pass
`quick_check` and each report 209,958 documents. The raw manifest is
28,295,168 bytes with SHA-256
`f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e`; it has
one fetch error, code `6`, absent from `documents`. The normalized manifest
is 31,571,968 bytes with SHA-256
`1679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf`; all
entries are `success`, schema `1`, normalizer `1.0.0`.
- The read-only full-file inventory in `/tmp/akyldash-corpus-audit.Mjr6aa`
stopped before validation in run `b4a4556e30cc464d90bc8d75ef1673e9` with
`Python int too large to convert to SQLite INTEGER`. At stop it had processed
2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856
normalized entries (246,321 files, 151,535 directories). This is a verifier
metadata storage failure, not a corpus finding. A full rehash with the
corrected, versioned verifier is required before readiness can be stated.
- A fresh full rehash is running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`,
run `f4dd5e53020f442bbbecee2d0e4986a6`, with verifier SHA-256
`83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`.
`result.json` currently reports `scan_status: running`; its live counters
must be recorded after completion. Sources remain read-only.
- Through the repository client, bounded MinJust OpenData requests for
`GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and
`CheckAvailable` each returned HTTP `404`. The list API returned no code list;
the direct requests do not establish whether code `6` exists. The public
listing page is reachable, but it does not map the numeric API code. A
metadata-only `CheckAvailable` request from the VPS failed in HTTP/2 with
`curl` exit 92 (`INTERNAL_ERROR`), HTTP status `000`, and TLS verification 0;
the HTTP/1.1 retry was not completed. Treat the code `6` disposition and
full corpus readiness as unresolved.
- On the VPS, read-only `stat` of `/var/lib/akyldash/data/minjust-normalized`
and its manifest was denied. Production corpus presence and metadata remain
unknown. No corpus was transferred and no production service, Docker socket,
runner policy, or VPS configuration was changed.
## Current MinJust API staging (2026-10-20)
- The public site's current API uses `POST /api/v1/GetDocuments` with page size
12, `GET /api/v1/GetDocument?DocumentCode=...`, and
`GET /api/v1/GetEdition?editionId=...&lang=ru|kg`. A bounded page-1 probe
reported 210,639 filtered records; no full listing was enumerated for this
recovery. Official alias `1-4` resolves to canonical document code `205064`.
- Document code `6` has 201 official editions. The API's `lang=kg` response
places Kyrgyz HTML in `contentRu` while returning `contentKg=null`; staging
maps by requested language (`kg` to archive `ky`) and preserves the edition
order from the document response. Normalization and search export now retain
and use that order instead of sorting IDs.
- Without `exact=true`, one probe for edition `920292` returned response field
`id=920492` despite a different text length/hash. Two bounded `exact=true`
probes returned matching IDs and names/dates from `GetDocument`: edition
`920292` / `04.08.1998` (485,487 RU characters, SHA-256
`2b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718`) and
edition `920492` / `13.04.2017` (1,139,908 RU characters, SHA-256
`e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8`).
Every edition request now uses `exact=true`, requires returned ID equality,
and checks RU/KY names against `GetDocument` metadata.
- `backend/ingestion/minjust_current.py` stages one document into a separate
generation, at no more than one request per second, with a 16 MiB response
limit, a 4 GiB text budget per document, contract-bound checkpoints, and
atomic publication. Staging requires a private 0700 directory owned by the
current user and rejects symlinks under checkpoints/document paths; legacy
corpus exclusions are anchored to the repository root. The corpus verifier
continues to validate unchanged normalized manifests from versions 1.0.0
and 1.1.0. The earlier non-exact pass is quarantined at
`/tmp/akyldash-current-code6.ZAMC37u4ex/qualification.json`: its 240 language
responses (304,493,520 bytes) remain unverified and are not reusable. The
retained work-tree SHA-256 is
`1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a`.
- Exact-only code-6 staging is complete at
`/tmp/akyldash-current-code6-exact.L4k3AvnOlP`: 201 editions and 402
complete language checkpoints, 604,377,885 raw bytes. The raw manifest
passes `quick_check`; its document checksum matches the staged directory at
`81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea`.
Separate normalization at
`/tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalized` contains 201
editions in official response order, passes `quick_check`, and records
normalizer `1.1.0` (2,048,797,024 bytes). The prior `data/minjust-cbd` and
`data/minjust-normalized` trees and production were not used as input or
modified. The raw staging manifest still records attachments and embedded
media as unverified; this single document does not establish corpus
completeness or production readiness.
## Checks not completed
- Docker Engine version from the server API, container and volume inventory,
protected env/secret metadata, sudoers, and detailed UFW rules require root.
`infra/bootstrap/audit-host.sh` is provided for an owner-run read-only audit.
- The backup archive/restore tool was tested with synthetic local data; a backup
and restore of real production data remain unverified because production is
not deployed.
- Cloudflare edge response was checked; origin DNS target, origin certificate,
Full (strict), registry publishing, host-wide runner metrics, and production
deployment have not been verified. The
owner-reported Idle state has not been independently checked through runner
inventory because the API returned 403.
- This audit made no host changes. No package or service was installed by this
audit; the owner performed the initial bootstrap beforehand.
## Next owner actions
1. Install the approved root-owned checkout after its PR is reviewed, then run
`sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh` and share
the output. The script prints no secret values and does not mutate the host.
2. After reviewing the audit output and confirming the target files are absent,
run
`sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh`;
it refuses to overwrite existing files. Set the approved `DATA_ROOT` and
place the approved corpus/index and Cloudflare origin configuration. Do not
deploy until data/service state and the required backup path are reviewed.
3. Run `sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh` after
reviewing the checkout; it grants only fixed verification, Compose
`config`/`status`, and guarded rollback commands.
4. After merging the reviewed PR, confirm a subsequent CI and smoke run on
`akyldash-ci`. Image publishing and production deployment remain manual and
are not part of CI.
5. Test backup/restore and rollback only with synthetic data in an isolated
environment before any production deployment.
## Change in this PR
Adds a read-only, resumable full-file corpus inventory and normalized-data
cross-check, a source/staging comparator requiring nonzero documents and
fragments plus held shared artifact locks, synthetic regression tests, and an
operator handoff runbook. CI's
existing `infra/deploy` unittest discovery includes the new tests. No workflow,
runner, VPS, production data, or Docker settings are changed. The first real
corpus scan stopped on an integer-storage bug; full validation with the
corrected verifier is still required. The source API's code `6` outcome remains
unconfirmed.