274 lines
17 KiB
Markdown
274 lines
17 KiB
Markdown
# Infrastructure current state
|
|
|
|
Read-only host recheck: 2026-10-18. Source: SSH as `codex`, local repository
|
|
inspection, and Gitea API. Root-only Docker state, UFW rule details, and
|
|
protected file metadata remain unverified.
|
|
|
|
## VPS facts
|
|
|
|
- Host: `avprogs.hlab.kz`, `206.223.249.118`, SSH port `22`, user `codex`.
|
|
- OS: Ubuntu Server `24.04.5 LTS`; kernel `6.8.0-142-generic` from the prior
|
|
inventory audit.
|
|
- Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with
|
|
84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop
|
|
mount with 12 GiB free at that check.
|
|
- Network: interface `ens3`, IPv4 `206.223.249.118/32`.
|
|
- `codex` is UID/GID 1000, groups `codex` and `users`; `sudo -n true` requires
|
|
a password. Docker CLI cannot access `/var/run/docker.sock` as `codex`.
|
|
- SSH service is running. The inspected sshd configuration has public-key and
|
|
password authentication enabled and `PermitRootLogin yes`; no SSH changes
|
|
were made.
|
|
- Bootstrap completed. Docker CLI `29.9.0` and Compose `5.6.0` were rechecked
|
|
over SSH on 2026-10-18; containerd `2.4.1` and runc `1.5.2` are from the
|
|
prior package audit. The Docker daemon API remains root-only.
|
|
- `/etc/docker/daemon.json` has the expected `local` log driver and `10m` / `3`
|
|
rotation limits. Bootstrap paths exist with root ownership: `/opt/akyldash`
|
|
mode `0755`, `/etc/akyldash` and `/var/backups/akyldash` mode `0700`, and
|
|
`/var/lib/akyldash/data` mode `0750`.
|
|
- `/opt/akyldash/current`, verifier/rollback binaries, and the sudoers file are
|
|
absent. `codex` has not received sudo or Docker-group access. Metadata for
|
|
`/etc/akyldash/production.env` and `/etc/akyldash/review_secret` could not be
|
|
checked because access is denied. The target
|
|
`/var/lib/akyldash/data/minjust-normalized` and its manifest metadata were
|
|
also denied; their presence, ownership, and contents are unknown. No sudo was
|
|
used.
|
|
- TCP listeners observed: SSH `22` and loopback DNS only; no listener on `1500`,
|
|
`80`, or `443`. No ISPmanager package or service unit was found through the
|
|
unprivileged checks; root can confirm with the audit script below.
|
|
- Owner reports UFW is active and allows TCP `22`, `80`, `443`, and `1500`.
|
|
Rule details were not independently read; no firewall changes were made.
|
|
- Additional read-only runner checks on 2026-10-12 confirmed `/` is ext4 with
|
|
about 92 GiB free at the prior audit, `/usr/bin/dockerd-rootless.sh` exists,
|
|
and `git.vakanaut.ru` resolves to `195.209.48.28`. From the VPS,
|
|
`https://git.vakanaut.ru/api/v1/version` returned Gitea `1.25.5`; runner
|
|
registration and Gitea Actions protocol connectivity are not confirmed.
|
|
- `akyldash.kg` resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS,
|
|
HTTPS returned Cloudflare `521` with TLS verification result `0` on
|
|
2026-10-18. TCP listeners were SSH `22` and local DNS only, so no origin
|
|
listener on `80`/`443` was observed. Origin TLS and Cloudflare Full (strict)
|
|
are not verified.
|
|
|
|
## Application and Gitea facts
|
|
|
|
- Repository: `admin/akyldash`, default branch `main`; Gitea version `1.25.5`.
|
|
- The repository API reports push permission, Actions and packages enabled, and
|
|
PRs enabled. `main` is protected and requires one approval; the current API
|
|
principal cannot push or merge directly to `main`.
|
|
- The repository runner-list API returns `403` because this API principal is
|
|
not the repository owner. The latest verified CI run 75 and smoke run 76 both
|
|
succeeded on `main` head `0e276ae6`. The Actions run-list endpoint returned
|
|
`403` for this API principal during this recheck, so no newer run state was
|
|
independently retrieved. The owner reports the runner is registered and
|
|
Idle; repository runner inventory and package publishing permissions remain
|
|
unverified.
|
|
- No public application listener is observed on TCP `80`/`443`. Docker
|
|
container/volume state still requires the root audit. Repository Compose describes
|
|
Caddy `2.11.6` pinned by digest, Python Search API `0.12.2` in this branch,
|
|
SQLite account/review data, and OpenSearch `3.7.0` with `analysis-icu`.
|
|
- The target publishes only Caddy on TCP `80`/`443`. Backend and Caddy share a
|
|
proxy network; OpenSearch and backend share a separate internal network, and
|
|
backend has no host port. Caddy certificate storage, OpenSearch, and
|
|
application data use persistent volumes/bind mounts.
|
|
- The root-owned repository checkout, production env/secret files, Compose
|
|
volumes/containers, backup, verifier/rollback helpers, and application are
|
|
not confirmed installed. TLS and registry digest deployment for
|
|
backend/OpenSearch are not configured.
|
|
- The repository has a cold-backup archive/verification tool and a one-shot
|
|
restore apply tool that requires a marked replacement host. Backup/restore
|
|
tests use synthetic data; neither tool has run against production. The
|
|
archive preflight permits a clean first deployment to skip backup, requires
|
|
backup for existing persistent data even without containers, allows absent
|
|
named volumes only in explicit corpus-only initial-deployment mode, and
|
|
rejects inaccessible/ambiguous mounts and incomplete/running service
|
|
containers. Initial deployment docs require a second verified backup after
|
|
index and admin initialization.
|
|
- Recovery bootstrap has an explicit `--recovery-target` mode that refuses the
|
|
inventoried production hostname/IP; its host-identity guard has shell tests.
|
|
The mode has not been run on a VPS.
|
|
- The repository defines a root-owned read-only verifier, Compose `config` and
|
|
`status` helper, guarded rollback helper, and installer with exact sudoers
|
|
command rules. None is installed on the VPS. CI contains no production
|
|
deployment job; starting/stopping services remains owner-run.
|
|
- Rollback preflight validates digest-pinned image references against archived
|
|
image IDs and local availability, then refuses before stopping services or
|
|
creating a backup because the current flow cannot safely switch immutable
|
|
digest references. This behavior is covered by synthetic local tests; it has
|
|
not been exercised on the VPS.
|
|
- This branch adds an offline release-manifest validator and tests for the
|
|
repository allowlist intended for a future root-owned host integration. The
|
|
manifest format does not prove source provenance. No registry, host policy,
|
|
publish credential, or deployment integration is configured or installed.
|
|
- A repository-scoped official `act_runner` `0.2.13` installation is
|
|
configured in `infra/runner/`: rootless Docker under `akyldash-runner`, 15 GiB
|
|
ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU,
|
|
3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048
|
|
tasks), rotated file logs, and a systemd health timer. Jobs target the
|
|
`akyldash-ci` label and do not receive either Docker socket. On 2026-10-18,
|
|
SSH showed `akyldash-act-runner.service` active under `akyldash-runner`, with
|
|
zero restarts and about 115 MiB current memory. Its rootless Docker store is
|
|
mounted from `/dev/loop0` as ext4 with `rw,nosuid,nodev`; the 15 GiB mount had
|
|
12 GiB free. The separate `akyldash-runner-restart.service` remains failed
|
|
with exit status 1 since 2026-10-14; its journal had no entries, so the cause
|
|
is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on
|
|
head `0e276ae6`. The Actions API returned 403 during this recheck, so no newer
|
|
run state was independently retrieved. A read-only SSH recheck on 2026-10-21
|
|
confirmed the active `akyldash-act-runner.service` runs as
|
|
`akyldash-runner` with `ProtectHome=read-only`; `ReadWritePaths` includes the
|
|
runner runtime/config/log paths and `/run/user/999`. Its `ExecStart` uses
|
|
`/etc/akyldash-runner/config.yaml`. The live unit retains a writable carveout
|
|
for `/run/user/999`; the installer keeps that existing carveout and changes
|
|
`ProtectHome` from `yes` to `read-only` so the runtime path remains visible.
|
|
No VPS unit was changed. Main commit
|
|
`d2fb4b81` passed CI run 115 and smoke run 116. On
|
|
head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run 64
|
|
succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
|
|
with `exec: "validate": executable file not found`; Kaniko was skipped and
|
|
smoke run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the
|
|
Caddy action's fixed `/github/workspace/...` path did not exist; Kaniko was
|
|
skipped and smoke run 68 succeeded. On head `5d72bb09`, CI run 69 and smoke
|
|
run 70 succeeded. CI validated the Caddyfile and built the backend and
|
|
OpenSearch images using `${{ github.workspace }}`. Actual per-action
|
|
resource-limit enforcement and peak usage were not independently measured or
|
|
verified. Image publishing, host-wide resource metrics, and production
|
|
deployment remain unverified. No VPS changes are made.
|
|
- An isolated local Compose smoke test built backend/OpenSearch images,
|
|
reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP
|
|
200 for `/review` through Caddy. Its containers, networks, and volumes were
|
|
removed afterward. This does not verify VPS startup, DNS, or public TLS.
|
|
- The app has no migration framework or SMTP setup. Repository versions on this
|
|
branch are:
|
|
backend `0.13.0`, frontend `0.2.0`, Telegram bot `0.2.2` (the bot is not
|
|
deployed on this VPS).
|
|
|
|
## Corpus readiness (2026-10-18)
|
|
|
|
- The local raw archive and normalized corpus are ignored under `data/`; no
|
|
corpus files were changed. Their read-only SQLite manifests both pass
|
|
`quick_check` and each report 209,958 documents. The raw manifest is
|
|
28,295,168 bytes with SHA-256
|
|
`f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e`; it has
|
|
one fetch error, code `6`, absent from `documents`. The normalized manifest
|
|
is 31,571,968 bytes with SHA-256
|
|
`1679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf`; all
|
|
entries are `success`, schema `1`, normalizer `1.0.0`.
|
|
- The read-only full-file inventory in `/tmp/akyldash-corpus-audit.Mjr6aa`
|
|
stopped before validation in run `b4a4556e30cc464d90bc8d75ef1673e9` with
|
|
`Python int too large to convert to SQLite INTEGER`. At stop it had processed
|
|
2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856
|
|
normalized entries (246,321 files, 151,535 directories). This is a verifier
|
|
metadata storage failure, not a corpus finding. A full rehash with the
|
|
corrected, versioned verifier is required before readiness can be stated.
|
|
- A fresh full rehash is running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`,
|
|
run `f4dd5e53020f442bbbecee2d0e4986a6`, with verifier SHA-256
|
|
`83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`.
|
|
`result.json` currently reports `scan_status: running`; its live counters
|
|
must be recorded after completion. Sources remain read-only.
|
|
- Through the repository client, bounded MinJust OpenData requests for
|
|
`GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and
|
|
`CheckAvailable` each returned HTTP `404`. The list API returned no code list;
|
|
the direct requests do not establish whether code `6` exists. The public
|
|
listing page is reachable, but it does not map the numeric API code. A
|
|
metadata-only `CheckAvailable` request from the VPS failed in HTTP/2 with
|
|
`curl` exit 92 (`INTERNAL_ERROR`), HTTP status `000`, and TLS verification 0;
|
|
the HTTP/1.1 retry was not completed. Treat the code `6` disposition and
|
|
full corpus readiness as unresolved.
|
|
- On the VPS, read-only `stat` of `/var/lib/akyldash/data/minjust-normalized`
|
|
and its manifest was denied. Production corpus presence and metadata remain
|
|
unknown. No corpus was transferred and no production service, Docker socket,
|
|
runner policy, or VPS configuration was changed.
|
|
|
|
## Current MinJust API staging (2026-10-20)
|
|
|
|
- The public site's current API uses `POST /api/v1/GetDocuments` with page size
|
|
12, `GET /api/v1/GetDocument?DocumentCode=...`, and
|
|
`GET /api/v1/GetEdition?editionId=...&lang=ru|kg`. A bounded page-1 probe
|
|
reported 210,639 filtered records; no full listing was enumerated for this
|
|
recovery. Official alias `1-4` resolves to canonical document code `205064`.
|
|
- Document code `6` has 201 official editions. The API's `lang=kg` response
|
|
places Kyrgyz HTML in `contentRu` while returning `contentKg=null`; staging
|
|
maps by requested language (`kg` to archive `ky`) and preserves the edition
|
|
order from the document response. Normalization and search export now retain
|
|
and use that order instead of sorting IDs.
|
|
- Without `exact=true`, one probe for edition `920292` returned response field
|
|
`id=920492` despite a different text length/hash. Two bounded `exact=true`
|
|
probes returned matching IDs and names/dates from `GetDocument`: edition
|
|
`920292` / `04.08.1998` (485,487 RU characters, SHA-256
|
|
`2b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718`) and
|
|
edition `920492` / `13.04.2017` (1,139,908 RU characters, SHA-256
|
|
`e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8`).
|
|
Every edition request now uses `exact=true`, requires returned ID equality,
|
|
and checks RU/KY names against `GetDocument` metadata.
|
|
- `backend/ingestion/minjust_current.py` stages one document into a separate
|
|
generation, at no more than one request per second, with a 16 MiB response
|
|
limit, a 4 GiB text budget per document, contract-bound checkpoints, and
|
|
atomic publication. Staging requires a private 0700 directory owned by the
|
|
current user and rejects symlinks under checkpoints/document paths; legacy
|
|
corpus exclusions are anchored to the repository root. The corpus verifier
|
|
continues to validate unchanged normalized manifests from versions 1.0.0
|
|
and 1.1.0. The earlier non-exact pass is quarantined at
|
|
`/tmp/akyldash-current-code6.ZAMC37u4ex/qualification.json`: its 240 language
|
|
responses (304,493,520 bytes) remain unverified and are not reusable. The
|
|
retained work-tree SHA-256 is
|
|
`1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a`.
|
|
- Exact-only code-6 staging is complete at
|
|
`/tmp/akyldash-current-code6-exact.L4k3AvnOlP`: 201 editions and 402
|
|
complete language checkpoints, 604,377,885 raw bytes. The raw manifest
|
|
passes `quick_check`; its document checksum matches the staged directory at
|
|
`81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea`.
|
|
Separate normalization at
|
|
`/tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalized` contains 201
|
|
editions in official response order, passes `quick_check`, and records
|
|
normalizer `1.1.0` (2,048,797,024 bytes). The prior `data/minjust-cbd` and
|
|
`data/minjust-normalized` trees and production were not used as input or
|
|
modified. The raw staging manifest still records attachments and embedded
|
|
media as unverified; this single document does not establish corpus
|
|
completeness or production readiness.
|
|
|
|
## Checks not completed
|
|
|
|
- Docker Engine version from the server API, container and volume inventory,
|
|
protected env/secret metadata, sudoers, and detailed UFW rules require root.
|
|
`infra/bootstrap/audit-host.sh` is provided for an owner-run read-only audit.
|
|
- The backup archive/restore tool was tested with synthetic local data; a backup
|
|
and restore of real production data remain unverified because production is
|
|
not deployed.
|
|
- Cloudflare edge response was checked; origin DNS target, origin certificate,
|
|
Full (strict), registry publishing, host-wide runner metrics, and production
|
|
deployment have not been verified. The
|
|
owner-reported Idle state has not been independently checked through runner
|
|
inventory because the API returned 403.
|
|
- This audit made no host changes. No package or service was installed by this
|
|
audit; the owner performed the initial bootstrap beforehand.
|
|
|
|
## Next owner actions
|
|
|
|
1. Install the approved root-owned checkout after its PR is reviewed, then run
|
|
`sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh` and share
|
|
the output. The script prints no secret values and does not mutate the host.
|
|
2. After reviewing the audit output and confirming the target files are absent,
|
|
run
|
|
`sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh`;
|
|
it refuses to overwrite existing files. Set the approved `DATA_ROOT` and
|
|
place the approved corpus/index and Cloudflare origin configuration. Do not
|
|
deploy until data/service state and the required backup path are reviewed.
|
|
3. Run `sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh` after
|
|
reviewing the checkout; it grants only fixed verification, Compose
|
|
`config`/`status`, and guarded rollback commands.
|
|
4. After merging the reviewed PR, confirm a subsequent CI and smoke run on
|
|
`akyldash-ci`. Image publishing and production deployment remain manual and
|
|
are not part of CI.
|
|
5. Test backup/restore and rollback only with synthetic data in an isolated
|
|
environment before any production deployment.
|
|
|
|
## Change in this PR
|
|
|
|
Adds a read-only, resumable full-file corpus inventory and normalized-data
|
|
cross-check, a source/staging comparator requiring nonzero documents and
|
|
fragments plus held shared artifact locks, synthetic regression tests, and an
|
|
operator handoff runbook. CI's
|
|
existing `infra/deploy` unittest discovery includes the new tests. No workflow,
|
|
runner, VPS, production data, or Docker settings are changed. The first real
|
|
corpus scan stopped on an integer-storage bug; full validation with the
|
|
corrected verifier is still required. The source API's code `6` outcome remains
|
|
unconfirmed.
|