10 KiB
Corpus inventory and production handoff
The repository contains a read-only verifier for the ignored local corpus at
data/minjust-cbd/ and data/minjust-normalized/. It does not download,
normalize, copy, index, or modify source files. Keep inventory artifacts outside
the repository and both source trees.
Run the inventory
From the repository root, create a private artifact directory and run:
artifact_dir=$(mktemp -d /tmp/akyldash-corpus-audit.XXXXXXXX)
python3 infra/deploy/verify_corpus.py \
--raw-root data/minjust-cbd \
--normalized-root data/minjust-normalized \
--output-dir "$artifact_dir"
The tool writes result.json, inventory.sqlite3, and a persistent lock file
into that directory. Keep the directory private: the inventory contains source
paths and checksums. A second run against the same artifact directory is
rejected while a scan is active. Exit
status 0 means the scan and structural checks completed without errors; 1
means findings were recorded or the run is incomplete; 2 means the arguments
or artifact path were invalid. Read result.json to distinguish those states.
Warnings, including documents with zero text fragments, are reported
separately and do not turn the scan into a failure.
For an interrupted run, use the same artifact directory and add --resume.
Resume rehashes every source file instead of trusting cached timestamps, then
reports added, changed, and removed paths relative to entries already stored
in the inventory. After resuming an interrupted first pass, those counters are
relative to the partial saved inventory; the full rehash and final checks still
cover every path. A successful result therefore describes the latest complete
pass. Run against a quiescent corpus; do not normalize, fetch, or copy files
into either source tree during the scan. Do not point --output-dir inside
either source tree or through a symlink. The verifier rejects such paths before
creating or changing them; it also flags symlinks, reparse points, hard links,
unsupported file types, and files that change while being read.
The inventory database schema is versioned. Resume requires the same root
metadata and verifier SHA-256 recorded in result.json; start a new artifact
directory after changing the verifier. entries counts files plus directories;
files and directories are separate counts. Document-prefix checks use
indexed path ranges, and stale-path reporting streams rows from SQLite.
The audit checks SQLite integrity, manifest/document directory agreement, normalized schema and normalizer versions, source inventory checksums, fragment identities and text hashes, required output files, and text assembled from fragments. It does not establish the legal authority, completeness, or semantic correctness of MinJust content.
Compare source and staging before transfer
Run a complete inventory of the approved normalized source and the copied
staging tree into separate private artifact directories, using
--normalized-root for each. Require status: complete and
verification_status: passed in both results. The comparison command also
requires each manifest to contain at least one document and each normalized
tree to contain at least one fragment; an empty source or staging tree is a
failure.
python3 infra/deploy/compare_corpus_inventories.py \
"$source_artifacts/result.json" "$staging_artifacts/result.json"
The checker streams every relative path, size, and SHA-256 from both SQLite
inventories, and requires the same verifier/schema version and different
normalized roots. It opens both existing private .inventory.lock files
read-only and holds shared nonblocking locks through comparison; a missing,
unsafe, or busy lock fails closed. A zero exit status confirms a non-empty
staging tree matches the source. Keep both trees quiescent during their scans
and comparison. This is an equality check only; the owner still reviews the
code 6 disposition and approves the transfer destination.
Current data disposition
As measured on 2026-10-18, both local manifests contain 209,958 documents and
pass SQLite quick_check. All normalized manifest entries are success at
schema 1 and normalizer 1.0.0. The raw manifest records one fetch error for
code 6, which is absent from its documents table; this does not prove that
the source document is absent. On 2026-10-18, bounded calls through the
repository's rate-limited MinJust client received HTTP 404 for
GetDocumentListByQuery, GetDocument(6), control GetDocument(1096), and
CheckAvailable. The listing call failed before returning a code list, and
both document calls failed identically. The repeated 404 is consistent with
the OpenData endpoint being unavailable or its route having changed, but does
not identify which. It is not evidence that code 6 is absent. The public
document listing page was reachable,
but it does not confirm this numeric API code. Do not describe the corpus as
fully verified or silently waive this error. Recheck the official listing and
API when the endpoint responds, then record an explicit owner disposition.
From the VPS, a metadata-only HTTPS check to CheckAvailable failed at the
HTTP/2 transport layer (curl exit 92, INTERNAL_ERROR, HTTP status 000,
TLS verification 0). An HTTP/1.1 retry was not completed.
Current API and bounded code 6 recovery (2026-10-20)
The current official site uses POST /api/v1/GetDocuments (page size 12),
GET /api/v1/GetDocument?DocumentCode=..., and
GET /api/v1/GetEdition?editionId=...&lang=ru|kg. A bounded first-page probe
reported 210,639 records; this recovery does not enumerate the full list. The
official alias 1-4 resolves to canonical code 205064. Code 6 is present
and has 201 editions. The edition order in GetDocument is authoritative;
numeric IDs do not define currentness.
For lang=kg, the API returns Kyrgyz HTML in contentRu and contentKg=null.
The bounded adapter maps that response by the requested language to archive
ky, preserves the response and checkpoints, limits requests to one per
second, caps JSON responses at 16 MiB, and caps one document's text at 4 GiB.
It writes only to a separate generation under /tmp; it neither resumes nor
overwrites the existing ignored corpus.
Without exact=true, a request for 920292 returned response field
id=920492 despite different text length/hash. Two exact probes for 920292
and 920492 returned matching IDs and names/dates (04.08.1998 and
13.04.2017) matching GetDocument; their RU text lengths/hashes were
485,487 / 2b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718
and 1,139,908 / e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8.
The adapter now requires exact=true, returned-ID equality, and matching RU/KY
names before a language checkpoint can complete. Its versioned request
contract is part of both staging metadata and the SQLite resume identity;
pre-contract generations cannot be resumed.
The adapter does not independently validate attachments or embedded media.
Even after the text stage completes, the raw manifest records this as an
unresolved error. Do not merge this single document into the existing corpus,
claim corpus completeness, or transfer it to production without a separate
review and comparison. The prior no-exact pass at
/tmp/akyldash-current-code6.ZAMC37u4ex is marked quarantined in
qualification.json; its 240 text checkpoints and 304,493,520 bytes are
retained as unverified, with no document directory published. Its preserved
work-tree SHA-256 is
1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a. It must
not be reused or normalized.
The exact-only generation at /tmp/akyldash-current-code6-exact.L4k3AvnOlP
contains 201 editions and 402 complete RU/KY text checkpoints (604,377,885
raw bytes). The raw manifest passes quick_check; its stored source SHA-256
matches the staged document tree at
81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea. Separate
normalization at /tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalized
passes quick_check, records normalizer 1.1.0, and preserves first/last
edition order 920292 (04.08.1998) to 920492 (13.04.2017). The normalized
generation is about 2.05 GB. Neither temporary generation has been merged into
the existing corpus. Attachments and embedded media remain unverified.
The first full pass (run b4a4556e30cc464d90bc8d75ef1673e9) stopped before
validation with Python int too large to convert to SQLite INTEGER after
2,120,420 raw entries and 397,856 normalized entries. This was an inventory
metadata storage failure, not a corpus finding. The verifier now stores file
identity values as text and records its own SHA-256. A complete rehash with the
updated verifier is required; do not treat the failed pass or progress output
as a completed audit.
A fresh full rehash is now running in /tmp/akyldash-corpus-audit-final.gtsD87vc,
run f4dd5e53020f442bbbecee2d0e4986a6, verifier SHA-256
83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5. Check
its private result.json for final counters, findings, and manifest comparison.
Production transfer and indexing gates
Production deployment is not enabled. Before any transfer, the owner must
review the root-only host audit, confirm the target data location is empty or
backed up, resolve the raw code 6 finding, and approve the exact normalized
corpus to use. Transfer only minjust-normalized/; keep the raw source archive
offline. Do not preserve the source's Windows/WSL file permission bits. Copy
to a new staging location with owner-approved permissions, verify the copied
manifest and files there, and only then consider placing it under DATA_ROOT.
Never overwrite an existing corpus or OpenSearch volume.
The current bulk importer checks individual Bulk API item results, but it can
switch an alias after a zero-fragment input and does not enforce a refresh,
count comparison, or representative-query check before that switch. Until a
separate change adds those gates, do not pass --alias during production
loading. Load into a new versioned index only after the corpus transfer is
verified; require a nonzero fragment count, successful item results, an
explicit refresh, a matching OpenSearch document count, and representative
Russian/Kyrgyz queries before an owner performs an alias change. Take and
verify the required backup before any production index or service change.