12 KiB
Corpus inventory and production handoff
The repository contains a read-only verifier for the ignored local corpus at
data/minjust-cbd/ and data/minjust-normalized/. It does not download,
normalize, copy, index, or modify source files. Keep inventory artifacts outside
the repository and both source trees.
Run the inventory
From the repository root, create a private artifact directory and run:
artifact_dir=$(mktemp -d /tmp/akyldash-corpus-audit.XXXXXXXX)
python3 infra/deploy/verify_corpus.py \
--raw-root data/minjust-cbd \
--normalized-root data/minjust-normalized \
--output-dir "$artifact_dir"
The tool writes result.json, inventory.sqlite3, and a persistent lock file
into that directory. Keep the directory private: the inventory contains source
paths and checksums. A second run against the same artifact directory is
rejected while a scan is active. Exit
status 0 means the scan and structural checks completed without errors; 1
means findings were recorded or the run is incomplete; 2 means the arguments
or artifact path were invalid. Read result.json to distinguish those states.
Warnings, including documents with zero text fragments, are reported
separately and do not turn the scan into a failure.
For an interrupted run, use the same artifact directory and add --resume.
Resume rehashes every source file instead of trusting cached timestamps, then
reports added, changed, and removed paths relative to entries already stored
in the inventory. After resuming an interrupted first pass, those counters are
relative to the partial saved inventory; the full rehash and final checks still
cover every path. A successful result therefore describes the latest complete
pass. Run against a quiescent corpus; do not normalize, fetch, or copy files
into either source tree during the scan. Do not point --output-dir inside
either source tree or through a symlink. The verifier rejects such paths before
creating or changing them; it also flags symlinks, reparse points, hard links,
unsupported file types, and files that change while being read.
The inventory database schema is versioned. Resume requires the same root
metadata and verifier SHA-256 recorded in result.json; start a new artifact
directory after changing the verifier. entries counts files plus directories;
files and directories are separate counts. Document-prefix checks use
indexed path ranges, and stale-path reporting streams rows from SQLite.
The audit checks SQLite integrity, manifest/document directory agreement, normalized schema and normalizer versions, source inventory checksums, fragment identities and text hashes, required output files, and text assembled from fragments. It does not establish the legal authority, completeness, or semantic correctness of MinJust content.
Compare source and staging before transfer
Run a complete inventory of the approved normalized source and the copied
staging tree into separate private artifact directories, using
--normalized-root for each. Require status: complete and
verification_status: passed in both results. The comparison command also
requires each manifest to contain at least one document and each normalized
tree to contain at least one fragment; an empty source or staging tree is a
failure.
python3 infra/deploy/compare_corpus_inventories.py \
"$source_artifacts/result.json" "$staging_artifacts/result.json"
The checker streams every relative path, size, and SHA-256 from both SQLite
inventories, and requires the same verifier/schema version and different
normalized roots. It opens both existing private .inventory.lock files
read-only and holds shared nonblocking locks through comparison; a missing,
unsafe, or busy lock fails closed. A zero exit status confirms a non-empty
staging tree matches the source. Keep both trees quiescent during their scans
and comparison. This is an equality check only; the owner still reviews the
code 6 disposition and approves the transfer destination.
Current data disposition
As measured on 2026-10-18, both local manifests contain 209,958 documents and
pass SQLite quick_check. All normalized manifest entries are success at
schema 1 and normalizer 1.0.0. The raw manifest records one fetch error for
code 6, which is absent from its documents table; this does not prove that
the source document is absent. On 2026-10-18, bounded calls through the
repository's rate-limited MinJust client received HTTP 404 for
GetDocumentListByQuery, GetDocument(6), control GetDocument(1096), and
CheckAvailable. The listing call failed before returning a code list, and
both document calls failed identically. The repeated 404 is consistent with
the OpenData endpoint being unavailable or its route having changed, but does
not identify which. It is not evidence that code 6 is absent. The public
document listing page was reachable,
but it does not confirm this numeric API code. Do not describe the corpus as
fully verified or silently waive this error. Recheck the official listing and
API when the endpoint responds, then record an explicit owner disposition.
From the VPS, a metadata-only HTTPS check to CheckAvailable failed at the
HTTP/2 transport layer (curl exit 92, INTERNAL_ERROR, HTTP status 000,
TLS verification 0). An HTTP/1.1 retry was not completed.
Current API and bounded code 6 recovery (2026-10-20)
The current official site uses POST /api/v1/GetDocuments (page size 12),
GET /api/v1/GetDocument?DocumentCode=..., and
GET /api/v1/GetEdition?editionId=...&lang=ru|kg. A bounded first-page probe
reported 210,639 records; this recovery does not enumerate the full list. The
official alias 1-4 resolves to canonical code 205064. Code 6 is present
and has 201 editions. The edition order in GetDocument is authoritative;
numeric IDs do not define currentness.
For lang=kg, the API returns Kyrgyz HTML in contentRu and contentKg=null.
The bounded adapter maps that response by the requested language to archive
ky, preserves the response and checkpoints, limits requests to one per
second, caps JSON responses at 16 MiB, and caps one document's text at 4 GiB.
It writes only to a separate generation under /tmp; it neither resumes nor
overwrites the existing ignored corpus.
Without exact=true, a request for 920292 returned response field
id=920492 despite different text length/hash. Two exact probes for 920292
and 920492 returned matching IDs and names/dates (04.08.1998 and
13.04.2017) matching GetDocument; their RU text lengths/hashes were
485,487 / 2b51c2ff4e9b91cbe2a7ed052a5a9ed9150bda993fc15c120ac948580926b718
and 1,139,908 / e5f0fdbb47766e5a5dbb971183d765e4c1dcaf9c969a218a84206244ce9132e8.
The adapter now requires exact=true, returned-ID equality, and matching RU/KY
names before a language checkpoint can complete. Its versioned request
contract is part of both staging metadata and the SQLite resume identity;
pre-contract generations cannot be resumed.
The adapter does not independently validate attachments or embedded media.
Even after the text stage completes, the raw manifest records this as an
unresolved error. Do not merge this single document into the existing corpus,
claim corpus completeness, or transfer it to production without a separate
review and comparison. The prior no-exact pass at
/tmp/akyldash-current-code6.ZAMC37u4ex is marked quarantined in
qualification.json; its 240 text checkpoints and 304,493,520 bytes are
retained as unverified, with no document directory published. Its preserved
work-tree SHA-256 is
1e874c1d7fab1e00e7e45ff1b16422ea474e22ee13c901ac9a0612f47aa5d24a. It must
not be reused or normalized.
The exact-only generation at /tmp/akyldash-current-code6-exact.L4k3AvnOlP
contains 201 editions and 402 complete RU/KY text checkpoints (604,377,885
raw bytes). The raw manifest passes quick_check; its stored source SHA-256
matches the staged document tree at
81d8e1a53b2ebef61f08b56d04b3f0a80077d587a4fadae0152bb4b7f13055ea. Separate
normalization at /tmp/akyldash-current-code6-exact.L4k3AvnOlP-normalized
passes quick_check, records normalizer 1.1.0, and preserves first/last
edition order 920292 (04.08.1998) to 920492 (13.04.2017). The normalized
generation is about 2.05 GB. Neither temporary generation has been merged into
the existing corpus. Attachments and embedded media remain unverified.
The first full pass (run b4a4556e30cc464d90bc8d75ef1673e9) stopped before
validation with Python int too large to convert to SQLite INTEGER after
2,120,420 raw entries and 397,856 normalized entries. This was an inventory
metadata storage failure, not a corpus finding. The verifier now stores file
identity values as text and records its own SHA-256. A complete rehash with the
updated verifier is required; do not treat the failed pass or progress output
as a completed audit.
A fresh full rehash is now running in /tmp/akyldash-corpus-audit-final.gtsD87vc,
run f4dd5e53020f442bbbecee2d0e4986a6, verifier SHA-256
83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5. Check
its private result.json for final counters, findings, and manifest comparison.
Production transfer and indexing gates
Production deployment is not enabled. Before any transfer, the owner must
review the root-only host audit, confirm the target data location is empty or
backed up, resolve the raw code 6 finding, and approve the exact normalized
corpus to use. Transfer only minjust-normalized/; keep the raw source archive
offline. Do not preserve the source's Windows/WSL file permission bits. Copy
to a new staging location with owner-approved permissions, verify the copied
manifest and files there, and only then consider publishing it under
DATA_ROOT/minjust-normalized.
Atomic publication gate for DATA_ROOT
The verified staging tree must be published to
DATA_ROOT/minjust-normalized as one all-or-nothing operation, using an
owner-approved atomic mechanism or a separately documented equivalent. An
in-place copy into the live target is not an acceptable publication method.
Preserve the current published corpus until the candidate is fully verified and
ready for the atomic switch; never overwrite its contents in place.
The publication mechanism and its interrupted-operation recovery have not yet
been specified or tested; this is a release gate, not a claim that atomic corpus
publication is implemented. Until that gate is completed, do not publish or
replace a corpus in DATA_ROOT.
If preparation is interrupted before publication, keep the current target untouched. Resume only from the isolated staging tree after a full rehash and source comparison; if that cannot be established, fail closed and prepare a new staging candidate. If interruption makes the outcome of the final publication ambiguous, do not retry or replace anything automatically. Verify the target against the approved manifest and resolve the state with the owner before any retry. Never overwrite an existing corpus or OpenSearch volume.
The atomic filesystem publication of the approved corpus into DATA_ROOT and
the OpenSearch alias switch are separate gates. A successful corpus publish
does not authorize indexing or alias changes; do not conflate the two.
The bulk importer now refuses alias publication until all Bulk item results
are successful, the full staged index is refreshed, its fragment count exactly
matches the complete normalized inventory (including on resume), and approved
Russian and Kyrgyz canaries both return text. The alias must currently target
the exact --expected-old-index; wildcard removals and implicit first-time
alias creation are rejected. OpenSearch performs removal of that exact target
and addition of the candidate in one alias request. A mismatch or validation
failure leaves the existing alias untouched. This is a code-level gate only:
no production index or corpus was changed, and the live inventory and canary
terms still require owner verification. See backend/README.md for the
operator flags and use a new versioned index for every candidate.