7.3 KiB
Corpus inventory and production handoff
The repository contains a read-only verifier for the ignored local corpus at
data/minjust-cbd/ and data/minjust-normalized/. It does not download,
normalize, copy, index, or modify source files. Keep inventory artifacts outside
the repository and both source trees.
Run the inventory
From the repository root, create a private artifact directory and run:
artifact_dir=$(mktemp -d /tmp/akyldash-corpus-audit.XXXXXXXX)
python3 infra/deploy/verify_corpus.py \
--raw-root data/minjust-cbd \
--normalized-root data/minjust-normalized \
--output-dir "$artifact_dir"
The tool writes result.json, inventory.sqlite3, and a persistent lock file
into that directory. Keep the directory private: the inventory contains source
paths and checksums. A second run against the same artifact directory is
rejected while a scan is active. Exit
status 0 means the scan and structural checks completed without errors; 1
means findings were recorded or the run is incomplete; 2 means the arguments
or artifact path were invalid. Read result.json to distinguish those states.
Warnings, including documents with zero text fragments, are reported
separately and do not turn the scan into a failure.
For an interrupted run, use the same artifact directory and add --resume.
Resume rehashes every source file instead of trusting cached timestamps, then
reports added, changed, and removed paths relative to entries already stored
in the inventory. After resuming an interrupted first pass, those counters are
relative to the partial saved inventory; the full rehash and final checks still
cover every path. A successful result therefore describes the latest complete
pass. Run against a quiescent corpus; do not normalize, fetch, or copy files
into either source tree during the scan. Do not point --output-dir inside
either source tree or through a symlink. The verifier rejects such paths before
creating or changing them; it also flags symlinks, reparse points, hard links,
unsupported file types, and files that change while being read.
The inventory database schema is versioned. Resume requires the same root
metadata and verifier SHA-256 recorded in result.json; start a new artifact
directory after changing the verifier. entries counts files plus directories;
files and directories are separate counts. Document-prefix checks use
indexed path ranges, and stale-path reporting streams rows from SQLite.
The audit checks SQLite integrity, manifest/document directory agreement, normalized schema and normalizer versions, source inventory checksums, fragment identities and text hashes, required output files, and text assembled from fragments. It does not establish the legal authority, completeness, or semantic correctness of MinJust content.
Compare source and staging before transfer
Run a complete inventory of the approved normalized source and the copied
staging tree into separate private artifact directories, using
--normalized-root for each. Require status: complete and
verification_status: passed in both results. The comparison command also
requires each manifest to contain at least one document and each normalized
tree to contain at least one fragment; an empty source or staging tree is a
failure.
python3 infra/deploy/compare_corpus_inventories.py \
"$source_artifacts/result.json" "$staging_artifacts/result.json"
The checker streams every relative path, size, and SHA-256 from both SQLite
inventories, and requires the same verifier/schema version and different
normalized roots. It opens both existing private .inventory.lock files
read-only and holds shared nonblocking locks through comparison; a missing,
unsafe, or busy lock fails closed. A zero exit status confirms a non-empty
staging tree matches the source. Keep both trees quiescent during their scans
and comparison. This is an equality check only; the owner still reviews the
code 6 disposition and approves the transfer destination.
Current data disposition
As measured on 2026-10-18, both local manifests contain 209,958 documents and
pass SQLite quick_check. All normalized manifest entries are success at
schema 1 and normalizer 1.0.0. The raw manifest records one fetch error for
code 6, which is absent from its documents table; this does not prove that
the source document is absent. On 2026-10-18, bounded calls through the
repository's rate-limited MinJust client received HTTP 404 for
GetDocumentListByQuery, GetDocument(6), control GetDocument(1096), and
CheckAvailable. The listing call failed before returning a code list, and
both document calls failed identically. The repeated 404 is consistent with
the OpenData endpoint being unavailable or its route having changed, but does
not identify which. It is not evidence that code 6 is absent. The public
document listing page was reachable,
but it does not confirm this numeric API code. Do not describe the corpus as
fully verified or silently waive this error. Recheck the official listing and
API when the endpoint responds, then record an explicit owner disposition.
From the VPS, a metadata-only HTTPS check to CheckAvailable failed at the
HTTP/2 transport layer (curl exit 92, INTERNAL_ERROR, HTTP status 000,
TLS verification 0). An HTTP/1.1 retry was not completed.
The first full pass (run b4a4556e30cc464d90bc8d75ef1673e9) stopped before
validation with Python int too large to convert to SQLite INTEGER after
2,120,420 raw entries and 397,856 normalized entries. This was an inventory
metadata storage failure, not a corpus finding. The verifier now stores file
identity values as text and records its own SHA-256. A complete rehash with the
updated verifier is required; do not treat the failed pass or progress output
as a completed audit.
A fresh full rehash is now running in /tmp/akyldash-corpus-audit-final.gtsD87vc,
run f4dd5e53020f442bbbecee2d0e4986a6, verifier SHA-256
83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5. Check
its private result.json for final counters, findings, and manifest comparison.
Production transfer and indexing gates
Production deployment is not enabled. Before any transfer, the owner must
review the root-only host audit, confirm the target data location is empty or
backed up, resolve the raw code 6 finding, and approve the exact normalized
corpus to use. Transfer only minjust-normalized/; keep the raw source archive
offline. Do not preserve the source's Windows/WSL file permission bits. Copy
to a new staging location with owner-approved permissions, verify the copied
manifest and files there, and only then consider placing it under DATA_ROOT.
Never overwrite an existing corpus or OpenSearch volume.
The current bulk importer checks individual Bulk API item results, but it can
switch an alias after a zero-fragment input and does not enforce a refresh,
count comparison, or representative-query check before that switch. Until a
separate change adds those gates, do not pass --alias during production
loading. Load into a new versioned index only after the corpus transfer is
verified; require a nonzero fragment count, successful item results, an
explicit refresh, a matching OpenSearch document count, and representative
Russian/Kyrgyz queries before an owner performs an alias change. Take and
verify the required backup before any production index or service change.