Compare commits

..
Author SHA1 Message Date
agent 2d3b34f34b Merge remote-tracking branch 'origin/main' into fix/rollback-image-digest-reference
CI / tests-and-images (pull_request) Successful in 2m10s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
# Conflicts:
#	CHANGELOG.md
#	docs/operations/CURRENT_STATE.md
2026-10-20 00:35:46 +03:00
agent cc8cc1bce6 Merge pull request 'Add offline release manifest contract' (#49) from feature/release-manifest-contract into main
CI / tests-and-images (push) Successful in 2m8s
Runner smoke test / runner-smoke (push) Successful in 1s
2026-10-19 21:29:33 +00:00
agent a68d3c315e docs: record rollback preflight limitation
CI / tests-and-images (pull_request) Successful in 2m5s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-20 00:27:44 +03:00
agent ba669fad34 Merge remote-tracking branch 'origin/main' into feature/release-manifest-contract
CI / tests-and-images (pull_request) Successful in 2m5s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
# Conflicts:
#	CHANGELOG.md
2026-10-20 00:18:34 +03:00
codex 8f64131131 Merge remote-tracking branch 'origin/main' into fix/rollback-image-digest-reference
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-20 00:15:33 +03:00
codex 802bfd117e test: assert digest rollback fails before side effects
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 3s
2026-10-20 00:13:55 +03:00
agent 2281705c95 Merge pull request 'Make corpus inventory scalable and resumable' (#48) from feature/resumable-corpus-inventory into main
CI / tests-and-images (push) Successful in 2m9s
Runner smoke test / runner-smoke (push) Successful in 1s
2026-10-19 21:10:48 +00:00
codex fc0473e7e2 fix: validate digest-pinned rollback images safely 2026-10-20 00:10:25 +03:00
agent e12a8a919c Document release manifest preparation scope
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 22:07:45 +03:00
agent 37050b1458 Add validated release manifest contract
CI / tests-and-images (pull_request) Successful in 2m7s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 22:02:09 +03:00
agent 6bd4b05af2 Document corpus inventory safety gates
CI / tests-and-images (pull_request) Successful in 2m30s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:39:34 +03:00
agent 0da2bcd82d Lock corpus artifacts during comparison
CI / tests-and-images (pull_request) Successful in 2m31s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:33:52 +03:00
agent 0979b8f544 Reject empty corpus staging comparisons
CI / tests-and-images (pull_request) Successful in 2m33s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:28:27 +03:00
agent 4d106903e1 Fix corpus inventory indexing and provenance
CI / tests-and-images (pull_request) Successful in 2m25s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:22:00 +03:00
agent 4b2e5578cd Add resumable corpus inventory verification 2026-10-18 15:29:14 +03:00
agent 0e276ae652 Merge pull request 'ci: isolate image builds on rootless runner' (#47) from feature/autonomous-review-workflow into main
CI / tests-and-images (push) Successful in 2m5s
Runner smoke test / runner-smoke (push) Successful in 2s
2026-10-18 05:58:19 +00:00
codex 554a7ab5ff docs: qualify runner resource verification
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 08:55:16 +03:00
codex cf99f4532f docs: record successful isolated CI build
CI / tests-and-images (pull_request) Successful in 2m4s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:51:21 +03:00
codex 5d72bb092d fix: pass runner workspace to container actions
CI / tests-and-images (pull_request) Successful in 2m26s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:46:05 +03:00
codex d96fc4d36b ci: set container action entrypoints
CI / tests-and-images (pull_request) Failing after 9s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:37:44 +03:00
codex 436806a834 ci: exercise root guards under unprivileged uid
CI / tests-and-images (pull_request) Failing after 16s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:32:48 +03:00
codex 3a89dc2196 ci: build production images with isolated Kaniko
CI / tests-and-images (pull_request) Failing after 15s
Runner smoke test / runner-smoke (pull_request) Successful in 8s
2026-10-18 08:17:57 +03:00
agent 69c62466ca docs: complete autonomous review gates
Runner smoke test / runner-smoke (pull_request) Successful in 2s
CI / tests-and-images (pull_request) Has been cancelled
2026-10-17 23:13:07 +03:00
agent 36892aef77 docs: define autonomous review workflow
Runner smoke test / runner-smoke (pull_request) Successful in 1s
CI / tests-and-images (pull_request) Has been cancelled
2026-10-17 23:07:12 +03:00
admin ac611cc878 Merge pull request 'Fix rootless Docker HOME for runner' (#45) from fix/rootless-docker-home into main
Runner smoke test / runner-smoke (push) Successful in 3s
CI / tests-and-images (push) Has been cancelled
Reviewed-on: #45
Reviewed-by: agent <agent@vakanaut.ru>
2026-10-14 14:45:57 +00:00
27 changed files with 2738 additions and 103 deletions

No files matched your search

+1
View File
@@ -0,0 +1 @@
FROM caddy:2.11.6-alpine@sha256:881bbc60f9986d5ab8e7cfd6cf7e4ef3c9c0439fef2429d035d065577882f028
+16
View File
@@ -0,0 +1,16 @@
name: Validate Caddyfile
description: Run the pinned Caddy validator against the checked-out Caddyfile
inputs:
workspace:
description: Repository workspace path supplied by the workflow
required: true
runs:
using: docker
image: Dockerfile
entrypoint: /usr/bin/caddy
args:
- validate
- --config
- ${{ inputs.workspace }}/deploy/production/Caddyfile
- --adapter
- caddyfile
+1
View File
@@ -0,0 +1 @@
FROM ghcr.io/osscontainertools/kaniko:v1.28.5-debug@sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412
+22
View File
@@ -0,0 +1,22 @@
name: Build image with Kaniko in a rootless container
description: Validate a Dockerfile by building it without publishing the image or cache
inputs:
workspace:
description: Repository workspace path supplied by the workflow
required: true
dockerfile:
description: Dockerfile path relative to the checked-out repository
required: true
destination:
description: Image reference required by Kaniko; the image is not pushed
required: true
runs:
using: docker
image: Dockerfile
entrypoint: /kaniko/executor
args:
- --context=dir://${{ inputs.workspace }}
- --dockerfile=${{ inputs.workspace }}/${{ inputs.dockerfile }}
- --destination=${{ inputs.destination }}
- --no-push
- --no-push-cache
+26 -8
View File
@@ -7,7 +7,7 @@ on:
jobs:
tests-and-images:
runs-on: ubuntu-latest
runs-on: akyldash-ci
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
@@ -19,6 +19,8 @@ jobs:
run: python3 -m unittest discover -s tools/telegram-bot -p 'test_*.py' -v
- name: Run infrastructure backup tests
run: python3 -m unittest discover -s infra/backup -p 'test_*.py' -v
- name: Check root guard UID decisions
run: bash infra/bootstrap/test-root-guard.sh
- name: Check recovery host guard
run: bash infra/bootstrap/test-host-guard.sh
- name: Check verifier helper argument guard
@@ -34,26 +36,42 @@ jobs:
bash infra/runner/test-runner-config.sh
bash infra/runner/test-install-runner-idempotency.sh
bash infra/runner/test-install-runner-guard.sh
- name: Check CI container operation isolation
run: bash infra/runner/test-ci-container-builds.sh
- name: Check infrastructure shell syntax
run: |
bash -n infra/bootstrap/bootstrap-ubuntu-24.04.sh infra/bootstrap/host-guard.sh \
infra/bootstrap/verify.sh infra/bootstrap/verify-helper.sh infra/bootstrap/audit-host.sh \
infra/deploy/install-helper.sh infra/deploy/compose-helper.sh infra/deploy/test-compose-helper.sh \
infra/deploy/prepare-production-config.sh infra/deploy/test-production-config-guard.sh \
infra/bootstrap/test-audit-host.sh infra/bootstrap/test-verify-helper.sh \
infra/bootstrap/test-audit-host.sh infra/bootstrap/test-root-guard.sh \
infra/bootstrap/test-verify-helper.sh infra/test/run-as-unprivileged.sh \
infra/runner/install-runner.sh infra/runner/install-runner-lib.sh \
infra/runner/act-runner-healthcheck.sh infra/runner/test-runner-config.sh \
infra/runner/test-install-runner-idempotency.sh infra/runner/test-install-runner-guard.sh
infra/runner/test-install-runner-idempotency.sh infra/runner/test-install-runner-guard.sh \
infra/runner/test-ci-container-builds.sh
python3 -c "import ast,pathlib; ast.parse(pathlib.Path('infra/runner/secure-runner-image.py').read_text())"
- name: Run rollback helper tests
python3 -c "import ast,pathlib; ast.parse(pathlib.Path('infra/deploy/validate_release.py').read_text())"
- name: Run deployment contract and rollback tests
run: python3 -m unittest discover -s infra/deploy -p 'test_*.py' -v
- name: Validate production Compose and Caddyfile
run: |
secret_file=$(mktemp)
trap 'rm -f "$secret_file"' EXIT
REVIEW_SECRET_FILE="$secret_file" docker compose --env-file deploy/production/.env.example -f deploy/production/compose.yaml config --quiet
docker run --rm --entrypoint caddy -v "$PWD/deploy/production/Caddyfile:/etc/caddy/Caddyfile:ro" caddy:2.11.6-alpine@sha256:881bbc60f9986d5ab8e7cfd6cf7e4ef3c9c0439fef2429d035d065577882f028 validate --config /etc/caddy/Caddyfile --adapter caddyfile
- name: Validate Caddyfile in isolated container action
uses: ./.gitea/actions/caddy-validate
with:
workspace: ${{ github.workspace }}
- name: Build production images
run: |
docker build -f deploy/production/Dockerfile.backend -t akyldash-backend:ci .
docker build -f deploy/production/Dockerfile.opensearch -t akyldash-opensearch:ci .
uses: ./.gitea/actions/kaniko-build
with:
workspace: ${{ github.workspace }}
dockerfile: deploy/production/Dockerfile.backend
destination: akyldash-backend:ci
- name: Build OpenSearch production image
uses: ./.gitea/actions/kaniko-build
with:
workspace: ${{ github.workspace }}
dockerfile: deploy/production/Dockerfile.opensearch
destination: akyldash-opensearch:ci
+60 -1
View File
@@ -9,7 +9,9 @@ Python, SQLite и OpenSearch. Production пока не развернут.
[деплой](docs/operations/DEPLOYMENT.md), [backup](docs/operations/BACKUP.md),
[rollback](docs/operations/ROLLBACK.md), [восстановление](docs/operations/RECOVERY.md).
- Работай в feature/fix/refactor-ветке. `main` защищена; изменения вноси через
PR, не сливай его автоматически. Сохраняй незакоммиченные изменения.
PR. DEV не одобряет и не сливает собственный PR. Условный merge выполняет
только REVIEWER по правилам раздела «Автономный workflow PR» ниже. Сохраняй
незакоммиченные изменения.
- SSH alias — `akyldash_vps`; используй только read-only диагностику. Не давай
`codex` общий sudo или доступ к Docker socket. Root-действия требует владелец.
Для Gitea используй `gitea-api`, не читай файлы с учётными данными.
@@ -34,3 +36,60 @@ Python, SQLite и OpenSearch. Production пока не развернут.
для нового хоста.
- После каждого изменения инфраструктуры обновляй
`docs/operations/CURRENT_STATE.md` по фактическим проверкам.
## Автономный workflow PR
- Маршруты ниже указаны из контекста оркестратора/root: оркестратор — thread
`01a144a5-8b95-77c3-b311-9a9afd062ca8`, host `local`; DEV — thread
`01a138f1-3b34-7c91-a9ed-4b43c3c85f1f`, host
`remote-ssh-codex-managed:712b8d4e-24e1-4dd5-92af-85d1fbd26449`, Gitea
`codex` (ID 6); REVIEWER — thread `01a144f9-5b51-7923-8398-61d4a6a98306`,
host `remote-ssh-codex-managed:b61555d7-8d1a-4cfa-988b-c8da4791d426`, Gitea
`agent` (ID 7). Если этот же чат отображает REVIEWER как `local` внутри его
собственного app-server, это self-label и не является маршрутом из контекста
оркестратора.
- DEV принимает только явно зарегистрированную оркестратором человеческую
задачу с полями `task`, `repo`, `branch`, `pr`, `head_sha`, `phase`. Репозиторий
должен быть `admin/akyldash`; работай только в назначенной ветке и PR. Веди
одну задачу/PR за раз и не подхватывай старые PR самостоятельно.
- Допустимые фазы: `initial_review` — реализация и устранение review findings;
`final_review_and_merge` — только после первоначального успешного review,
changelog-коммита и отдельного назначения оркестратора. Не переходи к следующей
фазе без назначения. Если задача приостановлена, останови любые действия;
после resume перепроверь назначение, PR, head/base, checks и review с нуля.
- Повторная доставка одного назначения идемпотентна по ключу
`task`+`pr`+`head_sha`+`stage`: не дублируй commit, push, review-запрос или
merge; сообщи оркестратору о уже выполненном результате. Новый head или stage
образует новое назначение и требует повторной проверки.
- Межчатовое чтение недоступно: оркестратор через central relay сам читает
результаты и выдаёт назначения. Завершай каждый этап отчётом с полями `task`,
`repo`, `branch`, `pr`, `head_sha`, `stage`, `checks`, `findings`,
`next_action`. Не настраивай отдельные webhook, daemon или транспорт.
- DEV исследует, изменяет, проверяет, коммитит и отправляет изменения; REVIEWER
не меняет код и независимо проверяет полный diff от base после каждого push,
включая предыдущие findings. DEV не одобряет и не сливает свой PR.
- Только после первоначального успешного формального review DEV добавляет
значимые изменения в существующий changelog, коммитит и отправляет их в тот же
PR. Затем REVIEWER полностью проверяет итоговый diff. Для docs-only задачи не
придумывай изменения кода; правила версии применяй только если они явно
относятся к этой документационной правке.
- Слияние может выполнить только REVIEWER для зарегистрированной человеческой
задачи, в фазе `final_review_and_merge`, после явного назначения оркестратора.
Непосредственно перед ним REVIEWER подтверждает `admin/akyldash`, PR, base
`main`, ожидаемую ветку, автора PR `codex` (ID 6), неизменность base и head,
причём head должен совпадать с зарегистрированным `head_sha`, включая
`head_commit_id` для merge-запроса. Требуются все успешные required CI/test
checks и gates, отсутствие pending/failed/unknown checks, открытых
findings и активных `REQUEST_CHANGES`, плюс актуальный формальный `APPROVED`
от REVIEWER для точного текущего commit с параметрами `commit_id == head_sha`,
`stale == false`, `dismissed == false`. При изменении head или base review и
проверки выполняются заново. Комментарий с approval не является формальным
review.
Используй только обычный server-side merge с `head_commit_id`; не используй
`force_merge` и не обходи branch protection. Если любой существенный gate,
статус или защита неизвестны либо не проверяются — не сливай PR и сообщи
оркестратору, что именно осталось неизвестным.
- Все Gitea API-вызовы выполняй только через `gitea-api` с личным окружением;
Git используй с обычным credential helper. Не читай credentials из файлов
репозитория. Запускай Git- и API-команды отдельно, без объединения с другими
командами.
+15
View File
@@ -2,6 +2,21 @@
## Не выпущено
- Rollback preflight проверяет digest-pinned image references по сохранённому
image ID и останавливается до изменений deployment; переключение production
на digest и полноценный digest rollback пока не реализованы.
- Infrastructure: добавлен офлайн-контракт release manifest с точной схемой,
digest-ссылками и repository allowlist; проверен Compose override с
`--no-build` как требованием будущего deploy. Публикация, provenance и
production-деплой не включены.
- Добавлен read-only возобновляемый inventory корпуса с полным хешированием,
потоковой сверкой, индексируемыми диапазонами путей и provenance SHA-256
проверяющего; значения `stat` хранятся без ограничения SQLite INTEGER.
Сравнение source/staging сверяет путь, размер и SHA-256, удерживает shared
locks и отклоняет пустые корпуса без документов или фрагментов. Runbook и
`CURRENT_STATE.md` отражают, что полный аудит корпуса ещё выполняется; это
изменение не означает готовность корпуса или production.
- CI actions теперь используют workspace, переданный Gitea, для Caddyfile и Kaniko build context; CI run 69 и smoke run 70 прошли на self-hosted runner.
- Runner installer сохраняет `HOME` и XDG-пути в user-unit `docker.service`, перезапускает активный daemon и проверяет окружение его процесса.
- Runner installer защищает image и mountpoint root-owned home-каталогом, оставляет регистрацию и workspace в runner-owned каталоге, безопасно форматирует образ по открытому FD и проверяет `nosuid,nodev`.
- Backend `0.11.3`: ключ подписи поисковых снимков читается из Docker secret-файла, а не из аргументов процесса. Добавлена базовая инфраструктура VPS, Caddy HTTPS proxy с разделёнными Compose-сетями и CI-проверка конфигурации; публикация образов и production-деплой остаются выключены до проверки runner, registry, backup/restore и ограниченного deploy-helper.
+50 -13
View File
@@ -42,10 +42,29 @@ systemd timer checks the runner service and the rootless Docker API each minute;
on failure it restarts the fixed runner service. No UFW changes or public
metrics ports are needed. Runner cache is disabled.
The label `akyldash-ci` is used by `runner-smoke.yaml` on pull requests, pushes
to `main`, and manual dispatch. Existing CI image-build work remains on its current
`ubuntu-latest` target until it has been tested separately against the
isolated runner.
The CI workflow and `runner-smoke.yaml` target the repository runner label
`akyldash-ci`. CI keeps the backend, Telegram, backup/restore, deployment,
bootstrap, runner, Compose, and Caddy checks. Caddy validation runs in a pinned
container action with explicit `/usr/bin/caddy` entrypoint. The workflow passes
Gitea's `${{ github.workspace }}` value into each container action; Caddy and
Kaniko use that supplied path for repository files instead of assuming a
container workspace location. Both production
Dockerfiles are built in a separate pinned Kaniko container action with explicit
`/kaniko/executor` entrypoint using the runner's rootless container engine; the
job container receives no Docker socket. Kaniko uses `--no-push` and
`--no-push-cache`, so CI validates the full backend and OpenSearch builds but
does not publish images or export build output. The two builds run sequentially
within one job. CI run 69 completed both builds on the runner. Actual per-action
resource-limit enforcement and peak memory or disk usage were not independently
measured or verified. No registry credentials,
production secrets, elevated container options, or arbitrary host mounts are configured.
The Kaniko action uses the `linux/amd64` v1.28.5-debug manifest
`sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412`,
listed in the [official GitHub Packages entry](https://github.com/osscontainertools/kaniko/pkgs/container/kaniko/1271366984?tag=v1.28.5-debug).
The pinned Caddy image digest remains the same as the former `docker run`
validation. act_runner starts these container actions; they do not expose its
rootless Docker socket to workflow steps.
Production deployment is not automated.
## Installation and registration
@@ -136,9 +155,10 @@ that token; do not put it in command-line arguments or environment variables.
present in the job container.
3. Confirm the run reports the expected runner, succeeds, and the runner
service and health timer remain active.
4. Only after the repository owner reviews this boundary should CI jobs be moved
to `akyldash-ci`. Do not add a production deploy workflow or secrets to this
runner until a separate restricted deploy mechanism is reviewed.
4. Inspect the `CI` workflow run and confirm Caddy plus both Kaniko actions pass
on `akyldash-ci`. The Kaniko jobs are intentionally build-only: publishing
images requires a separately reviewed registry workflow and credentials.
Do not add production deploy workflows or secrets to this runner.
## Upgrade
@@ -170,9 +190,26 @@ automatic recovery step.
## Status
The root installer and registration are incomplete. The owner reports that the
15 GiB ext4 loop-backed mount is active on `/dev/loop0` after manually running
`systemctl daemon-reload`, `enable`, and `start` when the first installer run
stopped at mount activation. This PR has not changed the VPS and does not claim
that remaining runner components are installed, a repository token is
available, the runner is Online, or a workflow has run.
The owner reports that the repository-scoped runner is registered, appears
Idle in Gitea, and its rootless Docker daemon works; the report date was not
recorded. Gitea status checked 2026-10-18 showed smoke run 62 successful and CI
run 61 pending on head `69c62466`. That workflow requested `ubuntu-latest`,
while the smoke workflow used `akyldash-ci`; the label difference is a
plausible explanation for the pending status, but the runner inventory API
returned 403 and the scheduler's reason was not independently confirmed.
On head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run
64 succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
with `exec: "validate": executable file not found`; Kaniko was skipped and smoke
run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the Caddy action
looked for `/github/workspace/deploy/production/Caddyfile`, which was not
mounted at that fixed path; Kaniko was skipped and smoke run 68 succeeded. On
head `5d72bb09`, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile
and built the backend and OpenSearch images using `${{ github.workspace }}`.
Actual per-action resource-limit enforcement and peak usage were not
independently measured or verified. Image publishing, host-wide resource
metrics, and production deployment remain unverified. Root-guard tests
use `setpriv` for negative cases when jobs start as UID 0 and separately test
the host-audit guard with simulated UID 0 and UID 65534; the real host audit,
production config setup, and runner installer are never executed as root by
those tests. No VPS changes are part of this PR.
+131
View File
@@ -0,0 +1,131 @@
# Corpus inventory and production handoff
The repository contains a read-only verifier for the ignored local corpus at
`data/minjust-cbd/` and `data/minjust-normalized/`. It does not download,
normalize, copy, index, or modify source files. Keep inventory artifacts outside
the repository and both source trees.
## Run the inventory
From the repository root, create a private artifact directory and run:
```bash
artifact_dir=$(mktemp -d /tmp/akyldash-corpus-audit.XXXXXXXX)
python3 infra/deploy/verify_corpus.py \
--raw-root data/minjust-cbd \
--normalized-root data/minjust-normalized \
--output-dir "$artifact_dir"
```
The tool writes `result.json`, `inventory.sqlite3`, and a persistent lock file
into that directory. Keep the directory private: the inventory contains source
paths and checksums. A second run against the same artifact directory is
rejected while a scan is active. Exit
status `0` means the scan and structural checks completed without errors; `1`
means findings were recorded or the run is incomplete; `2` means the arguments
or artifact path were invalid. Read `result.json` to distinguish those states.
Warnings, including documents with zero text fragments, are reported
separately and do not turn the scan into a failure.
For an interrupted run, use the same artifact directory and add `--resume`.
Resume rehashes every source file instead of trusting cached timestamps, then
reports added, changed, and removed paths relative to entries already stored
in the inventory. After resuming an interrupted first pass, those counters are
relative to the partial saved inventory; the full rehash and final checks still
cover every path. A successful result therefore describes the latest complete
pass. Run against a quiescent corpus; do not normalize, fetch, or copy files
into either source tree during the scan. Do not point `--output-dir` inside
either source tree or through a symlink. The verifier rejects such paths before
creating or changing them; it also flags symlinks, reparse points, hard links,
unsupported file types, and files that change while being read.
The inventory database schema is versioned. Resume requires the same root
metadata and verifier SHA-256 recorded in `result.json`; start a new artifact
directory after changing the verifier. `entries` counts files plus directories;
`files` and `directories` are separate counts. Document-prefix checks use
indexed path ranges, and stale-path reporting streams rows from SQLite.
The audit checks SQLite integrity, manifest/document directory agreement,
normalized schema and normalizer versions, source inventory checksums,
fragment identities and text hashes, required output files, and text assembled
from fragments. It does not establish the legal authority, completeness, or
semantic correctness of MinJust content.
## Compare source and staging before transfer
Run a complete inventory of the approved normalized source and the copied
staging tree into separate private artifact directories, using
`--normalized-root` for each. Require `status: complete` and
`verification_status: passed` in both results. The comparison command also
requires each manifest to contain at least one document and each normalized
tree to contain at least one fragment; an empty source or staging tree is a
failure.
```bash
python3 infra/deploy/compare_corpus_inventories.py \
"$source_artifacts/result.json" "$staging_artifacts/result.json"
```
The checker streams every relative path, size, and SHA-256 from both SQLite
inventories, and requires the same verifier/schema version and different
normalized roots. It opens both existing private `.inventory.lock` files
read-only and holds shared nonblocking locks through comparison; a missing,
unsafe, or busy lock fails closed. A zero exit status confirms a non-empty
staging tree matches the source. Keep both trees quiescent during their scans
and comparison. This is an equality check only; the owner still reviews the
code `6` disposition and approves the transfer destination.
## Current data disposition
As measured on 2026-10-18, both local manifests contain 209,958 documents and
pass SQLite `quick_check`. All normalized manifest entries are `success` at
schema `1` and normalizer `1.0.0`. The raw manifest records one fetch error for
code `6`, which is absent from its `documents` table; this does not prove that
the source document is absent. On 2026-10-18, bounded calls through the
repository's rate-limited MinJust client received HTTP `404` for
`GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and
`CheckAvailable`. The listing call failed before returning a code list, and
both document calls failed identically. The repeated `404` is consistent with
the OpenData endpoint being unavailable or its route having changed, but does
not identify which. It is not evidence that code `6` is absent. The [public
document listing page](https://cbd.minjust.gov.kg/list-docs/kg) was reachable,
but it does not confirm this numeric API code. Do not describe the corpus as
fully verified or silently waive this error. Recheck the official listing and
API when the endpoint responds, then record an explicit owner disposition.
From the VPS, a metadata-only HTTPS check to `CheckAvailable` failed at the
HTTP/2 transport layer (`curl` exit 92, `INTERNAL_ERROR`, HTTP status `000`,
TLS verification `0`). An HTTP/1.1 retry was not completed.
The first full pass (run `b4a4556e30cc464d90bc8d75ef1673e9`) stopped before
validation with `Python int too large to convert to SQLite INTEGER` after
2,120,420 raw entries and 397,856 normalized entries. This was an inventory
metadata storage failure, not a corpus finding. The verifier now stores file
identity values as text and records its own SHA-256. A complete rehash with the
updated verifier is required; do not treat the failed pass or progress output
as a completed audit.
A fresh full rehash is now running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`,
run `f4dd5e53020f442bbbecee2d0e4986a6`, verifier SHA-256
`83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`. Check
its private `result.json` for final counters, findings, and manifest comparison.
## Production transfer and indexing gates
Production deployment is not enabled. Before any transfer, the owner must
review the root-only host audit, confirm the target data location is empty or
backed up, resolve the raw code `6` finding, and approve the exact normalized
corpus to use. Transfer only `minjust-normalized/`; keep the raw source archive
offline. Do not preserve the source's Windows/WSL file permission bits. Copy
to a new staging location with owner-approved permissions, verify the copied
manifest and files there, and only then consider placing it under `DATA_ROOT`.
Never overwrite an existing corpus or OpenSearch volume.
The current bulk importer checks individual Bulk API item results, but it can
switch an alias after a zero-fragment input and does not enforce a refresh,
count comparison, or representative-query check before that switch. Until a
separate change adds those gates, do not pass `--alias` during production
loading. Load into a new versioned index only after the corpus transfer is
verified; require a nonzero fragment count, successful item results, an
explicit refresh, a matching OpenSearch document count, and representative
Russian/Kyrgyz queries before an owner performs an alias change. Take and
verify the required backup before any production index or service change.
+112 -47
View File
@@ -1,25 +1,26 @@
# Infrastructure current state
Last audit: 2026-10-12. Source: read-only SSH session as `codex`, owner-reported
UFW state, local repository inspection, and Gitea API. Root-only Docker state,
UFW rule details, and protected file metadata remain unverified.
Read-only host recheck: 2026-10-18. Source: SSH as `codex`, local repository
inspection, and Gitea API. Root-only Docker state, UFW rule details, and
protected file metadata remain unverified.
## VPS facts
- Host: `avprogs.hlab.kz`, `206.223.249.118`, SSH port `22`, user `codex`.
- OS: Ubuntu Server `24.04.5 LTS`; kernel `6.8.0-142-generic` from the prior
inventory audit.
- Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB disk with 92 GiB free
at this audit.
- Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with
84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop
mount with 12 GiB free at that check.
- Network: interface `ens3`, IPv4 `206.223.249.118/32`.
- `codex` is UID/GID 1000, groups `codex` and `users`; `sudo -n true` requires
a password. Docker CLI cannot access `/var/run/docker.sock` as `codex`.
- SSH service is running. The inspected sshd configuration has public-key and
password authentication enabled and `PermitRootLogin yes`; no SSH changes
were made.
- Bootstrap completed. Docker CE/CLI `29.9.0`, Compose `5.6.0`, containerd
`2.4.1`, and runc `1.5.2` are installed; package versions and active Docker
service were independently checked. The Docker daemon API remains root-only.
- Bootstrap completed. Docker CLI `29.9.0` and Compose `5.6.0` were rechecked
over SSH on 2026-10-18; containerd `2.4.1` and runc `1.5.2` are from the
prior package audit. The Docker daemon API remains root-only.
- `/etc/docker/daemon.json` has the expected `local` log driver and `10m` / `3`
rotation limits. Bootstrap paths exist with root ownership: `/opt/akyldash`
mode `0755`, `/etc/akyldash` and `/var/backups/akyldash` mode `0700`, and
@@ -27,7 +28,10 @@ UFW rule details, and protected file metadata remain unverified.
- `/opt/akyldash/current`, verifier/rollback binaries, and the sudoers file are
absent. `codex` has not received sudo or Docker-group access. Metadata for
`/etc/akyldash/production.env` and `/etc/akyldash/review_secret` could not be
checked because `/etc/akyldash` is root-only.
checked because access is denied. The target
`/var/lib/akyldash/data/minjust-normalized` and its manifest metadata were
also denied; their presence, ownership, and contents are unknown. No sudo was
used.
- TCP listeners observed: SSH `22` and loopback DNS only; no listener on `1500`,
`80`, or `443`. No ISPmanager package or service unit was found through the
unprivileged checks; root can confirm with the audit script below.
@@ -39,8 +43,10 @@ UFW rule details, and protected file metadata remain unverified.
`https://git.vakanaut.ru/api/v1/version` returned Gitea `1.25.5`; runner
registration and Gitea Actions protocol connectivity are not confirmed.
- `akyldash.kg` resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS,
HTTP, HTTPS, and `/review` still return Cloudflare `521`; origin TLS and
Cloudflare Full (strict) are not verified.
HTTPS returned Cloudflare `521` with TLS verification result `0` on
2026-10-18. TCP listeners were SSH `22` and local DNS only, so no origin
listener on `80`/`443` was observed. Origin TLS and Cloudflare Full (strict)
are not verified.
## Application and Gitea facts
@@ -49,9 +55,12 @@ UFW rule details, and protected file metadata remain unverified.
PRs enabled. `main` is protected and requires one approval; the current API
principal cannot push or merge directly to `main`.
- The repository runner-list API returns `403` because this API principal is
not the repository owner. The Actions task API currently reports no workflow
runs; this does not establish runner availability. Runner inventory and
package publishing permissions remain unverified.
not the repository owner. The latest verified CI run 75 and smoke run 76 both
succeeded on `main` head `0e276ae6`. The Actions run-list endpoint returned
`403` for this API principal during this recheck, so no newer run state was
independently retrieved. The owner reports the runner is registered and
Idle; repository runner inventory and package publishing permissions remain
unverified.
- No public application listener is observed on TCP `80`/`443`. Docker
container/volume state still requires the root audit. Repository Compose describes
Caddy `2.11.6` pinned by digest, Python Search API `0.11.3`, SQLite
@@ -80,31 +89,40 @@ UFW rule details, and protected file metadata remain unverified.
`status` helper, guarded rollback helper, and installer with exact sudoers
command rules. None is installed on the VPS. CI contains no production
deployment job; starting/stopping services remains owner-run.
- A repository-scoped official `act_runner` `0.2.13` installation is prepared
in `infra/runner/`: rootless Docker under `akyldash-runner`, 15 GiB ext4
loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU,
- Rollback preflight validates digest-pinned image references against archived
image IDs and local availability, then refuses before stopping services or
creating a backup because the current flow cannot safely switch immutable
digest references. This behavior is covered by synthetic local tests; it has
not been exercised on the VPS.
- This branch adds an offline release-manifest validator and tests for the
repository allowlist intended for a future root-owned host integration. The
manifest format does not prove source provenance. No registry, host policy,
publish credential, or deployment integration is configured or installed.
- A repository-scoped official `act_runner` `0.2.13` installation is
configured in `infra/runner/`: rootless Docker under `akyldash-runner`, 15 GiB
ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU,
3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048
tasks), rotated file logs, and a systemd health timer. Jobs use the
`akyldash-ci` label and do not receive the rootless or production Docker
socket. The owner reports a partial runner installation: the 15 GiB ext4
loop-backed store is active on `/dev/loop0` after manually enabling and
starting its mount unit when the installer stopped. Remaining components,
repository registration, and Online state are unverified; this PR makes no
VPS changes. The reviewed installer locks the runner home to root ownership
(`0711`) so CI cannot replace the image or mountpoint, while writable config
and runtime state remain in runner-owned subdirectories. It reuses legacy
`0755` config directories, creates the image through a pinned file descriptor,
rejects path replacement, and requires mount options `nosuid,nodev`.
The reported installation blocker is that rootless setup receives this
protected directory as `HOME`; the installer assigns writable runner-owned
HOME/XDG paths, persists them in the `docker.service` user-unit drop-in, and
restarts an active daemon and checks the running process environment. Local
runner guards pass;
the change has not been run on the VPS.
- Gitea repository runner inventory could not be read by the current API
principal (403). The owner has not supplied runner state, and no runner
Online status or runner-hosted workflow is claimed. The manual
`runner-smoke.yaml` workflow is prepared but has not run.
tasks), rotated file logs, and a systemd health timer. Jobs target the
`akyldash-ci` label and do not receive either Docker socket. On 2026-10-18,
SSH showed `akyldash-act-runner.service` active under `akyldash-runner`, with
zero restarts and about 115 MiB current memory. Its rootless Docker store is
mounted from `/dev/loop0` as ext4 with `rw,nosuid,nodev`; the 15 GiB mount had
12 GiB free. The separate `akyldash-runner-restart.service` remains failed
with exit status 1 since 2026-10-14; its journal had no entries, so the cause
is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on
head `0e276ae6`. The Actions API returned 403 during this recheck, so no newer
run state was independently retrieved. On
head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run 64
succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
with `exec: "validate": executable file not found`; Kaniko was skipped and
smoke run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the
Caddy action's fixed `/github/workspace/...` path did not exist; Kaniko was
skipped and smoke run 68 succeeded. On head `5d72bb09`, CI run 69 and smoke
run 70 succeeded. CI validated the Caddyfile and built the backend and
OpenSearch images using `${{ github.workspace }}`. Actual per-action
resource-limit enforcement and peak usage were not independently measured or
verified. Image publishing, host-wide resource metrics, and production
deployment remain unverified. No VPS changes are made.
- An isolated local Compose smoke test built backend/OpenSearch images,
reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP
200 for `/review` through Caddy. Its containers, networks, and volumes were
@@ -113,6 +131,43 @@ UFW rule details, and protected file metadata remain unverified.
backend `0.11.3`, frontend `0.1.2`, Telegram bot `0.2.2` (the bot is not
deployed on this VPS).
## Corpus readiness (2026-10-18)
- The local raw archive and normalized corpus are ignored under `data/`; no
corpus files were changed. Their read-only SQLite manifests both pass
`quick_check` and each report 209,958 documents. The raw manifest is
28,295,168 bytes with SHA-256
`f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e`; it has
one fetch error, code `6`, absent from `documents`. The normalized manifest
is 31,571,968 bytes with SHA-256
`1679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf`; all
entries are `success`, schema `1`, normalizer `1.0.0`.
- The read-only full-file inventory in `/tmp/akyldash-corpus-audit.Mjr6aa`
stopped before validation in run `b4a4556e30cc464d90bc8d75ef1673e9` with
`Python int too large to convert to SQLite INTEGER`. At stop it had processed
2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856
normalized entries (246,321 files, 151,535 directories). This is a verifier
metadata storage failure, not a corpus finding. A full rehash with the
corrected, versioned verifier is required before readiness can be stated.
- A fresh full rehash is running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`,
run `f4dd5e53020f442bbbecee2d0e4986a6`, with verifier SHA-256
`83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`.
`result.json` currently reports `scan_status: running`; its live counters
must be recorded after completion. Sources remain read-only.
- Through the repository client, bounded MinJust OpenData requests for
`GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and
`CheckAvailable` each returned HTTP `404`. The list API returned no code list;
the direct requests do not establish whether code `6` exists. The public
listing page is reachable, but it does not map the numeric API code. A
metadata-only `CheckAvailable` request from the VPS failed in HTTP/2 with
`curl` exit 92 (`INTERNAL_ERROR`), HTTP status `000`, and TLS verification 0;
the HTTP/1.1 retry was not completed. Treat the code `6` disposition and
full corpus readiness as unresolved.
- On the VPS, read-only `stat` of `/var/lib/akyldash/data/minjust-normalized`
and its manifest was denied. Production corpus presence and metadata remain
unknown. No corpus was transferred and no production service, Docker socket,
runner policy, or VPS configuration was changed.
## Checks not completed
- Docker Engine version from the server API, container and volume inventory,
@@ -122,8 +177,10 @@ UFW rule details, and protected file metadata remain unverified.
and restore of real production data remain unverified because production is
not deployed.
- Cloudflare edge response was checked; origin DNS target, origin certificate,
Full (strict), runner registration/Online state, registry publishing, and production
deployment have not been verified.
Full (strict), registry publishing, host-wide runner metrics, and production
deployment have not been verified. The
owner-reported Idle state has not been independently checked through runner
inventory because the API returned 403.
- This audit made no host changes. No package or service was installed by this
audit; the owner performed the initial bootstrap beforehand.
@@ -141,12 +198,20 @@ UFW rule details, and protected file metadata remain unverified.
3. Run `sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh` after
reviewing the checkout; it grants only fixed verification, Compose
`config`/`status`, and guarded rollback commands.
4. After PR review and root audit, run
`sudo bash /opt/akyldash/current/infra/runner/install-runner.sh`. In the
repository Settings → Actions → Runners create a repository-level runner;
enter the token only at the interactive registration prompt as documented
in [CI_RUNNER.md](CI_RUNNER.md). Then enable the runner and health timer,
confirm **Online**, and dispatch the smoke workflow. Production deployment
remains manual and has no CI job.
4. After merging the reviewed PR, confirm a subsequent CI and smoke run on
`akyldash-ci`. Image publishing and production deployment remain manual and
are not part of CI.
5. Test backup/restore and rollback only with synthetic data in an isolated
environment before any production deployment.
## Change in this PR
Adds a read-only, resumable full-file corpus inventory and normalized-data
cross-check, a source/staging comparator requiring nonzero documents and
fragments plus held shared artifact locks, synthetic regression tests, and an
operator handoff runbook. CI's
existing `infra/deploy` unittest discovery includes the new tests. No workflow,
runner, VPS, production data, or Docker settings are changed. The first real
corpus scan stopped on an integer-storage bug; full validation with the
corrected verifier is still required. The source API's code `6` outcome remains
unconfirmed.
+54 -11
View File
@@ -34,8 +34,9 @@ backup/restore have been tested.
It creates `/etc/akyldash/production.env` from the tracked example and
generates `/etc/akyldash/review_secret`; both are root-owned mode `0600`.
It refuses to overwrite either file and does not start services or create
volumes. Review `DATA_ROOT` and place the approved normalized corpus there
before deployment.
volumes. Review `DATA_ROOT` and follow the
[corpus inventory and handoff gates](CORPUS_HANDOFF.md) before placing the
approved normalized corpus there.
3. After reviewing the root-owned checkout and audit output, run
`sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh`. It installs
the backup reader, read-only verifier, `akyldash-compose config|status`, and
@@ -65,13 +66,53 @@ backup/restore have been tested.
sudo chmod 0600 /opt/akyldash/deployed-image.txt
```
Application deployment remains owner-run. A repository-level Actions runner
installation is prepared in [CI_RUNNER.md](CI_RUNNER.md), but it is not yet
installed or registered; its Online state and smoke workflow are unverified.
The runner's job containers have no Docker socket or production secrets. CI has
no image publishing or production deployment step. The installed sudo helpers
support fixed verification, read-only Compose `config`/`status`, and guarded
rollback. They do not start or stop services.
Application deployment remains owner-run. The repository-scoped `act_runner`
`0.2.13` service is active under `akyldash-runner` and uses rootless Docker on a
separate loop-backed filesystem. CI run 75 and smoke run 76 succeeded on main
head `0e276ae6`. Runner job containers have neither Docker socket nor production
secrets. CI has no image publishing or production deployment step. The
repository defines sudo helpers for fixed verification, read-only Compose
`config`/`status`, and guarded rollback, but none is installed on the VPS. They
do not start or stop services.
## Release image contract (preparation only)
`infra/deploy/validate_release.py` checks an offline JSON release manifest
against an explicitly supplied repository allowlist. Schema version 1 requires
exactly `schema_version`, `source_ref`, `commit`, and `images`; the source ref
must be `refs/heads/main`, the commit must be a 40-character lowercase Git
SHA-1, and the images must contain exactly `backend` and `opensearch` as
`repository@sha256:<64 lowercase hex>` references. The policy has schema
version 1 and one non-empty repository allowlist for each service. Unknown or
duplicate JSON keys are rejected.
The validator takes the manifest path and an explicit policy path:
```bash
python3 infra/deploy/validate_release.py release.json --policy /etc/akyldash/release-policy.json
```
The policy shape is `{"schema_version":1,"repositories":{"backend":["registry.invalid/akyldash/backend"],"opensearch":["registry.invalid/akyldash/opensearch"]}}`.
These `registry.invalid` names are format examples only; this change does not
create a policy file or approve a registry.
The repository name in the policy must be approved for the registry. No real
registry namespace or root-owned host policy is configured by this change; do
not use the synthetic `registry.invalid` names in tests as production values.
The host integration must load policy only from its fixed root-owned path.
This validator checks format and allowlist only: a manifest can claim a main
ref and commit, but is not proof of source, build provenance, signature, or
trusted publication. CI still has no publish credentials, image publication,
host helper, or deploy trigger.
Compose merges an image override with the existing service `build` entries.
The regression test renders fake digest references and confirms both entries
remain; any future deploy command must use `docker compose up --no-build` so
the host cannot silently build a different image. This does not make digest
rollback available: the current rollback helper expects local tags. A separate
host-side deploy/rollback design, owner-approved registry permissions and
retention, and manual backup/health/rollback end-to-end validation are still
required before publishing or enabling CD.
## CI
@@ -79,5 +120,7 @@ rollback. They do not start or stop services.
production images on pull requests and pushes to `main`. It does not publish an
image or deploy production. `.gitea/workflows/runner-smoke.yaml` exercises the
`akyldash-ci` runner on pull requests, pushes to `main`, or manual dispatch.
Runner registration is incomplete, so neither workflow execution on that
runner nor current `ubuntu-latest` runner availability has been confirmed.
Both workflows passed on main head `0e276ae6` in runs 75 and 76 respectively.
The repository runner-list endpoint is not available to the current API
principal, so its registration details cannot be independently read through
Gitea API.
+12 -5
View File
@@ -18,11 +18,18 @@ sudo -n /usr/local/sbin/akyldash-rollback \
The helper verifies archive integrity, Compose image references, deployment
file hashes, and the availability of both prior local images. It makes and
verifies a second cold backup, restores only the prior backend/OpenSearch image
tags, waits for Compose healthchecks, and checks HTTPS locally. It refuses to
restore if the current Compose/Caddyfile differs from the saved release. This
does not restore data or automatically undo a release with an unknown schema
change. The application has no formal migration framework; compatibility must
be checked for every change. If it is uncertain, stop and ask the owner.
tags, waits for Compose healthchecks, and checks HTTPS locally. Digest-pinned
references are validated against their recorded local image IDs, but rollback
fails before stopping services or making a backup when either image reference
uses `@sha256:`. The helper cannot safely switch immutable digest references
with the current release-recording flow. Do not retag a digest manually; digest
rollback needs a separately reviewed atomic release-reference mechanism.
It refuses to restore if the current Compose/Caddyfile differs from the saved
release. This does not restore data or automatically undo a release with an
unknown schema change. The application has no formal migration framework;
compatibility must be checked for every change. If it is uncertain, stop and ask
the owner.
The initial owner-run deployment must record its running backend image ID in
`/opt/akyldash/deployed-image.txt` with mode `0600`; the rollback helper updates
@@ -11,8 +11,13 @@
- загрузчик находится в `backend/ingestion/minjust_cbd.py`;
- сырой архив хранится в `data/minjust-cbd` и исключён из Git;
- в манифесте 209 811 успешно загруженных документов;
- 13 кодов остаются в таблице `errors` и отсутствуют в таблице `documents`;
- в read-only проверенном манифесте 209 958 документов;
- один код, `6`, остаётся в таблице `errors` и отсутствует в `documents`;
это не подтверждает отсутствие документа в источнике: при последней
проверке OpenData API возвращал HTTP `404` для списка, `GetDocument(6)`,
контрольного `GetDocument(1096)` и `CheckAvailable`;
- в нормализованном манифесте 209 958 успешных записей, schema `1`,
normalizer `1.0.0`; полная потоковая проверка всех файлов ещё выполняется.
- документ содержит `metadata.json` и каталог `editions`;
- редакция содержит `metadata.json`, `ru.html` и/или `ky.html`, иногда
изображения;
+5 -2
View File
@@ -1,10 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
source "$script_dir/../test/run-as-unprivileged.sh"
output=$(mktemp)
trap 'rm -f "$output"' EXIT
if bash "$(dirname "$0")/audit-host.sh" >"$output" 2>&1; then
echo 'Host audit unexpectedly succeeded without root.' >&2
if run_as_unprivileged bash "$script_dir/audit-host.sh" >"$output" 2>&1; then
echo 'Host audit unexpectedly succeeded as an unprivileged UID.' >&2
exit 1
fi
grep -qx 'Run this read-only audit as root.' "$output"
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
guard=$(mktemp)
output=$(mktemp)
trap 'rm -f "$guard" "$output"' EXIT
awk '
/^if \[\[ \$EUID -ne 0 \]\]; then$/ { capture = 1 }
capture { print }
capture && /^fi$/ { found = 1; exit }
END { if (!found) exit 1 }
' "$script_dir/audit-host.sh" | sed 's/\$EUID/\$TEST_EUID/g' >"$guard" || {
echo 'Could not extract the host audit root guard.' >&2
exit 1
}
if ! grep -Fq 'if [[ $TEST_EUID -ne 0 ]]; then' "$guard"; then
echo 'Extracted guard does not use the simulated effective UID.' >&2
exit 1
fi
if TEST_EUID=0 bash "$guard" >"$output" 2>&1; then
[[ ! -s $output ]] || { cat "$output" >&2; exit 1; }
else
echo 'Host audit root guard rejected simulated UID 0.' >&2
exit 1
fi
if TEST_EUID=65534 bash "$guard" >"$output" 2>&1; then
echo 'Host audit root guard accepted simulated non-root UID.' >&2
exit 1
fi
grep -qx 'Run this read-only audit as root.' "$output"
echo 'PASS host audit root guard accepts UID 0 and rejects non-root UIDs'
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env python3
"""Require non-empty completed normalized audits and compare every file row."""
from __future__ import annotations
import argparse
import fcntl
import itertools
import json
import os
import sqlite3
import stat
import sys
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Any
class ComparisonError(ValueError):
pass
@contextmanager
def shared_inventory_lock(result_path: Path, label: str):
lock_path = result_path.absolute().parent / ".inventory.lock"
if not hasattr(os, "O_NOFOLLOW"):
raise ComparisonError(f"{label}: safe inventory lock opens are unsupported")
try:
descriptor = os.open(lock_path, os.O_RDONLY | os.O_NOFOLLOW)
except OSError as error:
raise ComparisonError(f"{label}: cannot open inventory lock: {error}") from error
try:
metadata = os.fstat(descriptor)
if (
not stat.S_ISREG(metadata.st_mode)
or metadata.st_nlink != 1
or stat.S_IMODE(metadata.st_mode) & 0o077
):
raise ComparisonError(f"{label}: inventory lock must be a private regular file")
try:
fcntl.flock(descriptor, fcntl.LOCK_SH | fcntl.LOCK_NB)
except BlockingIOError as error:
raise ComparisonError(f"{label}: inventory is active (lock busy)") from error
yield
finally:
os.close(descriptor)
def read_audit(result_path: Path, label: str) -> tuple[sqlite3.Connection, dict[str, Any]]:
result_path = result_path.absolute()
try:
metadata = result_path.lstat()
if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode):
raise ComparisonError(f"{label}: result.json must be a regular file")
result = json.loads(result_path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as error:
raise ComparisonError(f"{label}: cannot read result.json: {error}") from error
if not isinstance(result, dict):
raise ComparisonError(f"{label}: result.json must contain an object")
if result.get("status") != "complete" or result.get("verification_status") != "passed":
raise ComparisonError(f"{label}: inventory must be complete and passed")
verifier_sha256 = result.get("verifier_sha256")
if not isinstance(verifier_sha256, str) or len(verifier_sha256) != 64:
raise ComparisonError(f"{label}: verifier SHA-256 is missing")
normalized_root = result.get("roots", {}).get("normalized")
if not isinstance(normalized_root, str):
raise ComparisonError(f"{label}: normalized root is missing")
summary = result.get("manifest_summary", {}).get("normalized")
if not isinstance(summary, dict):
raise ComparisonError(f"{label}: normalized manifest summary is missing")
for key in ("documents", "fragments"):
if not isinstance(summary.get(key), int) or summary[key] <= 0:
raise ComparisonError(f"{label}: normalized {key} must be greater than zero")
database_path = result_path.with_name("inventory.sqlite3")
try:
db_metadata = database_path.lstat()
if (
stat.S_ISLNK(db_metadata.st_mode)
or not stat.S_ISREG(db_metadata.st_mode)
or db_metadata.st_nlink != 1
):
raise ComparisonError(f"{label}: inventory.sqlite3 must be a regular unlinked file")
for suffix in ("-wal", "-shm", "-journal"):
if Path(f"{database_path}{suffix}").exists():
raise ComparisonError(f"{label}: SQLite sidecar files are present")
connection = sqlite3.connect(database_path.resolve().as_uri() + "?mode=ro&immutable=1", uri=True)
connection.row_factory = sqlite3.Row
if connection.execute("PRAGMA quick_check").fetchone()[0] != "ok":
raise ComparisonError(f"{label}: inventory database failed quick_check")
manifest_documents = connection.execute(
"SELECT count(*) FROM manifest_docs WHERE root_id='normalized'"
).fetchone()[0]
if manifest_documents != summary["documents"]:
raise ComparisonError(f"{label}: normalized document count does not match inventory")
files, byte_count, unfinished = connection.execute(
"SELECT count(*),coalesce(sum(size),0),"
"sum(CASE WHEN status!='hashed' OR size IS NULL OR sha256 IS NULL THEN 1 ELSE 0 END) "
"FROM files WHERE root_id='normalized'"
).fetchone()
progress = result.get("progress", {}).get("normalized", {})
if unfinished or files != progress.get("files") or byte_count != progress.get("bytes"):
raise ComparisonError(f"{label}: inventory rows do not match completed scan counters")
return connection, {
"root": normalized_root,
"documents": summary["documents"],
"fragments": summary["fragments"],
"files": files,
"bytes": byte_count,
"verifier_sha256": verifier_sha256,
"schema_version": result.get("schema_version"),
}
except sqlite3.Error as error:
raise ComparisonError(f"{label}: cannot validate inventory database: {error}") from error
def compare_inventories(source_result: Path, staging_result: Path) -> dict[str, int]:
with ExitStack() as locks:
locks.enter_context(shared_inventory_lock(source_result, "source"))
locks.enter_context(shared_inventory_lock(staging_result, "staging"))
source_db, source = read_audit(source_result, "source")
try:
staging_db, staging = read_audit(staging_result, "staging")
except Exception:
source_db.close()
raise
try:
if source["root"] == staging["root"]:
raise ComparisonError("source and staging must be different normalized roots")
if source["verifier_sha256"] != staging["verifier_sha256"] or source["schema_version"] != staging["schema_version"]:
raise ComparisonError("source and staging audits must use the same verifier and schema")
if (source["documents"], source["fragments"]) != (staging["documents"], staging["fragments"]):
raise ComparisonError("source and staging document or fragment counts differ")
query = (
"SELECT relative_path,size,sha256 FROM files "
"WHERE root_id='normalized' ORDER BY relative_path"
)
left = iter(source_db.execute(query))
right = iter(staging_db.execute(query))
mismatch_count = 0
for source_row, staging_row in itertools.zip_longest(left, right):
if source_row != staging_row:
mismatch_count += 1
if mismatch_count:
raise ComparisonError(f"source and staging differ in {mismatch_count} inventory rows")
return {"documents": source["documents"], "fragments": source["fragments"], "files": source["files"], "bytes": source["bytes"]}
finally:
source_db.close()
staging_db.close()
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("source_result", type=Path)
parser.add_argument("staging_result", type=Path)
arguments = parser.parse_args()
try:
summary = compare_inventories(arguments.source_result, arguments.staging_result)
except (ComparisonError, OSError) as error:
print(f"FAIL: {error}", file=sys.stderr)
return 1
print(
"PASS: matched "
f"{summary['documents']} documents, {summary['fragments']} fragments, "
f"{summary['files']} files, {summary['bytes']} bytes"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+45 -8
View File
@@ -19,6 +19,12 @@ ENV_FILE = Path("/etc/akyldash/production.env")
BACKUP_DIR = Path("/var/backups/akyldash")
BACKUP_TOOL = Path("/usr/local/lib/akyldash/archive.py")
APP_SERVICES = {"backend", "opensearch"}
IMAGE_TAG_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]*")
IMAGE_DIGEST_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]*@sha256:[0-9a-f]{64}")
def is_digest_reference(reference: str) -> bool:
return IMAGE_DIGEST_RE.fullmatch(reference) is not None
def image_restore_plan(manifest: dict, configured: dict[str, str]) -> dict[str, tuple[str, str]]:
@@ -34,14 +40,48 @@ def image_restore_plan(manifest: dict, configured: dict[str, str]) -> dict[str,
image_id = image.get("image_id")
if reference != configured.get(service):
raise ValueError(f"Compose image reference changed for {service}; refusing rollback")
if not isinstance(reference, str) or not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._:/-]*", reference):
raise ValueError(f"{service} image reference is not a local tag")
if not isinstance(reference, str) or not (
IMAGE_TAG_RE.fullmatch(reference) or is_digest_reference(reference)
):
raise ValueError(f"invalid image reference for {service}")
if not isinstance(image_id, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id):
raise ValueError(f"invalid previous image ID for {service}")
plan[service] = (image_id, reference)
return plan
def verify_previous_images(plan: dict[str, tuple[str, str]]) -> None:
for image_id, reference in plan.values():
try:
actual = run(["docker", "image", "inspect", "--format", "{{.Id}}", image_id], capture=True)
except subprocess.CalledProcessError as error:
raise ValueError(f"previous image is unavailable: {image_id}") from error
if actual != image_id:
raise ValueError(f"previous image is unavailable: {image_id}")
if is_digest_reference(reference):
try:
referenced_id = run(
["docker", "image", "inspect", "--format", "{{.Id}}", reference], capture=True
)
except subprocess.CalledProcessError as error:
raise ValueError(f"previous digest reference is unavailable: {reference}") from error
if referenced_id != image_id:
raise ValueError(f"previous digest reference does not match its recorded image: {reference}")
def require_tag_rollback(plan: dict[str, tuple[str, str]]) -> None:
if any(is_digest_reference(reference) for _image_id, reference in plan.values()):
raise ValueError(
"rollback of digest-pinned images is not supported safely; no services or image references were changed"
)
def tag_previous_images(plan: dict[str, tuple[str, str]]) -> None:
require_tag_rollback(plan)
for image_id, reference in plan.values():
run(["docker", "image", "tag", image_id, reference])
def run(command: list[str], *, capture: bool = False) -> str:
result = subprocess.run(command, check=True, text=True, capture_output=capture)
return result.stdout.strip() if capture else ""
@@ -96,10 +136,8 @@ def rollback(backup: Path, data_compatible: bool) -> None:
rendered = json.loads(compose("config", "--format", "json", capture=True))
configured = {service: rendered["services"][service]["image"] for service in APP_SERVICES}
plan = image_restore_plan(manifest, configured)
for image_id, _reference in plan.values():
actual = run(["docker", "image", "inspect", "--format", "{{.Id}}", image_id], capture=True)
if actual != image_id:
raise ValueError(f"previous image is unavailable: {image_id}")
verify_previous_images(plan)
require_tag_rollback(plan)
backup_name = f"akyldash-pre-rollback-{dt.datetime.now(dt.timezone.utc):%Y%m%dT%H%M%SZ}.tar.gz"
backup_output = BACKUP_DIR / backup_name
@@ -123,8 +161,7 @@ def rollback(backup: Path, data_compatible: bool) -> None:
compose("up", "--detach")
raise
for image_id, reference in plan.values():
run(["docker", "image", "tag", image_id, reference])
tag_previous_images(plan)
compose("up", "--detach", "--no-build", "--wait", "--wait-timeout", "300")
run(
[
+5 -2
View File
@@ -1,10 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
source "$script_dir/../test/run-as-unprivileged.sh"
output=$(mktemp)
trap 'rm -f "$output"' EXIT
if bash "$(dirname "$0")/prepare-production-config.sh" >"$output" 2>&1; then
echo 'Production config setup unexpectedly succeeded without root.' >&2
if run_as_unprivileged bash "$script_dir/prepare-production-config.sh" >"$output" 2>&1; then
echo 'Production config setup unexpectedly succeeded as an unprivileged UID.' >&2
exit 1
fi
grep -qx 'Run this configuration step as root.' "$output"
@@ -0,0 +1,132 @@
import fcntl
import json
import os
import sqlite3
import tempfile
import unittest
from pathlib import Path
from compare_corpus_inventories import ComparisonError, compare_inventories
class CompareCorpusInventoryTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.base = Path(self.temp.name)
self.source = self._artifact("source", "/data/source")
self.staging = self._artifact("staging", "/data/staging")
def tearDown(self):
self.temp.cleanup()
def _artifact(self, name, root, documents=1, fragments=2, rows=None):
directory = self.base / name
directory.mkdir(mode=0o700)
(directory / ".inventory.lock").touch(mode=0o600)
database = directory / "inventory.sqlite3"
connection = sqlite3.connect(database)
connection.execute(
"CREATE TABLE files (root_id TEXT, relative_path TEXT, size INTEGER, sha256 TEXT, status TEXT)"
)
connection.execute("CREATE TABLE manifest_docs (root_id TEXT, code TEXT)")
connection.executemany(
"INSERT INTO manifest_docs VALUES ('normalized',?)",
[(str(code),) for code in range(documents)],
)
if rows is None:
rows = [
("manifest.sqlite3", 16, "a" * 64),
("documents/1/document.json", 24, "b" * 64),
]
connection.executemany(
"INSERT INTO files VALUES ('normalized',?,?,?,'hashed')", rows
)
connection.commit()
connection.close()
result = {
"status": "complete",
"verification_status": "passed",
"schema_version": 2,
"verifier_sha256": "c" * 64,
"roots": {"normalized": root},
"manifest_summary": {
"normalized": {"documents": documents, "fragments": fragments}
},
"progress": {
"normalized": {
"files": len(rows),
"bytes": sum(row[1] for row in rows),
}
},
}
result_path = directory / "result.json"
result_path.write_text(json.dumps(result), encoding="utf-8")
return result_path
def test_nonempty_source_and_staging_inventories_match(self):
self.assertEqual(
compare_inventories(self.source, self.staging),
{"documents": 1, "fragments": 2, "files": 2, "bytes": 40},
)
def test_empty_source_is_rejected(self):
empty = self._artifact("empty-source", "/data/empty-source", documents=0, fragments=0)
with self.assertRaisesRegex(ComparisonError, "source: normalized documents"):
compare_inventories(empty, self.staging)
def test_empty_staging_is_rejected(self):
empty = self._artifact("empty-staging", "/data/empty-staging", documents=0, fragments=0)
with self.assertRaisesRegex(ComparisonError, "staging: normalized documents"):
compare_inventories(self.source, empty)
def test_zero_fragments_are_rejected_even_with_documents(self):
empty = self._artifact("no-fragments", "/data/no-fragments", fragments=0)
with self.assertRaisesRegex(ComparisonError, "normalized fragments"):
compare_inventories(self.source, empty)
def test_different_file_sha_is_rejected(self):
changed = self._artifact(
"changed", "/data/changed",
rows=[("manifest.sqlite3", 16, "a" * 64), ("documents/1/document.json", 24, "d" * 64)],
)
with self.assertRaisesRegex(ComparisonError, "differ in 1 inventory rows"):
compare_inventories(self.source, changed)
def test_busy_source_lock_is_rejected(self):
self._assert_busy_lock(self.source, "source")
def test_busy_staging_lock_is_rejected(self):
self._assert_busy_lock(self.staging, "staging")
def _assert_busy_lock(self, result_path, label):
descriptor = os.open(result_path.parent / ".inventory.lock", os.O_RDWR)
try:
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
with self.assertRaisesRegex(ComparisonError, rf"{label}: inventory is active"):
compare_inventories(self.source, self.staging)
finally:
os.close(descriptor)
def test_missing_lock_is_rejected_without_creating_it(self):
lock = self.source.parent / ".inventory.lock"
lock.unlink()
with self.assertRaisesRegex(ComparisonError, "source: cannot open inventory lock"):
compare_inventories(self.source, self.staging)
self.assertFalse(lock.exists())
def test_symlink_lock_is_rejected(self):
lock = self.source.parent / ".inventory.lock"
lock.unlink()
target = self.base / "external-lock"
target.touch(mode=0o600)
lock.symlink_to(target)
with self.assertRaisesRegex(ComparisonError, "source: cannot open inventory lock"):
compare_inventories(self.source, self.staging)
def test_read_only_lock_can_be_shared(self):
(self.source.parent / ".inventory.lock").chmod(0o400)
self.assertEqual(compare_inventories(self.source, self.staging)["documents"], 1)
if __name__ == "__main__":
unittest.main()
+131
View File
@@ -0,0 +1,131 @@
import json
import os
import shutil
import subprocess
import tempfile
import unittest
from pathlib import Path
from validate_release import validate_manifest
ROOT = Path(__file__).resolve().parents[2]
REPOSITORIES = {
"backend": ["registry.invalid/akyldash/backend"],
"opensearch": ["registry.invalid/akyldash/opensearch"],
}
class ReleaseManifestTests(unittest.TestCase):
def setUp(self):
self.policy = {"schema_version": 1, "repositories": REPOSITORIES}
self.manifest = {
"schema_version": 1,
"source_ref": "refs/heads/main",
"commit": "a" * 40,
"images": {
"backend": "registry.invalid/akyldash/backend@sha256:" + "b" * 64,
"opensearch": "registry.invalid/akyldash/opensearch@sha256:" + "c" * 64,
},
}
def test_accepts_both_policy_pinned_images(self):
self.assertEqual(validate_manifest(self.manifest, self.policy), self.manifest)
def test_rejects_untrusted_ref(self):
self.manifest["source_ref"] = "refs/pull/12/head"
with self.assertRaisesRegex(ValueError, "source_ref"):
validate_manifest(self.manifest, self.policy)
def test_rejects_tag_instead_of_digest(self):
self.manifest["images"]["backend"] = "registry.invalid/akyldash/backend:latest"
with self.assertRaisesRegex(ValueError, "sha256"):
validate_manifest(self.manifest, self.policy)
def test_rejects_empty_image_reference(self):
self.manifest["images"]["backend"] = ""
with self.assertRaisesRegex(ValueError, "sha256"):
validate_manifest(self.manifest, self.policy)
def test_rejects_invalid_commit(self):
self.manifest["commit"] = "not-a-commit"
with self.assertRaisesRegex(ValueError, "commit"):
validate_manifest(self.manifest, self.policy)
def test_rejects_repository_outside_policy(self):
self.manifest["images"]["backend"] = "registry.invalid/other/backend@sha256:" + "b" * 64
with self.assertRaisesRegex(ValueError, "not allowed"):
validate_manifest(self.manifest, self.policy)
def test_rejects_missing_or_extra_images(self):
self.manifest["images"].pop("opensearch")
with self.assertRaisesRegex(ValueError, "exactly backend and opensearch"):
validate_manifest(self.manifest, self.policy)
self.manifest["images"]["opensearch"] = "registry.invalid/akyldash/opensearch@sha256:" + "c" * 64
self.manifest["images"]["extra"] = "registry.invalid/akyldash/extra@sha256:" + "d" * 64
with self.assertRaisesRegex(ValueError, "exactly backend and opensearch"):
validate_manifest(self.manifest, self.policy)
def test_rejects_duplicate_json_keys(self):
with tempfile.TemporaryDirectory() as directory:
manifest_path = Path(directory) / "manifest.json"
policy_path = Path(directory) / "policy.json"
manifest_path.write_text(
'{"schema_version":1,"source_ref":"refs/heads/main","commit":"' + "a" * 40
+ '","images":{"backend":"first","backend":"second","opensearch":"third"}}',
encoding="utf-8",
)
policy_path.write_text(json.dumps(self.policy), encoding="utf-8")
command = [
"python3", str(ROOT / "infra/deploy/validate_release.py"), str(manifest_path),
"--policy", str(policy_path),
]
result = subprocess.run(command, text=True, capture_output=True)
self.assertEqual(result.returncode, 1)
self.assertIn("duplicate JSON key: backend", result.stderr)
def test_rejects_malformed_json(self):
with tempfile.TemporaryDirectory() as directory:
manifest_path = Path(directory) / "manifest.json"
policy_path = Path(directory) / "policy.json"
manifest_path.write_text('{"schema_version":', encoding="utf-8")
policy_path.write_text(json.dumps(self.policy), encoding="utf-8")
result = subprocess.run(
["python3", str(ROOT / "infra/deploy/validate_release.py"), str(manifest_path), "--policy", str(policy_path)],
text=True,
capture_output=True,
)
self.assertEqual(result.returncode, 1)
self.assertIn("cannot read valid JSON", result.stderr)
@unittest.skipUnless(shutil.which("docker"), "Docker CLI is unavailable")
def test_compose_override_renders_digest_images_and_preserves_build(self):
backend = "registry.invalid/akyldash/backend@sha256:" + "b" * 64
opensearch = "registry.invalid/akyldash/opensearch@sha256:" + "c" * 64
with tempfile.TemporaryDirectory() as directory:
temp = Path(directory)
secret = temp / "review_secret"
secret.touch()
override = temp / "release-override.json"
override.write_text(json.dumps({"services": {
"backend": {"image": backend},
"opensearch": {"image": opensearch},
}}), encoding="utf-8")
command = [
"docker", "compose", "--env-file", str(ROOT / "deploy/production/.env.example"),
"-f", str(ROOT / "deploy/production/compose.yaml"), "-f", str(override),
"config", "--format", "json",
]
result = subprocess.run(
command, text=True, capture_output=True,
env={**os.environ, "REVIEW_SECRET_FILE": str(secret)},
)
self.assertEqual(result.returncode, 0, result.stderr)
services = json.loads(result.stdout)["services"]
self.assertEqual(services["backend"]["image"], backend)
self.assertEqual(services["opensearch"]["image"], opensearch)
self.assertIn("build", services["backend"])
self.assertIn("build", services["opensearch"])
if __name__ == "__main__":
unittest.main()
+95 -2
View File
@@ -1,6 +1,12 @@
import json
import subprocess
import sys
import unittest
from types import SimpleNamespace
from unittest.mock import patch
from rollback import image_restore_plan
import rollback
from rollback import image_restore_plan, require_tag_rollback, tag_previous_images, verify_previous_images
class RollbackPlanTests(unittest.TestCase):
@@ -28,9 +34,96 @@ class RollbackPlanTests(unittest.TestCase):
def test_refuses_digest_reference_that_cannot_be_retagged(self):
self.manifest["images"]["backend"]["configured"] = "registry/backend@sha256:" + "c" * 64
self.configured["backend"] = self.manifest["images"]["backend"]["configured"]
with self.assertRaisesRegex(ValueError, "local tag"):
plan = image_restore_plan(self.manifest, self.configured)
self.assertEqual(plan["backend"], ("sha256:" + "a" * 64, self.configured["backend"]))
with self.assertRaisesRegex(ValueError, "not supported safely"):
require_tag_rollback(plan)
def test_refuses_malformed_digest_reference(self):
self.manifest["images"]["backend"]["configured"] = "registry/backend@sha256:xyz"
self.configured["backend"] = self.manifest["images"]["backend"]["configured"]
with self.assertRaisesRegex(ValueError, "invalid image reference"):
image_restore_plan(self.manifest, self.configured)
def test_refuses_invalid_recorded_image_id(self):
self.manifest["images"]["backend"]["image_id"] = "sha256:not-a-digest"
with self.assertRaisesRegex(ValueError, "invalid previous image ID"):
image_restore_plan(self.manifest, self.configured)
def test_digest_reference_must_resolve_to_recorded_image_id(self):
digest = "registry/backend@sha256:" + "c" * 64
self.manifest["images"]["backend"]["configured"] = digest
self.configured["backend"] = digest
plan = image_restore_plan(self.manifest, self.configured)
with patch("rollback.run", side_effect=["sha256:" + "a" * 64, "sha256:" + "a" * 64,
"sha256:" + "b" * 64]):
verify_previous_images(plan)
with patch("rollback.run", side_effect=["sha256:" + "a" * 64, "sha256:" + "d" * 64]):
with self.assertRaisesRegex(ValueError, "does not match its recorded image"):
verify_previous_images(plan)
def test_missing_previous_image_fails_closed(self):
plan = image_restore_plan(self.manifest, self.configured)
with patch("rollback.run", side_effect=subprocess.CalledProcessError(1, "docker")):
with self.assertRaisesRegex(ValueError, "previous image is unavailable"):
verify_previous_images(plan)
def test_digest_reference_is_never_tagged(self):
digest = "registry/backend@sha256:" + "c" * 64
plan = {"backend": ("sha256:" + "a" * 64, digest)}
with patch("rollback.run") as run_mock:
with self.assertRaisesRegex(ValueError, "not supported safely"):
tag_previous_images(plan)
run_mock.assert_not_called()
def test_digest_rollback_fails_before_stopping_or_changing_references(self):
backend_id = "sha256:" + "a" * 64
opensearch_id = "sha256:" + "b" * 64
digest = "registry/backend@sha256:" + "c" * 64
manifest = {
"images": {
"backend": {"configured": digest, "image_id": backend_id},
"opensearch": self.manifest["images"]["opensearch"],
},
"deployment_files": {
"deploy/production/compose.yaml": "compose-hash",
"deploy/production/Caddyfile": "caddy-hash",
},
}
rendered = {
"services": {
"backend": {"image": digest},
"opensearch": {"image": self.configured["opensearch"]},
}
}
fake_archive = SimpleNamespace(
digest=lambda path: "compose-hash" if path.name == "compose.yaml" else "caddy-hash",
verify_archive=lambda _backup: manifest,
)
backup = rollback.BACKUP_DIR / "synthetic.tar.gz"
with (
patch("rollback.Path.is_file", return_value=True),
patch("rollback.Path.is_symlink", return_value=False),
patch("rollback.Path.stat", return_value=SimpleNamespace(st_uid=0, st_mode=0o100600)),
patch("rollback.compose", return_value=json.dumps(rendered)) as compose_mock,
patch("rollback.run", side_effect=[backend_id, backend_id, opensearch_id]) as run_mock,
patch.dict(sys.modules, {"archive": fake_archive}),
):
with self.assertRaisesRegex(ValueError, "not supported safely"):
rollback.rollback(backup, data_compatible=True)
compose_mock.assert_called_once_with("config", "--format", "json", capture=True)
self.assertEqual(run_mock.call_count, 3)
self.assertTrue(all(call.args[0][:3] == ["docker", "image", "inspect"]
for call in run_mock.call_args_list))
def test_tag_rollback_keeps_existing_image_tag_operation(self):
plan = image_restore_plan(self.manifest, self.configured)
with patch("rollback.run") as run_mock:
tag_previous_images(plan)
self.assertEqual(run_mock.call_count, 2)
run_mock.assert_any_call(["docker", "image", "tag", "sha256:" + "a" * 64,
"akyldash/backend:local"])
if __name__ == "__main__":
unittest.main()
+328
View File
@@ -0,0 +1,328 @@
import hashlib
import json
import os
import fcntl
import sqlite3
import tempfile
import unittest
from pathlib import Path
from verify_corpus import (
InventoryError,
database,
insert_or_update_file,
prefix_upper_bound,
raw_file_rows,
combined_source_hash,
safe_relative_path,
validate_fragment_file,
verify_corpus,
)
class CorpusInventoryTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.base = Path(self.temp.name)
self.raw = self.base / "raw"
self.raw.mkdir()
self._manifest()
def tearDown(self):
self.temp.cleanup()
def _manifest(self):
connection = sqlite3.connect(self.raw / "manifest.sqlite3")
connection.executescript(
"CREATE TABLE documents (code INTEGER PRIMARY KEY, source_sha256 TEXT, fetched_at TEXT);"
"CREATE TABLE errors (code INTEGER PRIMARY KEY, message TEXT, failed_at TEXT);"
)
connection.close()
def _nonempty_corpus(self):
document = self.raw / "documents/1"
edition = document / "editions/2"
edition.mkdir(parents=True)
(document / "metadata.json").write_text('{"Code":"1"}\n', encoding="utf-8")
(edition / "metadata.json").write_text('{"Code":"2"}\n', encoding="utf-8")
html = edition / "ru.html"
html.write_text("<p>Фрагмент</p>\n", encoding="utf-8")
source_files = []
for path in sorted(document.rglob("*")):
if path.is_file():
digest = hashlib.sha256(path.read_bytes()).hexdigest()
source_files.append({"path": path.relative_to(self.raw).as_posix(), "sha256": digest})
source_sha = combined_source_hash(source_files)
raw_manifest = sqlite3.connect(self.raw / "manifest.sqlite3")
raw_manifest.execute("INSERT INTO documents VALUES (1, ?, 'now')", (source_sha,))
raw_manifest.commit()
raw_manifest.close()
normalized = self.base / "normalized"
language = normalized / "documents/1/editions/2/ru"
language.mkdir(parents=True)
fragment_text = "Фрагмент"
fragments = [{
"id": "document:1:edition:2:lang:ru:fragment:1",
"document_code": "1",
"edition_code": "2",
"language": "ru",
"position": 1,
"text": fragment_text,
"text_sha256": hashlib.sha256(fragment_text.encode("utf-8")).hexdigest(),
"source_path": "documents/1/editions/2/ru.html",
"source_sha256": next(item["sha256"] for item in source_files if item["path"].endswith("/ru.html")),
}]
(language / "fragments.json").write_text(json.dumps(fragments, ensure_ascii=False), encoding="utf-8")
(language / "content.txt").write_text(fragment_text + "\n", encoding="utf-8")
(language / "content.html").write_text("<p>Фрагмент</p>\n", encoding="utf-8")
document_json = {
"schema_version": "1",
"source_code": "1",
"normalizer": {"version": "1.0.0"},
"source": {"path": "documents/1", "files": source_files, "sha256": source_sha},
"editions": [{"source_code": "2"}],
}
(normalized / "documents/1/document.json").write_text(json.dumps(document_json), encoding="utf-8")
edition_json = {
"source_code": "2",
"available_languages": ["ru"],
"source": {
"path": "documents/1/editions/2",
"files": [item for item in source_files if item["path"].startswith("documents/1/editions/2/")],
},
"images": [],
}
(normalized / "documents/1/editions/2/edition.json").write_text(json.dumps(edition_json), encoding="utf-8")
normalized_manifest = sqlite3.connect(normalized / "manifest.sqlite3")
normalized_manifest.execute(
"CREATE TABLE documents (code TEXT PRIMARY KEY, source_sha256 TEXT, schema_version TEXT, normalizer_version TEXT, state TEXT)"
)
normalized_manifest.execute("INSERT INTO documents VALUES ('1', ?, '1', '1.0.0', 'success')", (source_sha,))
normalized_manifest.commit()
normalized_manifest.close()
return normalized, source_sha
def test_output_inside_source_is_rejected_without_creating_it(self):
output = self.raw / "artifacts"
before_mode = self.raw.stat().st_mode
with self.assertRaisesRegex(InventoryError, "separate"):
verify_corpus(self.raw, None, output)
self.assertFalse(output.exists())
self.assertEqual(self.raw.stat().st_mode, before_mode)
self.assertEqual({path.name for path in self.raw.iterdir()}, {"manifest.sqlite3"})
def test_symlink_output_is_rejected_without_changing_target(self):
target = self.base / "target"
target.mkdir(mode=0o755)
link = self.base / "output-link"
try:
link.symlink_to(target, target_is_directory=True)
except (NotImplementedError, OSError):
self.skipTest("directory symlinks are unavailable")
before_mode = target.stat().st_mode
with self.assertRaisesRegex(InventoryError, "link or file"):
verify_corpus(self.raw, None, link)
self.assertEqual(target.stat().st_mode, before_mode)
self.assertEqual(list(target.iterdir()), [])
def test_public_existing_output_is_rejected_without_chmod(self):
output = self.base / "public-output"
output.mkdir(mode=0o755)
output.chmod(0o755)
before_mode = output.stat().st_mode
with self.assertRaisesRegex(InventoryError, "must be private"):
verify_corpus(self.raw, None, output)
self.assertEqual(output.stat().st_mode, before_mode)
self.assertEqual(list(output.iterdir()), [])
def test_resume_rejects_symlink_inventory_without_touching_target(self):
output = self.base / "artifacts"
output.mkdir(mode=0o700)
target = self.base / "do-not-touch.sqlite3"
target.write_bytes(b"sentinel")
link = output / "inventory.sqlite3"
try:
link.symlink_to(target)
except (NotImplementedError, OSError):
self.skipTest("file symlinks are unavailable")
with self.assertRaisesRegex(InventoryError, "private regular file"):
verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(target.read_bytes(), b"sentinel")
def test_rejects_unsafe_cross_platform_relative_paths(self):
for value in ("../outside", "C:\\outside", "folder\\file", "//server/share"):
with self.subTest(value=value), self.assertRaises(InventoryError):
safe_relative_path(value)
def test_interrupted_inventory_resumes_and_detects_changed_input(self):
payload = self.raw / "payload.txt"
payload.write_text("before", encoding="utf-8")
output = self.base / "artifacts"
stopped = verify_corpus(self.raw, None, output, stop_after_entries=1)
self.assertEqual(stopped["status"], "incomplete")
resumed = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(resumed["status"], "complete")
self.assertEqual(resumed["input_changes"]["raw"]["added"], 1)
payload.write_text("after", encoding="utf-8")
changed = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(changed["status"], "complete")
self.assertEqual(changed["input_changes"]["raw"]["changed"], 1)
rows = sqlite3.connect(output / "inventory.sqlite3").execute(
"SELECT sha256 FROM files WHERE relative_path='payload.txt'"
).fetchone()
self.assertEqual(rows[0], hashlib.sha256(b"after").hexdigest())
def test_large_filesystem_identity_values_are_stored_without_integer_overflow(self):
connection = database(self.base / "identity.sqlite3")
insert_or_update_file(
connection, "run", "raw", "payload", 1, "a" * 64,
(2**100, 2**120, 1, 2**110, 2**115, 1), {"raw": {"added": 0, "changed": 0}},
)
device, inode, mtime_ns, ctime_ns = connection.execute(
"SELECT device,inode,mtime_ns,ctime_ns FROM files"
).fetchone()
self.assertEqual((device, inode, mtime_ns, ctime_ns), tuple(map(str, (2**100, 2**120, 2**110, 2**115))))
connection.close()
def test_document_prefix_uses_primary_key_range_and_excludes_neighbor_codes(self):
connection = database(self.base / "paths.sqlite3")
prefix = "documents/1/"
upper = prefix_upper_bound(prefix)
plan = connection.execute(
"EXPLAIN QUERY PLAN SELECT relative_path FROM files WHERE root_id='raw' AND relative_path>=? AND relative_path<?",
(prefix, upper),
).fetchone()[3]
self.assertIn("USING COVERING INDEX sqlite_autoindex_files_1", plan)
connection.executemany(
"INSERT INTO files(root_id,relative_path,status,last_seen_run) VALUES('raw',?,'hashed','run')",
[(f"documents/{code}/a",) for code in range(20000)]
+ [("documents/1x/a",)],
)
self.assertEqual([item["path"] for item in raw_file_rows(connection, "1")], ["documents/1/a"])
connection.close()
def test_inventory_rejects_a_concurrent_writer_and_records_verifier_hash(self):
output = self.base / "artifacts"
output.mkdir(mode=0o700)
lock_path = output / ".inventory.lock"
descriptor = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
try:
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
with self.assertRaisesRegex(InventoryError, "another inventory run"):
verify_corpus(self.raw, None, output)
finally:
os.close(descriptor)
result = verify_corpus(self.raw, None, output)
self.assertEqual(result["status"], "complete")
self.assertRegex(result["verifier_sha256"], r"^[a-f0-9]{64}$")
def test_removed_file_is_reported_on_resume(self):
payload = self.raw / "payload.txt"
payload.write_text("payload", encoding="utf-8")
output = self.base / "artifacts"
self.assertEqual(verify_corpus(self.raw, None, output)["status"], "complete")
payload.unlink()
result = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(result["input_changes"]["raw"]["removed"], 1)
self.assertIn("removed_file", {item["kind"] for item in result["finding_examples"]})
def test_added_file_is_reported_on_resume(self):
output = self.base / "artifacts"
self.assertEqual(verify_corpus(self.raw, None, output)["status"], "complete")
(self.raw / "extra.txt").write_text("extra", encoding="utf-8")
result = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(result["input_changes"]["raw"]["added"], 1)
def test_manifest_error_missing_document_and_extra_directory_are_reported(self):
connection = sqlite3.connect(self.raw / "manifest.sqlite3")
connection.execute("INSERT INTO documents VALUES (1, ?, 'now')", ("a" * 64,))
connection.execute("INSERT INTO errors VALUES (6, 'fetch failed', 'now')")
connection.commit()
connection.close()
(self.raw / "documents" / "999").mkdir(parents=True)
result = verify_corpus(self.raw, None, self.base / "artifacts")
kinds = {finding["kind"] for finding in result["finding_examples"]}
self.assertEqual(result["status"], "failed")
self.assertIn("raw_manifest_error", kinds)
self.assertIn("missing_raw_document_directory", kinds)
self.assertIn("extra_raw_document_directory", kinds)
def test_empty_raw_and_normalized_manifests_complete_successfully(self):
normalized = self.base / "normalized"
normalized.mkdir()
connection = sqlite3.connect(normalized / "manifest.sqlite3")
connection.execute(
"CREATE TABLE documents (code TEXT PRIMARY KEY, source_sha256 TEXT, schema_version TEXT, normalizer_version TEXT, state TEXT)"
)
connection.commit()
connection.close()
result = verify_corpus(self.raw, normalized, self.base / "artifacts")
self.assertEqual(result["status"], "complete", result["finding_examples"])
self.assertEqual(result["manifest_summary"]["raw"]["documents"], 0)
self.assertEqual(result["manifest_summary"]["normalized"]["documents"], 0)
def test_nonempty_raw_and_normalized_manifests_cross_validate_fragment_sources(self):
normalized, _ = self._nonempty_corpus()
result = verify_corpus(self.raw, normalized, self.base / "artifacts")
self.assertEqual(result["status"], "complete", result["finding_examples"])
self.assertEqual(result["manifest_summary"]["raw"]["documents"], 1)
self.assertEqual(result["manifest_summary"]["normalized"]["fragments"], 1)
def test_raw_normalized_source_mismatch_is_reported(self):
normalized, _ = self._nonempty_corpus()
manifest = sqlite3.connect(normalized / "manifest.sqlite3")
manifest.execute("UPDATE documents SET source_sha256=? WHERE code='1'", ("0" * 64,))
manifest.commit()
manifest.close()
result = verify_corpus(self.raw, normalized, self.base / "artifacts")
kinds = {item["kind"] for item in result["finding_examples"]}
self.assertEqual(result["status"], "failed")
self.assertIn("normalized_source_hash_mismatch", kinds)
self.assertIn("document_source_manifest_mismatch", kinds)
def test_fragment_text_hash_mismatch_is_reported(self):
normalized = self.base / "normalized"
fragment_rel = "documents/1/editions/2/ru/fragments.json"
fragment_path = normalized / fragment_rel
fragment_path.parent.mkdir(parents=True)
fragment = {
"id": "document:1:edition:2:lang:ru:fragment:1",
"document_code": "1",
"edition_code": "2",
"language": "ru",
"position": 1,
"text": "text",
"text_sha256": "0" * 64,
"source_path": "documents/1/source.txt",
"source_sha256": "1" * 64,
}
fragment_path.write_text(json.dumps([fragment]), encoding="utf-8")
content_path = fragment_path.with_name("content.txt")
content_path.write_text("text\n", encoding="utf-8")
connection = database(self.base / "test.sqlite3")
content_digest = hashlib.sha256(b"text\n").hexdigest()
connection.execute(
"INSERT INTO files(root_id,relative_path,sha256,status,last_seen_run) VALUES('normalized',?,?, 'hashed','run')",
(content_path.relative_to(normalized).as_posix(), content_digest),
)
validate_fragment_file(
connection,
"run",
normalized,
fragment_path,
fragment_rel,
hashlib.sha256(fragment_path.read_bytes()).hexdigest(),
{"documents/1/source.txt": "1" * 64},
)
kinds = {row[0] for row in connection.execute("SELECT kind FROM findings WHERE run_id='run'")}
connection.close()
self.assertIn("fragment_text_checksum_mismatch", kinds)
if __name__ == "__main__":
unittest.main()
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Validate the shape and repository allowlist of a prepared release manifest."""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
SERVICES = {"backend", "opensearch"}
COMMIT_RE = re.compile(r"[0-9a-f]{40}\Z")
REPOSITORY_RE = re.compile(
r"[a-z0-9]+(?:[._-][a-z0-9]+)*(?::[0-9]+)?"
r"(?:/[a-z0-9]+(?:[._-][a-z0-9]+)*)*\Z"
)
IMAGE_RE = re.compile(r"(.+)@sha256:([0-9a-f]{64})\Z")
def unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]:
result: dict[str, object] = {}
for key, value in pairs:
if key in result:
raise ValueError(f"duplicate JSON key: {key}")
result[key] = value
return result
def read_json(path: Path) -> object:
try:
return json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=unique_object)
except (OSError, UnicodeError, json.JSONDecodeError) as error:
raise ValueError(f"cannot read valid JSON from {path}: {error}") from error
def validate_policy(policy: object) -> dict[str, list[str]]:
if not isinstance(policy, dict) or set(policy) != {"schema_version", "repositories"}:
raise ValueError("policy must contain exactly schema_version and repositories")
if type(policy["schema_version"]) is not int or policy["schema_version"] != 1:
raise ValueError("unsupported policy schema_version")
repositories = policy["repositories"]
if not isinstance(repositories, dict) or set(repositories) != SERVICES:
raise ValueError("policy repositories must contain exactly backend and opensearch")
result: dict[str, list[str]] = {}
for service in sorted(SERVICES):
allowed = repositories[service]
if not isinstance(allowed, list) or not allowed:
raise ValueError(f"policy repository allowlist for {service} must be a non-empty list")
if any(not isinstance(repository, str) or not REPOSITORY_RE.fullmatch(repository) for repository in allowed):
raise ValueError(f"policy contains an invalid repository for {service}")
if len(set(allowed)) != len(allowed):
raise ValueError(f"policy repository allowlist for {service} contains duplicates")
result[service] = allowed
return result
def validate_manifest(manifest: object, policy: object) -> dict[str, object]:
allowed_repositories = validate_policy(policy)
if not isinstance(manifest, dict) or set(manifest) != {
"schema_version", "source_ref", "commit", "images"
}:
raise ValueError("manifest must contain exactly schema_version, source_ref, commit, and images")
if type(manifest["schema_version"]) is not int or manifest["schema_version"] != 1:
raise ValueError("unsupported manifest schema_version")
if manifest["source_ref"] != "refs/heads/main":
raise ValueError("manifest source_ref must be refs/heads/main")
commit = manifest["commit"]
if not isinstance(commit, str) or not COMMIT_RE.fullmatch(commit):
raise ValueError("manifest commit must be a 40-character lowercase Git SHA-1")
images = manifest["images"]
if not isinstance(images, dict) or set(images) != SERVICES:
raise ValueError("manifest images must contain exactly backend and opensearch")
for service in sorted(SERVICES):
image = images[service]
match = IMAGE_RE.fullmatch(image) if isinstance(image, str) else None
if not match:
raise ValueError(f"{service} image must use repository@sha256:<64 lowercase hex>")
repository = match.group(1)
if not REPOSITORY_RE.fullmatch(repository):
raise ValueError(f"{service} image has an invalid OCI repository")
if repository not in allowed_repositories[service]:
raise ValueError(f"{service} image repository is not allowed by policy")
return manifest
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("manifest", type=Path)
parser.add_argument("--policy", required=True, type=Path)
args = parser.parse_args()
try:
manifest = validate_manifest(read_json(args.manifest), read_json(args.policy))
except ValueError as error:
print(f"ERROR: {error}", file=sys.stderr)
return 1
images = manifest["images"]
print(
"PASS release manifest "
f"commit={manifest['commit']} backend={images['backend']} opensearch={images['opensearch']}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
File diff suppressed because it is too large. Load diff
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
set -euo pipefail
workflow=.gitea/workflows/ci.yaml
kaniko_action=.gitea/actions/kaniko-build/action.yml
caddy_action=.gitea/actions/caddy-validate/action.yml
grep -Fq 'runs-on: akyldash-ci' "$workflow"
grep -Fq 'uses: ./.gitea/actions/caddy-validate' "$workflow"
[[ $(grep -Fc 'uses: ./.gitea/actions/kaniko-build' "$workflow") -eq 2 ]]
check_workspace_wiring() {
local workflow_file=$1 caddy_file=${2:-$caddy_action} kaniko_file=${3:-$kaniko_action}
[[ $(grep -Fc 'workspace: ${{ github.workspace }}' "$workflow_file") -eq 3 ]] &&
grep -Fxq ' - ${{ inputs.workspace }}/deploy/production/Caddyfile' "$caddy_file" &&
grep -Fxq ' - --context=dir://${{ inputs.workspace }}' "$kaniko_file" &&
grep -Fxq ' - --dockerfile=${{ inputs.workspace }}/${{ inputs.dockerfile }}' "$kaniko_file" &&
! grep -Fq '/github/workspace' "$caddy_file" "$kaniko_file"
}
if ! check_workspace_wiring "$workflow"; then
echo 'Container actions must receive github.workspace and use it for repository paths.' >&2
exit 1
fi
grep -Fq 'dockerfile: deploy/production/Dockerfile.backend' "$workflow"
grep -Fq 'dockerfile: deploy/production/Dockerfile.opensearch' "$workflow"
grep -Fq 'docker compose --env-file deploy/production/.env.example -f deploy/production/compose.yaml config --quiet' "$workflow"
grep -Fxq ' entrypoint: /usr/bin/caddy' "$caddy_action"
grep -Fxq ' - validate' "$caddy_action"
grep -Fxq ' - --config' "$caddy_action"
grep -Fxq ' - ${{ inputs.workspace }}/deploy/production/Caddyfile' "$caddy_action"
grep -Fxq ' - --adapter' "$caddy_action"
grep -Fxq ' - caddyfile' "$caddy_action"
if grep -Eq 'docker (build|run)([[:space:]]|$)' "$workflow"; then
echo 'workflow must not invoke docker build/run in a job container' >&2
exit 1
fi
check_kaniko_action() {
grep -Fxq ' entrypoint: /kaniko/executor' "$1" &&
grep -Fxq ' - --context=dir://${{ inputs.workspace }}' "$1" &&
grep -Fxq ' - --dockerfile=${{ inputs.workspace }}/${{ inputs.dockerfile }}' "$1" &&
grep -Fxq ' - --destination=${{ inputs.destination }}' "$1" &&
grep -Fxq -- ' - --no-push' "$1" &&
grep -Fxq -- ' - --no-push-cache' "$1"
}
if ! check_kaniko_action "$kaniko_action"; then
echo 'Kaniko action must define its entrypoint, build arguments, and no-push flags.' >&2
exit 1
fi
without_entrypoint=$(mktemp)
without_no_push=$(mktemp)
without_workspace=$(mktemp)
wrong_workspace=$(mktemp)
trap 'rm -f "$without_entrypoint" "$without_no_push" "$without_workspace" "$wrong_workspace"' EXIT
sed '/^ entrypoint: \/kaniko\/executor$/d' "$kaniko_action" >"$without_entrypoint"
if check_kaniko_action "$without_entrypoint"; then
echo 'CI isolation test accepted a Kaniko action without its executor entrypoint.' >&2
exit 1
fi
sed '/^ - --no-push$/d' "$kaniko_action" >"$without_no_push"
if check_kaniko_action "$without_no_push"; then
echo 'CI isolation test accepted --no-push-cache without standalone --no-push.' >&2
exit 1
fi
sed '/workspace: \${{ github.workspace }}/d' "$workflow" >"$without_workspace"
if check_workspace_wiring "$without_workspace"; then
echo 'CI isolation test accepted container actions without workflow workspace inputs.' >&2
exit 1
fi
sed 's#${{ inputs.workspace }}#/github/workspace#g' "$kaniko_action" >"$wrong_workspace"
if check_workspace_wiring "$workflow" "$caddy_action" "$wrong_workspace"; then
echo 'CI isolation test accepted a fixed Kaniko workspace root.' >&2
exit 1
fi
grep -Fq 'FROM ghcr.io/osscontainertools/kaniko:v1.28.5-debug@sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412' .gitea/actions/kaniko-build/Dockerfile
grep -Fq 'FROM caddy:2.11.6-alpine@sha256:881bbc60f9986d5ab8e7cfd6cf7e4ef3c9c0439fef2429d035d065577882f028' .gitea/actions/caddy-validate/Dockerfile
if grep -REn 'docker.sock|privileged:|container.options:|^[[:space:]]*volumes:' "$workflow" .gitea/actions/kaniko-build .gitea/actions/caddy-validate; then
echo 'workflow and container actions must not request sockets, privileges, or additional host mounts' >&2
exit 1
fi
echo 'PASS CI container actions use the supplied workspace and remain isolated without socket or publish access'
+5 -2
View File
@@ -1,10 +1,13 @@
#!/usr/bin/env bash
set -Eeuo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
source "$script_dir/../test/run-as-unprivileged.sh"
output=$(mktemp)
trap 'rm -f "$output"' EXIT
if bash "$(dirname "$0")/install-runner.sh" >"$output" 2>&1; then
echo 'Runner installer unexpectedly succeeded without root.' >&2
if run_as_unprivileged bash "$script_dir/install-runner.sh" >"$output" 2>&1; then
echo 'Runner installer unexpectedly succeeded as an unprivileged UID.' >&2
exit 1
fi
grep -qx 'ERROR: run this script as root' "$output"
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
run_as_unprivileged() {
if [[ $EUID -eq 0 ]]; then
if ! command -v setpriv >/dev/null 2>&1; then
echo 'setpriv is required to run guard tests as an unprivileged UID.' >&2
return 127
fi
setpriv --reuid=65534 --regid=65534 --clear-groups -- "$@"
else
"$@"
fi
}