Compare commits

...
Author SHA1 Message Date
agent 2d3b34f34b Merge remote-tracking branch 'origin/main' into fix/rollback-image-digest-reference
CI / tests-and-images (pull_request) Successful in 2m10s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
# Conflicts:
#	CHANGELOG.md
#	docs/operations/CURRENT_STATE.md
2026-10-20 00:35:46 +03:00
agent cc8cc1bce6 Merge pull request 'Add offline release manifest contract' (#49) from feature/release-manifest-contract into main
CI / tests-and-images (push) Successful in 2m8s
Runner smoke test / runner-smoke (push) Successful in 1s
2026-10-19 21:29:33 +00:00
agent a68d3c315e docs: record rollback preflight limitation
CI / tests-and-images (pull_request) Successful in 2m5s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-20 00:27:44 +03:00
agent ba669fad34 Merge remote-tracking branch 'origin/main' into feature/release-manifest-contract
CI / tests-and-images (pull_request) Successful in 2m5s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
# Conflicts:
#	CHANGELOG.md
2026-10-20 00:18:34 +03:00
codex 8f64131131 Merge remote-tracking branch 'origin/main' into fix/rollback-image-digest-reference
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-20 00:15:33 +03:00
codex 802bfd117e test: assert digest rollback fails before side effects
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 3s
2026-10-20 00:13:55 +03:00
agent 2281705c95 Merge pull request 'Make corpus inventory scalable and resumable' (#48) from feature/resumable-corpus-inventory into main
CI / tests-and-images (push) Successful in 2m9s
Runner smoke test / runner-smoke (push) Successful in 1s
2026-10-19 21:10:48 +00:00
codex fc0473e7e2 fix: validate digest-pinned rollback images safely 2026-10-20 00:10:25 +03:00
agent e12a8a919c Document release manifest preparation scope
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 22:07:45 +03:00
agent 37050b1458 Add validated release manifest contract
CI / tests-and-images (pull_request) Successful in 2m7s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 22:02:09 +03:00
agent 6bd4b05af2 Document corpus inventory safety gates
CI / tests-and-images (pull_request) Successful in 2m30s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:39:34 +03:00
agent 0da2bcd82d Lock corpus artifacts during comparison
CI / tests-and-images (pull_request) Successful in 2m31s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:33:52 +03:00
agent 0979b8f544 Reject empty corpus staging comparisons
CI / tests-and-images (pull_request) Successful in 2m33s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:28:27 +03:00
agent 4d106903e1 Fix corpus inventory indexing and provenance
CI / tests-and-images (pull_request) Successful in 2m25s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 19:22:00 +03:00
agent 4b2e5578cd Add resumable corpus inventory verification 2026-10-18 15:29:14 +03:00
agent 0e276ae652 Merge pull request 'ci: isolate image builds on rootless runner' (#47) from feature/autonomous-review-workflow into main
CI / tests-and-images (push) Successful in 2m5s
Runner smoke test / runner-smoke (push) Successful in 2s
2026-10-18 05:58:19 +00:00
codex 554a7ab5ff docs: qualify runner resource verification
CI / tests-and-images (pull_request) Successful in 2m6s
Runner smoke test / runner-smoke (pull_request) Successful in 2s
2026-10-18 08:55:16 +03:00
codex cf99f4532f docs: record successful isolated CI build
CI / tests-and-images (pull_request) Successful in 2m4s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:51:21 +03:00
codex 5d72bb092d fix: pass runner workspace to container actions
CI / tests-and-images (pull_request) Successful in 2m26s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:46:05 +03:00
codex d96fc4d36b ci: set container action entrypoints
CI / tests-and-images (pull_request) Failing after 9s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:37:44 +03:00
codex 436806a834 ci: exercise root guards under unprivileged uid
CI / tests-and-images (pull_request) Failing after 16s
Runner smoke test / runner-smoke (pull_request) Successful in 1s
2026-10-18 08:32:48 +03:00
codex 3a89dc2196 ci: build production images with isolated Kaniko
CI / tests-and-images (pull_request) Failing after 15s
Runner smoke test / runner-smoke (pull_request) Successful in 8s
2026-10-18 08:17:57 +03:00
agent 69c62466ca docs: complete autonomous review gates
Runner smoke test / runner-smoke (pull_request) Successful in 2s
CI / tests-and-images (pull_request) Has been cancelled
2026-10-17 23:13:07 +03:00
agent 36892aef77 docs: define autonomous review workflow
Runner smoke test / runner-smoke (pull_request) Successful in 1s
CI / tests-and-images (pull_request) Has been cancelled
2026-10-17 23:07:12 +03:00
admin ac611cc878 Merge pull request 'Fix rootless Docker HOME for runner' (#45) from fix/rootless-docker-home into main
Runner smoke test / runner-smoke (push) Successful in 3s
CI / tests-and-images (push) Has been cancelled
Reviewed-on: #45
Reviewed-by: agent <agent@vakanaut.ru>
2026-10-14 14:45:57 +00:00
agent 9ec03200fb fix: restart rootless Docker with updated environment
Runner smoke test / runner-smoke (pull_request) Successful in 4s
CI / tests-and-images (pull_request) Has been cancelled
2026-10-14 17:41:24 +03:00
agent 03c5809d69 fix: persist rootless Docker unit environment
Runner smoke test / runner-smoke (pull_request) Successful in 1m0s
CI / tests-and-images (pull_request) Has been cancelled
2026-10-14 11:48:57 +03:00
agent 00922e6c09 fix: make rootless Docker home writable
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 22:18:51 +03:00
admin e6b099cf31 Merge pull request 'Make runner installer safe to resume' (#44) from fix/runner-installer-resume into main
CI / tests-and-images (push) Has been cancelled
Runner smoke test / runner-smoke (push) Has been cancelled
Reviewed-on: #44
Reviewed-by: agent <agent@vakanaut.ru>
2026-10-11 23:22:16 +00:00
agent d8a0646829 Protect runner mount image from CI path replacement
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 02:16:13 +03:00
agent 433c3059b9 Create runner image through a pinned file descriptor
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 02:01:19 +03:00
agent 4110aa54c9 Make runner setup race-safe and resumable
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 01:41:20 +03:00
agent db6ded4075 Harden runner installer path and mount checks
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 01:23:03 +03:00
agent b68421bb5b Make runner installer safe to resume
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 01:16:11 +03:00
admin 223e06ee77 Merge pull request 'Prepare isolated AkylDash infrastructure and repository CI runner' (#43) from feature/infrastructure-deployment-phase-2 into main
CI / tests-and-images (push) Has been cancelled
Runner smoke test / runner-smoke (push) Has been cancelled
Reviewed-on: #43
Reviewed-by: agent <agent@vakanaut.ru>
2026-10-11 21:57:01 +00:00
agent 52027ecf81 Accept empty subordinate ID allocation files
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 00:51:29 +03:00
agent e5a4dab5f3 Fix runner installer ID allocation and guards
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 00:47:19 +03:00
agent e034fdb816 feat(infra): prepare isolated repository CI runner
CI / tests-and-images (pull_request) Has been cancelled
Runner smoke test / runner-smoke (pull_request) Has been cancelled
2026-10-12 00:35:42 +03:00
admin a34f516de8 Merge pull request 'Подготовить воспроизводимую инфраструктуру production VPS' (#42) from feature/infrastructure-documentation-and-bootstrap into main
CI / tests-and-images (push) Has been cancelled
Reviewed-on: #42
Reviewed-by: agent <agent@vakanaut.ru>
2026-10-11 20:22:22 +00:00
48 changed files with 4278 additions and 153 deletions

No files matched your search

+1
View File
@@ -0,0 +1 @@
FROM caddy:2.11.6-alpine@sha256:881bbc60f9986d5ab8e7cfd6cf7e4ef3c9c0439fef2429d035d065577882f028
+16
View File
@@ -0,0 +1,16 @@
name: Validate Caddyfile
description: Run the pinned Caddy validator against the checked-out Caddyfile
inputs:
workspace:
description: Repository workspace path supplied by the workflow
required: true
runs:
using: docker
image: Dockerfile
entrypoint: /usr/bin/caddy
args:
- validate
- --config
- ${{ inputs.workspace }}/deploy/production/Caddyfile
- --adapter
- caddyfile
+1
View File
@@ -0,0 +1 @@
FROM ghcr.io/osscontainertools/kaniko:v1.28.5-debug@sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412
+22
View File
@@ -0,0 +1,22 @@
name: Build image with Kaniko in a rootless container
description: Validate a Dockerfile by building it without publishing the image or cache
inputs:
workspace:
description: Repository workspace path supplied by the workflow
required: true
dockerfile:
description: Dockerfile path relative to the checked-out repository
required: true
destination:
description: Image reference required by Kaniko; the image is not pushed
required: true
runs:
using: docker
image: Dockerfile
entrypoint: /kaniko/executor
args:
- --context=dir://${{ inputs.workspace }}
- --dockerfile=${{ inputs.workspace }}/${{ inputs.dockerfile }}
- --destination=${{ inputs.destination }}
- --no-push
- --no-push-cache
+43 -8
View File
@@ -7,7 +7,7 @@ on:
jobs:
tests-and-images:
runs-on: ubuntu-latest
runs-on: akyldash-ci
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
@@ -19,24 +19,59 @@ jobs:
run: python3 -m unittest discover -s tools/telegram-bot -p 'test_*.py' -v
- name: Run infrastructure backup tests
run: python3 -m unittest discover -s infra/backup -p 'test_*.py' -v
- name: Check root guard UID decisions
run: bash infra/bootstrap/test-root-guard.sh
- name: Check recovery host guard
run: bash infra/bootstrap/test-host-guard.sh
- name: Check verifier helper argument guard
run: bash infra/bootstrap/test-verify-helper.sh
- name: Check host audit root guard
run: bash infra/bootstrap/test-audit-host.sh
- name: Check Compose helper allowlist
run: bash infra/deploy/test-compose-helper.sh
- name: Check production config root guard
run: bash infra/deploy/test-production-config-guard.sh
- name: Check runner config and isolation guards
run: |
bash infra/runner/test-runner-config.sh
bash infra/runner/test-install-runner-idempotency.sh
bash infra/runner/test-install-runner-guard.sh
- name: Check CI container operation isolation
run: bash infra/runner/test-ci-container-builds.sh
- name: Check infrastructure shell syntax
run: |
bash -n infra/bootstrap/bootstrap-ubuntu-24.04.sh infra/bootstrap/host-guard.sh \
infra/bootstrap/verify.sh infra/bootstrap/verify-helper.sh infra/deploy/install-helper.sh \
infra/bootstrap/test-verify-helper.sh
- name: Run rollback helper tests
infra/bootstrap/verify.sh infra/bootstrap/verify-helper.sh infra/bootstrap/audit-host.sh \
infra/deploy/install-helper.sh infra/deploy/compose-helper.sh infra/deploy/test-compose-helper.sh \
infra/deploy/prepare-production-config.sh infra/deploy/test-production-config-guard.sh \
infra/bootstrap/test-audit-host.sh infra/bootstrap/test-root-guard.sh \
infra/bootstrap/test-verify-helper.sh infra/test/run-as-unprivileged.sh \
infra/runner/install-runner.sh infra/runner/install-runner-lib.sh \
infra/runner/act-runner-healthcheck.sh infra/runner/test-runner-config.sh \
infra/runner/test-install-runner-idempotency.sh infra/runner/test-install-runner-guard.sh \
infra/runner/test-ci-container-builds.sh
python3 -c "import ast,pathlib; ast.parse(pathlib.Path('infra/runner/secure-runner-image.py').read_text())"
python3 -c "import ast,pathlib; ast.parse(pathlib.Path('infra/deploy/validate_release.py').read_text())"
- name: Run deployment contract and rollback tests
run: python3 -m unittest discover -s infra/deploy -p 'test_*.py' -v
- name: Validate production Compose and Caddyfile
run: |
secret_file=$(mktemp)
trap 'rm -f "$secret_file"' EXIT
REVIEW_SECRET_FILE="$secret_file" docker compose --env-file deploy/production/.env.example -f deploy/production/compose.yaml config --quiet
docker run --rm --entrypoint caddy -v "$PWD/deploy/production/Caddyfile:/etc/caddy/Caddyfile:ro" caddy:2.11.6-alpine@sha256:881bbc60f9986d5ab8e7cfd6cf7e4ef3c9c0439fef2429d035d065577882f028 validate --config /etc/caddy/Caddyfile --adapter caddyfile
- name: Validate Caddyfile in isolated container action
uses: ./.gitea/actions/caddy-validate
with:
workspace: ${{ github.workspace }}
- name: Build production images
run: |
docker build -f deploy/production/Dockerfile.backend -t akyldash-backend:ci .
docker build -f deploy/production/Dockerfile.opensearch -t akyldash-opensearch:ci .
uses: ./.gitea/actions/kaniko-build
with:
workspace: ${{ github.workspace }}
dockerfile: deploy/production/Dockerfile.backend
destination: akyldash-backend:ci
- name: Build OpenSearch production image
uses: ./.gitea/actions/kaniko-build
with:
workspace: ${{ github.workspace }}
dockerfile: deploy/production/Dockerfile.opensearch
destination: akyldash-opensearch:ci
+24
View File
@@ -0,0 +1,24 @@
name: Runner smoke test
on:
pull_request:
push:
branches: [main]
workflow_dispatch:
jobs:
runner-smoke:
runs-on: akyldash-ci
steps:
- name: Verify isolated runner boundary
run: |
host_uid=$(awk 'NR == 1 {print $2}' /proc/self/uid_map)
test "$host_uid" -gt 0
test ! -S /var/run/docker.sock
test ! -e /run/user/1000/docker.sock
test -z "${DOCKER_HOST:-}"
test ! -e /etc/akyldash/production.env
test ! -e /etc/akyldash/review_secret
test ! -e /opt/akyldash/current
printf 'Isolated runner: version=%s host=%s uid-map=%s\n' \
"$GITEA_ACTIONS_RUNNER_VERSION" "$(hostname)" "$host_uid"
+67 -1
View File
@@ -9,10 +9,19 @@ Python, SQLite и OpenSearch. Production пока не развернут.
[деплой](docs/operations/DEPLOYMENT.md), [backup](docs/operations/BACKUP.md),
[rollback](docs/operations/ROLLBACK.md), [восстановление](docs/operations/RECOVERY.md).
- Работай в feature/fix/refactor-ветке. `main` защищена; изменения вноси через
PR, не сливай его автоматически. Сохраняй незакоммиченные изменения.
PR. DEV не одобряет и не сливает собственный PR. Условный merge выполняет
только REVIEWER по правилам раздела «Автономный workflow PR» ниже. Сохраняй
незакоммиченные изменения.
- SSH alias — `akyldash_vps`; используй только read-only диагностику. Не давай
`codex` общий sudo или доступ к Docker socket. Root-действия требует владелец.
Для Gitea используй `gitea-api`, не читай файлы с учётными данными.
- Репозиторный Gitea Actions runner описан в
[CI_RUNNER.md](docs/operations/CI_RUNNER.md). Регистрация выполняется
владельцем через Settings → Actions → Runners для `admin/akyldash`; токен
вводится интерактивно и не попадает в Git. Runner использует отдельный
rootless Docker daemon; job-контейнеры не получают Docker socket или
production secrets. Не включай production deployment из CI без отдельного
проверенного deploy-helper.
- Локальные проверки: `PYTHONPATH=backend python3 -m unittest discover -s backend -p 'test_*.py' -v` и
`python3 -m unittest discover -s tools/telegram-bot -p 'test_*.py' -v`,
`python3 -m unittest discover -s infra/backup -p 'test_*.py' -v` и
@@ -27,3 +36,60 @@ Python, SQLite и OpenSearch. Production пока не развернут.
для нового хоста.
- После каждого изменения инфраструктуры обновляй
`docs/operations/CURRENT_STATE.md` по фактическим проверкам.
## Автономный workflow PR
- Маршруты ниже указаны из контекста оркестратора/root: оркестратор — thread
`01a144a5-8b95-77c3-b311-9a9afd062ca8`, host `local`; DEV — thread
`01a138f1-3b34-7c91-a9ed-4b43c3c85f1f`, host
`remote-ssh-codex-managed:712b8d4e-24e1-4dd5-92af-85d1fbd26449`, Gitea
`codex` (ID 6); REVIEWER — thread `01a144f9-5b51-7923-8398-61d4a6a98306`,
host `remote-ssh-codex-managed:b61555d7-8d1a-4cfa-988b-c8da4791d426`, Gitea
`agent` (ID 7). Если этот же чат отображает REVIEWER как `local` внутри его
собственного app-server, это self-label и не является маршрутом из контекста
оркестратора.
- DEV принимает только явно зарегистрированную оркестратором человеческую
задачу с полями `task`, `repo`, `branch`, `pr`, `head_sha`, `phase`. Репозиторий
должен быть `admin/akyldash`; работай только в назначенной ветке и PR. Веди
одну задачу/PR за раз и не подхватывай старые PR самостоятельно.
- Допустимые фазы: `initial_review` — реализация и устранение review findings;
`final_review_and_merge` — только после первоначального успешного review,
changelog-коммита и отдельного назначения оркестратора. Не переходи к следующей
фазе без назначения. Если задача приостановлена, останови любые действия;
после resume перепроверь назначение, PR, head/base, checks и review с нуля.
- Повторная доставка одного назначения идемпотентна по ключу
`task`+`pr`+`head_sha`+`stage`: не дублируй commit, push, review-запрос или
merge; сообщи оркестратору о уже выполненном результате. Новый head или stage
образует новое назначение и требует повторной проверки.
- Межчатовое чтение недоступно: оркестратор через central relay сам читает
результаты и выдаёт назначения. Завершай каждый этап отчётом с полями `task`,
`repo`, `branch`, `pr`, `head_sha`, `stage`, `checks`, `findings`,
`next_action`. Не настраивай отдельные webhook, daemon или транспорт.
- DEV исследует, изменяет, проверяет, коммитит и отправляет изменения; REVIEWER
не меняет код и независимо проверяет полный diff от base после каждого push,
включая предыдущие findings. DEV не одобряет и не сливает свой PR.
- Только после первоначального успешного формального review DEV добавляет
значимые изменения в существующий changelog, коммитит и отправляет их в тот же
PR. Затем REVIEWER полностью проверяет итоговый diff. Для docs-only задачи не
придумывай изменения кода; правила версии применяй только если они явно
относятся к этой документационной правке.
- Слияние может выполнить только REVIEWER для зарегистрированной человеческой
задачи, в фазе `final_review_and_merge`, после явного назначения оркестратора.
Непосредственно перед ним REVIEWER подтверждает `admin/akyldash`, PR, base
`main`, ожидаемую ветку, автора PR `codex` (ID 6), неизменность base и head,
причём head должен совпадать с зарегистрированным `head_sha`, включая
`head_commit_id` для merge-запроса. Требуются все успешные required CI/test
checks и gates, отсутствие pending/failed/unknown checks, открытых
findings и активных `REQUEST_CHANGES`, плюс актуальный формальный `APPROVED`
от REVIEWER для точного текущего commit с параметрами `commit_id == head_sha`,
`stale == false`, `dismissed == false`. При изменении head или base review и
проверки выполняются заново. Комментарий с approval не является формальным
review.
Используй только обычный server-side merge с `head_commit_id`; не используй
`force_merge` и не обходи branch protection. Если любой существенный gate,
статус или защита неизвестны либо не проверяются — не сливай PR и сообщи
оркестратору, что именно осталось неизвестным.
- Все Gitea API-вызовы выполняй только через `gitea-api` с личным окружением;
Git используй с обычным credential helper. Не читай credentials из файлов
репозитория. Запускай Git- и API-команды отдельно, без объединения с другими
командами.
+17
View File
@@ -2,6 +2,23 @@
## Не выпущено
- Rollback preflight проверяет digest-pinned image references по сохранённому
image ID и останавливается до изменений deployment; переключение production
на digest и полноценный digest rollback пока не реализованы.
- Infrastructure: добавлен офлайн-контракт release manifest с точной схемой,
digest-ссылками и repository allowlist; проверен Compose override с
`--no-build` как требованием будущего deploy. Публикация, provenance и
production-деплой не включены.
- Добавлен read-only возобновляемый inventory корпуса с полным хешированием,
потоковой сверкой, индексируемыми диапазонами путей и provenance SHA-256
проверяющего; значения `stat` хранятся без ограничения SQLite INTEGER.
Сравнение source/staging сверяет путь, размер и SHA-256, удерживает shared
locks и отклоняет пустые корпуса без документов или фрагментов. Runbook и
`CURRENT_STATE.md` отражают, что полный аудит корпуса ещё выполняется; это
изменение не означает готовность корпуса или production.
- CI actions теперь используют workspace, переданный Gitea, для Caddyfile и Kaniko build context; CI run 69 и smoke run 70 прошли на self-hosted runner.
- Runner installer сохраняет `HOME` и XDG-пути в user-unit `docker.service`, перезапускает активный daemon и проверяет окружение его процесса.
- Runner installer защищает image и mountpoint root-owned home-каталогом, оставляет регистрацию и workspace в runner-owned каталоге, безопасно форматирует образ по открытому FD и проверяет `nosuid,nodev`.
- Backend `0.11.3`: ключ подписи поисковых снимков читается из Docker secret-файла, а не из аргументов процесса. Добавлена базовая инфраструктура VPS, Caddy HTTPS proxy с разделёнными Compose-сетями и CI-проверка конфигурации; публикация образов и production-деплой остаются выключены до проверки runner, registry, backup/restore и ограниченного deploy-helper.
- Frontend `0.1.2`: на планшетной ширине экран поиска переходит в одну колонку, чтобы фильтры, результаты и документ помещались в окно.
- Backend `0.11.2`: смена пароля требует текущий пароль и запрещает оставить прежний, включая временный пароль при первом входе.
+15 -7
View File
@@ -14,18 +14,26 @@ account is intentionally outside the `sudo` and `docker` groups because access
to the Docker socket grants root-equivalent control. Run host operations with
the narrowly required `sudo` command; do not use `docker` as `codex` directly.
```bash
sudo install -d -o root -g root -m 0700 /etc/akyldash
sudo install -o root -g root -m 0600 deploy/production/.env.example /etc/akyldash/production.env
sudo sh -c 'umask 077; openssl rand -hex 32 > /etc/akyldash/review_secret'
sudoedit /etc/akyldash/production.env
```
After reviewing the host audit and confirming these files do not already exist,
the owner can create them from the approved checkout with
`sudo bash infra/deploy/prepare-production-config.sh`. The script refuses to
overwrite existing configuration, installs the tracked env example with mode
`0600`, and generates a random review secret with mode `0600`. It does not
start Compose or create volumes. Review and set `DATA_ROOT` in the protected
env file before deployment.
Set `DATA_ROOT` to the approved data directory and ensure the normalized corpus
is at `$DATA_ROOT/minjust-normalized/`. The first host setup is documented in
[`docs/operations/DEPLOYMENT.md`](../../docs/operations/DEPLOYMENT.md); the
root bootstrap installs packages and creates directories, but does not deploy
the application or alter SSH, firewall, user groups, or sudoers.
the application or alter SSH, firewall, user groups, or sudoers. The restricted
`akyldash-compose` helper allows only root-owned Compose `config` and `status`;
service start/stop remains an owner-run root operation.
The repository Gitea Actions runner is an isolated CI worker described in
[`docs/operations/CI_RUNNER.md`](../../docs/operations/CI_RUNNER.md). It has no
production Docker socket, production env files, or application checkout. CI
does not deploy or change production; all deployment remains owner-run.
## First deployment and upgrades
+34 -15
View File
@@ -8,7 +8,8 @@ printing key contents, then check the connection:
```bash
ssh -F ~/.ssh/config -G akyldash_vps | awk '$1 == "hostname" || $1 == "user" || $1 == "port" || $1 == "identityfile" { print }'
test -r ~/.ssh/akyldash_codex && echo 'SSH key is available'
ssh -F ~/.ssh/config -o BatchMode=yes akyldash_vps 'id; hostname'
ssh -F ~/.ssh/config -o BatchMode=yes akyldash_vps id
ssh -F ~/.ssh/config -o BatchMode=yes akyldash_vps hostname -f
gitea-api GET /version
gitea-api GET /repos/admin/akyldash
```
@@ -21,14 +22,24 @@ checks. Never print a private key, token, or complete environment file.
- SSH alias: `akyldash_vps`, user `codex`, host `206.223.249.118`, port `22`.
- SSH configuration: `~/.ssh/config`.
- Identity file: `~/.ssh/akyldash_codex` (path only; never copy its contents).
- `codex` can run unprivileged read-only diagnostics. The audited account has no
non-interactive sudo and must not be added to `sudo` or the `docker` group.
- Host package installation, service setup, firewall changes, secret creation,
backup creation, recovery apply, and deployment require owner approval and a
reviewed root command. After the owner installs the root-owned repository
checkout, `infra/deploy/install-helper.sh` can grant a fixed read-only
verifier and guarded rollback command through sudoers; neither helper is
installed on the audited VPS.
- `codex` can run unprivileged read-only diagnostics. It has no sudo or Docker
group membership; Docker CLI access to the daemon is denied. Do not add it to
either group.
- Bootstrap is complete. Docker and Compose are active, but no root-owned
repository checkout or restricted helper is installed yet. The owner can run
`infra/bootstrap/audit-host.sh` from the reviewed root-owned checkout to
report Docker containers/volumes, protected-file metadata, UFW rules, ports,
and ISPmanager status without showing secret contents or changing state.
- Host package/service configuration, firewall changes, protected env/secret
creation, backup creation, recovery apply, and production deployment require
a reviewed owner/root command. `infra/deploy/install-helper.sh` grants the
fixed verifier, read-only Compose `config`/`status`, and guarded rollback
commands. It does not grant general sudo or Docker access; it is not
installed on the VPS.
- `infra/deploy/prepare-production-config.sh` is an owner-run root operation.
It refuses to overwrite existing files, copies only the tracked env example,
generates a random review secret, and writes both files with mode `0600`.
It does not start Compose or create volumes.
- Never connect to other VPS hosts or Synology as part of this deployment.
This execution environment is WSL2. The repository is on the Windows-mounted
@@ -47,19 +58,27 @@ limitation and do not retrieve it elsewhere.
environment. Never inspect local credential files, print environment values,
or construct HTTP authorization headers.
- The audited API token can read/write this repository and create issues, but
does not have `read:user` or `read:package`. An API `403` caused by a missing
scope is not evidence that the account itself lacks the permission.
does not have `read:user` or `read:package`. Listing repository Actions
runners returns `403` because the API principal is not the repository owner.
An API `403` caused by missing scope/ownership is not evidence that no runner
exists.
- Protected `main` requires one approval. Do not push directly to `main`, merge
automatically, or start a review. Work on a feature branch and use a PR.
- Gitea Actions and package features are enabled for the repository, but runner
availability and package publishing permissions have not been confirmed.
- Gitea Actions and package features are enabled for the repository. The Actions
task API currently returns no runs, but runner availability and package
publishing permissions have not been confirmed. The repository-level runner
is prepared by `infra/runner/install-runner.sh`, but is not installed or
registered yet. Only a repository owner may obtain its registration token in
`admin/akyldash` Settings → Actions → Runners; do not use organization or
instance tokens and never put the token in Git, a command argument, or a
report. See [CI_RUNNER.md](CI_RUNNER.md).
## Who can do what
| Role | Operations |
|---|---|
| `codex` over SSH | Read-only host diagnostics; inspect service health and public listeners |
| Restricted verifier | Runs the root-owned read-only drift check; not installed on the audited VPS |
| `codex` over SSH | Unprivileged diagnostics; Docker daemon, UFW rules, and protected paths require the root audit script |
| Restricted helpers | Read-only drift check and Compose `config`/`status`, plus guarded rollback; not installed on the audited VPS |
| Restricted rollback helper | Can restore prior local backend/OpenSearch images from a verified backup after an explicit data-compatibility confirmation; not installed on the audited VPS |
| VPS owner/root | Bootstrap, package installation, SSH/firewall, secret setup, recovery, and approval of production changes |
| Gitea repository writer | Feature branch push and PR creation; protected-main merge needs an approval |
+215
View File
@@ -0,0 +1,215 @@
# AkylDash Gitea Actions runner
## Scope and isolation
The deployment follows the official [Gitea 1.25 act_runner guide](https://docs.gitea.com/1.25/usage/actions/act-runner/)
for repository-scoped registration. Gitea warns that access to a Docker daemon
is root-equivalent; the dedicated rootless daemon is used instead of the
production daemon. The pinned binary and checksum are published in the
[official Gitea download directory](https://dl.gitea.com/gitea-runner/0.2.13/).
The runner is reserved to repository `admin/akyldash`, using the token under
`https://git.vakanaut.ru/admin/akyldash/settings/actions/runners`. Do not use
organization or instance runner settings. Pin the official `act_runner`
`0.2.13` x86_64 binary from `dl.gitea.com`; the installer verifies its SHA-256
before placing it on the host.
CI jobs run in `docker.gitea.com/runner-images:ubuntu-24.04` containers with
capacity 1, per-job limits of 2 CPU, 3 GiB RAM and 512 processes, and a 15 GiB
dedicated ext4 loop-backed Docker data filesystem. The runner account's home
directory is root-owned mode `0711`, so CI cannot rename the image or mountpoint
entries. Docker config remains under `.config`; registration state and the
runner workspace live in `.config/act-runner`, which is runner-owned. The
rootless Docker setup and user service use `.config/act-runner` as writable
`HOME`, `.config` as `XDG_CONFIG_HOME`, and
`.config/act-runner/.local/share` as `XDG_DATA_HOME`. This keeps Docker config
and its systemd user unit in the existing `.config` tree without granting
write access to the protected home.
rootless Docker daemon
and runner user are separate from production Docker; `codex` is not added to
sudo or Docker groups. Jobs do not receive the rootless Docker socket, the
production Docker socket, production environment/secrets, or arbitrary host
bind mounts. The runner account has no access to `/etc/akyldash` or the
root-owned application checkout. These are host permission boundaries, not a
claim that untrusted project code is safe; review workflow changes and actions
before merging them.
The runner service restarts on failure and boot. Systemd caps the runner
account's shared slice to 2.5 CPU, 4 GiB RAM and 2048 tasks; the rootless
daemon is separately capped to 2 CPU, 4 GiB and 1024 tasks, while each job has
the limits above. Runner logs go to a rotated file (10 MiB, 7 copies). A
systemd timer checks the runner service and the rootless Docker API each minute;
on failure it restarts the fixed runner service. No UFW changes or public
metrics ports are needed. Runner cache is disabled.
The CI workflow and `runner-smoke.yaml` target the repository runner label
`akyldash-ci`. CI keeps the backend, Telegram, backup/restore, deployment,
bootstrap, runner, Compose, and Caddy checks. Caddy validation runs in a pinned
container action with explicit `/usr/bin/caddy` entrypoint. The workflow passes
Gitea's `${{ github.workspace }}` value into each container action; Caddy and
Kaniko use that supplied path for repository files instead of assuming a
container workspace location. Both production
Dockerfiles are built in a separate pinned Kaniko container action with explicit
`/kaniko/executor` entrypoint using the runner's rootless container engine; the
job container receives no Docker socket. Kaniko uses `--no-push` and
`--no-push-cache`, so CI validates the full backend and OpenSearch builds but
does not publish images or export build output. The two builds run sequentially
within one job. CI run 69 completed both builds on the runner. Actual per-action
resource-limit enforcement and peak memory or disk usage were not independently
measured or verified. No registry credentials,
production secrets, elevated container options, or arbitrary host mounts are configured.
The Kaniko action uses the `linux/amd64` v1.28.5-debug manifest
`sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412`,
listed in the [official GitHub Packages entry](https://github.com/osscontainertools/kaniko/pkgs/container/kaniko/1271366984?tag=v1.28.5-debug).
The pinned Caddy image digest remains the same as the former `docker run`
validation. act_runner starts these container actions; they do not expose its
rootless Docker socket to workflow steps.
Production deployment is not automated.
## Installation and registration
Review the root script and host audit output before running it. It installs
`uidmap`, `dbus-user-session`, `curl`, `logrotate`, and Docker's rootless extras;
creates the dedicated system user and selects the first free, shared 65536-ID
range from `/etc/subuid` and `/etc/subgid`; reserves 15 GiB
for runner images/workspaces; and configures systemd services. It starts only
the separate rootless Docker daemon; the runner and health timer stay disabled
until repository registration. It does not change
SSH/UFW, production Docker configuration, application data, or production
secrets. It can be rerun after partial setup: it validates and reuses an
existing runner account, subordinate ID mappings, image, and mount unit. An
existing image must be exactly 15 GiB and contain ext4; a mismatch stops the
installer without resizing or formatting it. The runner account must retain
a non-root system UID below `UID_MIN`, its expected home, primary group, and
sole supplementary group. After validating the account, the installer locks
its home directory to root ownership and mode `0711`; it refuses to rerun while
the runner service is active, so it cannot interrupt a job. The mount
is started only when it is not already active, and an active mount must be the
expected ext4 loop device backed by the runner image with both `nosuid` and
`nodev` active. Runner-home directories and user configuration files are
created in runner-owned subdirectories. Root creates only the first `.config`
entry beneath the already locked home; subsequent directory and config writes
run as `akyldash-runner`. The installer rejects static symlinks and validates
ownership and mode after setup. This also normalizes the `0755` intermediate
directories left by the first installer version. Conflicting files or state
stop the installer for manual inspection. A new image requires at least 25 GiB
free on `/`. Image creation opens each parent component without following
symlinks, creates the image exclusively relative to the opened directory, and
passes the open file descriptor to `mkfs.ext4`; validation checks the same inode
and rejects path replacement. Systemd mounts the image under the root-owned home
entry, which the runner cannot replace after validation.
If the reviewed checkout is not present at the inventoried path, first verify
that the target is absent and clone the reviewed `main` there. Do not overwrite
an existing checkout.
```bash
sudo test ! -e /opt/akyldash/current
sudo git clone --branch main \
--single-branch https://git.vakanaut.ru/admin/akyldash.git \
/opt/akyldash/current
```
From that root-owned reviewed checkout on the VPS:
```bash
sudo bash /opt/akyldash/current/infra/runner/install-runner.sh
```
Then, as a repository owner, open repository **Settings → Actions → Runners**,
create a repository-level runner, and copy its registration token locally. Do
not send the token in chat or save it in the checkout. Register interactively
in a terminal so it is not placed in shell history or process arguments:
```bash
sudo -u akyldash-runner env HOME=/var/lib/akyldash-runner/.config/act-runner \
/usr/local/bin/act_runner \
--config /etc/akyldash-runner/config.yaml register
```
Enter `https://git.vakanaut.ru/`, the token, runner name `akyldash-ci-vps-01`,
and label `akyldash-ci:docker://docker.gitea.com/runner-images:ubuntu-24.04`
when prompted. The config sets the label and capacity; verify them in the
repository runner settings. The token is saved only in the mode-0700 runner
runtime directory, outside Git. Start/enable the runner after registration:
```bash
sudo systemctl enable --now akyldash-act-runner.service
sudo systemctl enable --now akyldash-runner-healthcheck.timer
sudo systemctl status akyldash-act-runner.service
sudo systemctl status akyldash-runner-healthcheck.timer
```
The interactive command's terminal behavior should be confirmed before pasting
the token. If it echoes input, stop and use the repository settings to revoke
that token; do not put it in command-line arguments or environment variables.
## Validation
1. Confirm the runner is **Online** in the repository runner settings, with the
expected name and `akyldash-ci` label.
2. Confirm **Runner smoke test** runs for the open PR (or dispatch it manually).
It checks that production
sockets, files, environment secrets, and writable application paths are not
present in the job container.
3. Confirm the run reports the expected runner, succeeds, and the runner
service and health timer remain active.
4. Inspect the `CI` workflow run and confirm Caddy plus both Kaniko actions pass
on `akyldash-ci`. The Kaniko jobs are intentionally build-only: publishing
images requires a separately reviewed registry workflow and credentials.
Do not add production deploy workflows or secrets to this runner.
## Upgrade
Update `runner_version` and its expected official SHA-256 together in
`infra/runner/install-runner.sh`, review upstream release compatibility with
the installed Gitea, and open a PR. Do not replace the binary while a job is
active. Stop the runner, install/verify the new binary, then start it and
confirm Online plus a smoke run. Preserve
`/var/lib/akyldash-runner/.config/act-runner/.runner`.
## Recovery and rollback
The runner is disposable: production data and secrets are not on its filesystem.
For a runner process failure, inspect `systemctl status` and
`journalctl -u akyldash-act-runner.service`; restart the fixed runner service.
If installation stops after creating or enabling the mount unit, inspect the
image and mount with `stat`, `blkid`, `findmnt`, and `losetup`, then rerun the
reviewed installer only when the image is 15 GiB ext4 and the active mount is
backed by it. The installer recognizes that state and does not format or
restart the mounted filesystem. If size, filesystem, unit contents, or backing
device differ, stop and resolve the state manually; never format the existing
image as a retry.
For lost/corrupt registration state, stop the service and obtain a fresh
repository-level token from Settings → Actions → Runners. Preserve the 15 GiB
data image and state for diagnosis; re-register only after the repository
owner confirms/revokes the old runner in Gitea. The images/cache can be rebuilt
from the workflow. Never format or remove an existing runner store as an
automatic recovery step.
## Status
The owner reports that the repository-scoped runner is registered, appears
Idle in Gitea, and its rootless Docker daemon works; the report date was not
recorded. Gitea status checked 2026-10-18 showed smoke run 62 successful and CI
run 61 pending on head `69c62466`. That workflow requested `ubuntu-latest`,
while the smoke workflow used `akyldash-ci`; the label difference is a
plausible explanation for the pending status, but the runner inventory API
returned 403 and the scheduler's reason was not independently confirmed.
On head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run
64 succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
with `exec: "validate": executable file not found`; Kaniko was skipped and smoke
run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the Caddy action
looked for `/github/workspace/deploy/production/Caddyfile`, which was not
mounted at that fixed path; Kaniko was skipped and smoke run 68 succeeded. On
head `5d72bb09`, CI run 69 and smoke run 70 succeeded. CI validated the Caddyfile
and built the backend and OpenSearch images using `${{ github.workspace }}`.
Actual per-action resource-limit enforcement and peak usage were not
independently measured or verified. Image publishing, host-wide resource
metrics, and production deployment remain unverified. Root-guard tests
use `setpriv` for negative cases when jobs start as UID 0 and separately test
the host-audit guard with simulated UID 0 and UID 65534; the real host audit,
production config setup, and runner installer are never executed as root by
those tests. No VPS changes are part of this PR.
+131
View File
@@ -0,0 +1,131 @@
# Corpus inventory and production handoff
The repository contains a read-only verifier for the ignored local corpus at
`data/minjust-cbd/` and `data/minjust-normalized/`. It does not download,
normalize, copy, index, or modify source files. Keep inventory artifacts outside
the repository and both source trees.
## Run the inventory
From the repository root, create a private artifact directory and run:
```bash
artifact_dir=$(mktemp -d /tmp/akyldash-corpus-audit.XXXXXXXX)
python3 infra/deploy/verify_corpus.py \
--raw-root data/minjust-cbd \
--normalized-root data/minjust-normalized \
--output-dir "$artifact_dir"
```
The tool writes `result.json`, `inventory.sqlite3`, and a persistent lock file
into that directory. Keep the directory private: the inventory contains source
paths and checksums. A second run against the same artifact directory is
rejected while a scan is active. Exit
status `0` means the scan and structural checks completed without errors; `1`
means findings were recorded or the run is incomplete; `2` means the arguments
or artifact path were invalid. Read `result.json` to distinguish those states.
Warnings, including documents with zero text fragments, are reported
separately and do not turn the scan into a failure.
For an interrupted run, use the same artifact directory and add `--resume`.
Resume rehashes every source file instead of trusting cached timestamps, then
reports added, changed, and removed paths relative to entries already stored
in the inventory. After resuming an interrupted first pass, those counters are
relative to the partial saved inventory; the full rehash and final checks still
cover every path. A successful result therefore describes the latest complete
pass. Run against a quiescent corpus; do not normalize, fetch, or copy files
into either source tree during the scan. Do not point `--output-dir` inside
either source tree or through a symlink. The verifier rejects such paths before
creating or changing them; it also flags symlinks, reparse points, hard links,
unsupported file types, and files that change while being read.
The inventory database schema is versioned. Resume requires the same root
metadata and verifier SHA-256 recorded in `result.json`; start a new artifact
directory after changing the verifier. `entries` counts files plus directories;
`files` and `directories` are separate counts. Document-prefix checks use
indexed path ranges, and stale-path reporting streams rows from SQLite.
The audit checks SQLite integrity, manifest/document directory agreement,
normalized schema and normalizer versions, source inventory checksums,
fragment identities and text hashes, required output files, and text assembled
from fragments. It does not establish the legal authority, completeness, or
semantic correctness of MinJust content.
## Compare source and staging before transfer
Run a complete inventory of the approved normalized source and the copied
staging tree into separate private artifact directories, using
`--normalized-root` for each. Require `status: complete` and
`verification_status: passed` in both results. The comparison command also
requires each manifest to contain at least one document and each normalized
tree to contain at least one fragment; an empty source or staging tree is a
failure.
```bash
python3 infra/deploy/compare_corpus_inventories.py \
"$source_artifacts/result.json" "$staging_artifacts/result.json"
```
The checker streams every relative path, size, and SHA-256 from both SQLite
inventories, and requires the same verifier/schema version and different
normalized roots. It opens both existing private `.inventory.lock` files
read-only and holds shared nonblocking locks through comparison; a missing,
unsafe, or busy lock fails closed. A zero exit status confirms a non-empty
staging tree matches the source. Keep both trees quiescent during their scans
and comparison. This is an equality check only; the owner still reviews the
code `6` disposition and approves the transfer destination.
## Current data disposition
As measured on 2026-10-18, both local manifests contain 209,958 documents and
pass SQLite `quick_check`. All normalized manifest entries are `success` at
schema `1` and normalizer `1.0.0`. The raw manifest records one fetch error for
code `6`, which is absent from its `documents` table; this does not prove that
the source document is absent. On 2026-10-18, bounded calls through the
repository's rate-limited MinJust client received HTTP `404` for
`GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and
`CheckAvailable`. The listing call failed before returning a code list, and
both document calls failed identically. The repeated `404` is consistent with
the OpenData endpoint being unavailable or its route having changed, but does
not identify which. It is not evidence that code `6` is absent. The [public
document listing page](https://cbd.minjust.gov.kg/list-docs/kg) was reachable,
but it does not confirm this numeric API code. Do not describe the corpus as
fully verified or silently waive this error. Recheck the official listing and
API when the endpoint responds, then record an explicit owner disposition.
From the VPS, a metadata-only HTTPS check to `CheckAvailable` failed at the
HTTP/2 transport layer (`curl` exit 92, `INTERNAL_ERROR`, HTTP status `000`,
TLS verification `0`). An HTTP/1.1 retry was not completed.
The first full pass (run `b4a4556e30cc464d90bc8d75ef1673e9`) stopped before
validation with `Python int too large to convert to SQLite INTEGER` after
2,120,420 raw entries and 397,856 normalized entries. This was an inventory
metadata storage failure, not a corpus finding. The verifier now stores file
identity values as text and records its own SHA-256. A complete rehash with the
updated verifier is required; do not treat the failed pass or progress output
as a completed audit.
A fresh full rehash is now running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`,
run `f4dd5e53020f442bbbecee2d0e4986a6`, verifier SHA-256
`83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`. Check
its private `result.json` for final counters, findings, and manifest comparison.
## Production transfer and indexing gates
Production deployment is not enabled. Before any transfer, the owner must
review the root-only host audit, confirm the target data location is empty or
backed up, resolve the raw code `6` finding, and approve the exact normalized
corpus to use. Transfer only `minjust-normalized/`; keep the raw source archive
offline. Do not preserve the source's Windows/WSL file permission bits. Copy
to a new staging location with owner-approved permissions, verify the copied
manifest and files there, and only then consider placing it under `DATA_ROOT`.
Never overwrite an existing corpus or OpenSearch volume.
The current bulk importer checks individual Bulk API item results, but it can
switch an alias after a zero-fragment input and does not enforce a refresh,
count comparison, or representative-query check before that switch. Until a
separate change adds those gates, do not pass `--alias` during production
loading. Load into a new versioned index only after the corpus transfer is
verified; require a nonzero fragment count, successful item results, an
explicit refresh, a matching OpenSearch document count, and representative
Russian/Kyrgyz queries before an owner performs an alias change. Take and
verify the required backup before any production index or service change.
+176 -56
View File
@@ -1,31 +1,52 @@
# Infrastructure current state
Last audit: 2026-10-11. Source: read-only SSH session as `codex`, local
repository inspection, and Gitea API. This document separates measured facts
from planned configuration.
Read-only host recheck: 2026-10-18. Source: SSH as `codex`, local repository
inspection, and Gitea API. Root-only Docker state, UFW rule details, and
protected file metadata remain unverified.
## VPS facts
- Host: `avprogs.hlab.kz`, `206.223.249.118`, SSH port `22`, user `codex`.
- OS/kernel: Ubuntu Server `24.04.5 LTS`, `6.8.0-142-generic`.
- Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB disk with 93 GiB free.
- OS: Ubuntu Server `24.04.5 LTS`; kernel `6.8.0-142-generic` from the prior
inventory audit.
- Resources: 4 vCPU, 7.7 GiB RAM, 3.8 GiB swap, 99 GiB root filesystem with
84 GiB free on 2026-10-18. Runner storage is a separate 15 GiB ext4 loop
mount with 12 GiB free at that check.
- Network: interface `ens3`, IPv4 `206.223.249.118/32`.
- `codex` is UID/GID 1000, groups `codex` and `users`; `sudo -n true` requires
a password. The current WSL2 session verified the SSH alias using the
explicit user config and approved read-only SSH execution.
a password. Docker CLI cannot access `/var/run/docker.sock` as `codex`.
- SSH service is running. The inspected sshd configuration has public-key and
password authentication enabled and `PermitRootLogin yes`; no SSH changes
were made.
- Docker, Docker Compose, Git, Nginx, Caddy, Certbot, application paths, and
application services are absent. Only base OS services and SSH were running.
- Public listeners observed: TCP `22`; DNS listeners are limited to loopback
(`127.0.0.53` and `127.0.0.54`).
- UFW is reported active in the supplied VPS notes, but the audit account could
not independently read its status. Treat firewall state and rules as
unverified.
- `akyldash.kg` resolves to Cloudflare IPv4/IPv6 edge addresses from the VPS.
Both HTTP and HTTPS return Cloudflare `521`; the origin is currently
unreachable through the proxy.
- Bootstrap completed. Docker CLI `29.9.0` and Compose `5.6.0` were rechecked
over SSH on 2026-10-18; containerd `2.4.1` and runc `1.5.2` are from the
prior package audit. The Docker daemon API remains root-only.
- `/etc/docker/daemon.json` has the expected `local` log driver and `10m` / `3`
rotation limits. Bootstrap paths exist with root ownership: `/opt/akyldash`
mode `0755`, `/etc/akyldash` and `/var/backups/akyldash` mode `0700`, and
`/var/lib/akyldash/data` mode `0750`.
- `/opt/akyldash/current`, verifier/rollback binaries, and the sudoers file are
absent. `codex` has not received sudo or Docker-group access. Metadata for
`/etc/akyldash/production.env` and `/etc/akyldash/review_secret` could not be
checked because access is denied. The target
`/var/lib/akyldash/data/minjust-normalized` and its manifest metadata were
also denied; their presence, ownership, and contents are unknown. No sudo was
used.
- TCP listeners observed: SSH `22` and loopback DNS only; no listener on `1500`,
`80`, or `443`. No ISPmanager package or service unit was found through the
unprivileged checks; root can confirm with the audit script below.
- Owner reports UFW is active and allows TCP `22`, `80`, `443`, and `1500`.
Rule details were not independently read; no firewall changes were made.
- Additional read-only runner checks on 2026-10-12 confirmed `/` is ext4 with
about 92 GiB free at the prior audit, `/usr/bin/dockerd-rootless.sh` exists,
and `git.vakanaut.ru` resolves to `195.209.48.28`. From the VPS,
`https://git.vakanaut.ru/api/v1/version` returned Gitea `1.25.5`; runner
registration and Gitea Actions protocol connectivity are not confirmed.
- `akyldash.kg` resolves to Cloudflare IPv4/IPv6 edge addresses. From the VPS,
HTTPS returned Cloudflare `521` with TLS verification result `0` on
2026-10-18. TCP listeners were SSH `22` and local DNS only, so no origin
listener on `80`/`443` was observed. Origin TLS and Cloudflare Full (strict)
are not verified.
## Application and Gitea facts
@@ -33,35 +54,76 @@ from planned configuration.
- The repository API reports push permission, Actions and packages enabled, and
PRs enabled. `main` is protected and requires one approval; the current API
principal cannot push or merge directly to `main`.
- The Gitea token lacks `read:user` and `read:package`; runner status and package
publishing permissions could not be checked. This change adds
`.gitea/workflows/ci.yaml`; PR #42's run remains `Waiting to run` at the last
status check, so a compatible runner is not confirmed.
- Nothing is running on the VPS. Repository Compose now describes Caddy
`2.11.6` pinned by digest, Python Search API `0.11.3`, SQLite account/review
data, and OpenSearch `3.7.0` with `analysis-icu`.
- The repository runner-list API returns `403` because this API principal is
not the repository owner. The latest verified CI run 75 and smoke run 76 both
succeeded on `main` head `0e276ae6`. The Actions run-list endpoint returned
`403` for this API principal during this recheck, so no newer run state was
independently retrieved. The owner reports the runner is registered and
Idle; repository runner inventory and package publishing permissions remain
unverified.
- No public application listener is observed on TCP `80`/`443`. Docker
container/volume state still requires the root audit. Repository Compose describes
Caddy `2.11.6` pinned by digest, Python Search API `0.11.3`, SQLite
account/review data, and OpenSearch `3.7.0` with `analysis-icu`.
- The target publishes only Caddy on TCP `80`/`443`. Backend and Caddy share a
proxy network; OpenSearch and backend share a separate internal network, and
the backend has no host port. Caddy certificate storage, OpenSearch, and
application data use persistent volumes/bind mounts. TLS, registry digest
deployment for backend/OpenSearch, backup, rollback helper, and systemd
deployment are not configured on the host.
backend has no host port. Caddy certificate storage, OpenSearch, and
application data use persistent volumes/bind mounts.
- The root-owned repository checkout, production env/secret files, Compose
volumes/containers, backup, verifier/rollback helpers, and application are
not confirmed installed. TLS and registry digest deployment for
backend/OpenSearch are not configured.
- The repository has a cold-backup archive/verification tool and a one-shot
restore apply tool that requires a marked replacement host. Backup/restore
tests use synthetic data; neither tool has run against production. The
archive preflight now permits a clean first deployment to skip backup,
requires backup for existing persistent data even without containers,
allows absent named volumes only in explicit corpus-only initial-deployment
mode, and rejects inaccessible/ambiguous mounts and incomplete/running
service containers. Initial deployment docs require a second verified backup
after index and admin initialization.
archive preflight permits a clean first deployment to skip backup, requires
backup for existing persistent data even without containers, allows absent
named volumes only in explicit corpus-only initial-deployment mode, and
rejects inaccessible/ambiguous mounts and incomplete/running service
containers. Initial deployment docs require a second verified backup after
index and admin initialization.
- Recovery bootstrap has an explicit `--recovery-target` mode that refuses the
inventoried production hostname/IP; its host-identity guard has shell tests.
The mode has not been run on a VPS.
- The repository defines a root-owned read-only verifier, rollback helper, and
owner-run installer with exact sudoers command rules. None is installed on
the VPS.
- An isolated local Compose smoke test built the backend/OpenSearch images,
- The repository defines a root-owned read-only verifier, Compose `config` and
`status` helper, guarded rollback helper, and installer with exact sudoers
command rules. None is installed on the VPS. CI contains no production
deployment job; starting/stopping services remains owner-run.
- Rollback preflight validates digest-pinned image references against archived
image IDs and local availability, then refuses before stopping services or
creating a backup because the current flow cannot safely switch immutable
digest references. This behavior is covered by synthetic local tests; it has
not been exercised on the VPS.
- This branch adds an offline release-manifest validator and tests for the
repository allowlist intended for a future root-owned host integration. The
manifest format does not prove source provenance. No registry, host policy,
publish credential, or deployment integration is configured or installed.
- A repository-scoped official `act_runner` `0.2.13` installation is
configured in `infra/runner/`: rootless Docker under `akyldash-runner`, 15 GiB
ext4 loop-backed image/workspace storage, capacity 1, per-job limits (2 CPU,
3 GiB RAM, 512 processes), account slice limits (2.5 CPU, 4 GiB RAM, 2048
tasks), rotated file logs, and a systemd health timer. Jobs target the
`akyldash-ci` label and do not receive either Docker socket. On 2026-10-18,
SSH showed `akyldash-act-runner.service` active under `akyldash-runner`, with
zero restarts and about 115 MiB current memory. Its rootless Docker store is
mounted from `/dev/loop0` as ext4 with `rw,nosuid,nodev`; the 15 GiB mount had
12 GiB free. The separate `akyldash-runner-restart.service` remains failed
with exit status 1 since 2026-10-14; its journal had no entries, so the cause
is unknown. The latest verified main CI run 75 and smoke run 76 succeeded on
head `0e276ae6`. The Actions API returned 403 during this recheck, so no newer
run state was independently retrieved. On
head `3a89dc21`, CI run 63 failed at the host audit root guard and smoke run 64
succeeded. On head `436806a8`, CI run 65 failed in the Caddy container action
with `exec: "validate": executable file not found`; Kaniko was skipped and
smoke run 66 succeeded. On head `d96fc4d3`, CI run 67 failed because the
Caddy action's fixed `/github/workspace/...` path did not exist; Kaniko was
skipped and smoke run 68 succeeded. On head `5d72bb09`, CI run 69 and smoke
run 70 succeeded. CI validated the Caddyfile and built the backend and
OpenSearch images using `${{ github.workspace }}`. Actual per-action
resource-limit enforcement and peak usage were not independently measured or
verified. Image publishing, host-wide resource metrics, and production
deployment remain unverified. No VPS changes are made.
- An isolated local Compose smoke test built backend/OpenSearch images,
reached healthy status for OpenSearch, backend, and Caddy, and returned HTTP
200 for `/review` through Caddy. Its containers, networks, and volumes were
removed afterward. This does not verify VPS startup, DNS, or public TLS.
@@ -69,29 +131,87 @@ from planned configuration.
backend `0.11.3`, frontend `0.1.2`, Telegram bot `0.2.2` (the bot is not
deployed on this VPS).
## Corpus readiness (2026-10-18)
- The local raw archive and normalized corpus are ignored under `data/`; no
corpus files were changed. Their read-only SQLite manifests both pass
`quick_check` and each report 209,958 documents. The raw manifest is
28,295,168 bytes with SHA-256
`f8e3dd2fdec607583ca09cfdb5b76171bb7bfa1e41a0e07ee5aca8481e18e17e`; it has
one fetch error, code `6`, absent from `documents`. The normalized manifest
is 31,571,968 bytes with SHA-256
`1679c20e6d0f3e5a9b50737c006c4417a28ebcaf7467d194d3bf8243538f83bf`; all
entries are `success`, schema `1`, normalizer `1.0.0`.
- The read-only full-file inventory in `/tmp/akyldash-corpus-audit.Mjr6aa`
stopped before validation in run `b4a4556e30cc464d90bc8d75ef1673e9` with
`Python int too large to convert to SQLite INTEGER`. At stop it had processed
2,120,420 raw entries (1,081,428 files, 1,038,992 directories) and 397,856
normalized entries (246,321 files, 151,535 directories). This is a verifier
metadata storage failure, not a corpus finding. A full rehash with the
corrected, versioned verifier is required before readiness can be stated.
- A fresh full rehash is running in `/tmp/akyldash-corpus-audit-final.gtsD87vc`,
run `f4dd5e53020f442bbbecee2d0e4986a6`, with verifier SHA-256
`83dac230099fc91972cd14e86e9f8ec72d13bca08f53d402f13a169e096d98f5`.
`result.json` currently reports `scan_status: running`; its live counters
must be recorded after completion. Sources remain read-only.
- Through the repository client, bounded MinJust OpenData requests for
`GetDocumentListByQuery`, `GetDocument(6)`, control `GetDocument(1096)`, and
`CheckAvailable` each returned HTTP `404`. The list API returned no code list;
the direct requests do not establish whether code `6` exists. The public
listing page is reachable, but it does not map the numeric API code. A
metadata-only `CheckAvailable` request from the VPS failed in HTTP/2 with
`curl` exit 92 (`INTERNAL_ERROR`), HTTP status `000`, and TLS verification 0;
the HTTP/1.1 retry was not completed. Treat the code `6` disposition and
full corpus readiness as unresolved.
- On the VPS, read-only `stat` of `/var/lib/akyldash/data/minjust-normalized`
and its manifest was denied. Production corpus presence and metadata remain
unknown. No corpus was transferred and no production service, Docker socket,
runner policy, or VPS configuration was changed.
## Checks not completed
- SSH completed with the user config explicitly selected; plain SSH was blocked
by permissions on a system config include. The private key was not read.
- UFW rules require root and remain unverified.
- Docker Engine version from the server API, container and volume inventory,
protected env/secret metadata, sudoers, and detailed UFW rules require root.
`infra/bootstrap/audit-host.sh` is provided for an owner-run read-only audit.
- The backup archive/restore tool was tested with synthetic local data; a backup
and restore of real production data remain unverified because production is
not deployed.
- Production instructions now use `/etc/akyldash/production.env` and root-run
Docker commands, consistent with the `codex` account's restricted access.
Initial deployment and upgrade backup ordering are documented; no production
host was changed for this PR.
- The Cloudflare edge response was checked, but origin DNS configuration,
Cloudflare Full (strict), certificate validity, Actions runner availability,
registry publishing, and production deployment have not been tested.
- No packages or services were installed or changed on the VPS.
- Cloudflare edge response was checked; origin DNS target, origin certificate,
Full (strict), registry publishing, host-wide runner metrics, and production
deployment have not been verified. The
owner-reported Idle state has not been independently checked through runner
inventory because the API returned 403.
- This audit made no host changes. No package or service was installed by this
audit; the owner performed the initial bootstrap beforehand.
## Next owner actions
1. Review and run the root bootstrap on the dedicated host.
2. Review SSH/password/root policy and firewall rules separately; this change
did not touch them.
3. Set up DNS, origin certificate, reverse proxy, protected secret files, and
initial corpus/index.
4. Confirm a Gitea Actions runner and package permissions, then test CI and
backup/restore before considering automated production deploy.
1. Install the approved root-owned checkout after its PR is reviewed, then run
`sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh` and share
the output. The script prints no secret values and does not mutate the host.
2. After reviewing the audit output and confirming the target files are absent,
run
`sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh`;
it refuses to overwrite existing files. Set the approved `DATA_ROOT` and
place the approved corpus/index and Cloudflare origin configuration. Do not
deploy until data/service state and the required backup path are reviewed.
3. Run `sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh` after
reviewing the checkout; it grants only fixed verification, Compose
`config`/`status`, and guarded rollback commands.
4. After merging the reviewed PR, confirm a subsequent CI and smoke run on
`akyldash-ci`. Image publishing and production deployment remain manual and
are not part of CI.
5. Test backup/restore and rollback only with synthetic data in an isolated
environment before any production deployment.
## Change in this PR
Adds a read-only, resumable full-file corpus inventory and normalized-data
cross-check, a source/staging comparator requiring nonzero documents and
fragments plus held shared artifact locks, synthetic regression tests, and an
operator handoff runbook. CI's
existing `infra/deploy` unittest discovery includes the new tests. No workflow,
runner, VPS, production data, or Docker settings are changed. The first real
corpus scan stopped on an integer-storage bug; full validation with the
corrected verifier is still required. The source API's code `6` outcome remains
unconfirmed.
+98 -34
View File
@@ -2,37 +2,54 @@
## Current status
Production deployment is not enabled or tested on the VPS. An isolated local
Compose smoke test passed; the VPS has no Docker, Git,
application data, proxy, registry pull configuration, backup process, or
restricted verification/rollback helpers. Do not deploy until the owner has
completed the root bootstrap, provided a protected secret file and a reviewed
proxy/firewall configuration, and a backup and restore have been tested.
Production deployment is not enabled or tested on the VPS. The owner completed
the root bootstrap: Docker/Compose are active and target directories exist.
The `codex` account cannot access the Docker socket. No root-owned checkout or
restricted helpers are installed; protected env/secret metadata and Docker
container/volume state remain unverified because they require root.
The unprivileged audit sees no listener on `80`, `443`, or `1500`; Cloudflare
still returns `521`. Do not deploy until the owner has reviewed root-only host
state, supplied protected configuration and the approved corpus/index, and
backup/restore have been tested.
## First host preparation
1. Review the current inventory and state, then have the owner run
`sudo bash infra/bootstrap/bootstrap-ubuntu-24.04.sh` from a verified copy
of this branch on the VPS. The script installs Docker Engine/Compose and Git,
sets Docker log rotation, and creates root-owned paths. It refuses other OS
versions and does not change SSH, UFW, sudoers, or user groups. Use
`--recovery-target` only on a replacement host; it refuses the known
production host identity.
2. Install the approved repository checkout at `/opt/akyldash/current`,
root-owned and not group/world-writable. Copy
`deploy/production/.env.example` to `/etc/akyldash/production.env`, set
`DATA_ROOT`, and configure
`/etc/akyldash/production.env` plus `/etc/akyldash/review_secret` outside
Git. The secret file must be root-readable only.
3. After reviewing the root-owned checkout, run
`sudo bash infra/deploy/install-helper.sh`. It installs the backup reader,
read-only verifier, and rollback helper with exact sudoers command rules; it
does not grant general sudo or Docker access.
4. Confirm DNS points through Cloudflare to this origin, then have the owner
review firewall rules and open only TCP `80` and `443` in addition to the
existing SSH access. The Compose Caddy service obtains and renews its HTTPS
certificate automatically; Cloudflare Full (strict) still needs owner-side
verification.
1. The initial root bootstrap is complete. After reviewing and merging the
infrastructure PR, the owner must install that approved `main` checkout at
`/opt/akyldash/current` (the path was absent during the unprivileged audit):
```bash
sudo git clone --branch main --single-branch https://git.vakanaut.ru/admin/akyldash.git /opt/akyldash/current
```
The checkout is root-owned by default. Then run
`sudo bash /opt/akyldash/current/infra/bootstrap/audit-host.sh`. This
read-only audit reports Docker state, protected-file metadata, UFW rules,
listeners, and ISPmanager without printing secret contents. Resolve any
existing containers, volumes, or data before proceeding; do not overwrite
or delete them.
2. After confirming the existing env and secret do not exist and reviewing the
tracked defaults, run
`sudo bash /opt/akyldash/current/infra/deploy/prepare-production-config.sh`.
It creates `/etc/akyldash/production.env` from the tracked example and
generates `/etc/akyldash/review_secret`; both are root-owned mode `0600`.
It refuses to overwrite either file and does not start services or create
volumes. Review `DATA_ROOT` and follow the
[corpus inventory and handoff gates](CORPUS_HANDOFF.md) before placing the
approved normalized corpus there.
3. After reviewing the root-owned checkout and audit output, run
`sudo bash /opt/akyldash/current/infra/deploy/install-helper.sh`. It installs
the backup reader, read-only verifier, `akyldash-compose config|status`, and
rollback helper with exact sudoers command rules. The Compose helper has no
start/stop/build operation and grants no direct Docker access. Actual
deployments remain owner-run through the documented Compose commands.
4. Confirm DNS points through Cloudflare to this origin. The owner reports UFW
already allows TCP `80` and `443`; read and document the full rules before
any firewall decision. TCP `1500` is allowed by owner report but has no
observed listener and no ISPmanager package/service was found; the owner
must identify its purpose. Do not change UFW as part of this procedure. The
Compose Caddy service obtains and renews HTTPS certificates automatically;
Cloudflare Full (strict) still needs end-to-end verification.
5. For a clean first deployment, skip backup only when Compose confirms there
are no containers and all persistent mounts are empty. If any data exists,
create and verify a backup before starting a new version. For a preloaded
@@ -49,14 +66,61 @@ proxy/firewall configuration, and a backup and restore have been tested.
sudo chmod 0600 /opt/akyldash/deployed-image.txt
```
Application deployment remains owner-run until registry permissions, a live
Gitea runner, and a controlled deploy procedure are tested. The helper only
supports a guarded application rollback.
Application deployment remains owner-run. The repository-scoped `act_runner`
`0.2.13` service is active under `akyldash-runner` and uses rootless Docker on a
separate loop-backed filesystem. CI run 75 and smoke run 76 succeeded on main
head `0e276ae6`. Runner job containers have neither Docker socket nor production
secrets. CI has no image publishing or production deployment step. The
repository defines sudo helpers for fixed verification, read-only Compose
`config`/`status`, and guarded rollback, but none is installed on the VPS. They
do not start or stop services.
## Release image contract (preparation only)
`infra/deploy/validate_release.py` checks an offline JSON release manifest
against an explicitly supplied repository allowlist. Schema version 1 requires
exactly `schema_version`, `source_ref`, `commit`, and `images`; the source ref
must be `refs/heads/main`, the commit must be a 40-character lowercase Git
SHA-1, and the images must contain exactly `backend` and `opensearch` as
`repository@sha256:<64 lowercase hex>` references. The policy has schema
version 1 and one non-empty repository allowlist for each service. Unknown or
duplicate JSON keys are rejected.
The validator takes the manifest path and an explicit policy path:
```bash
python3 infra/deploy/validate_release.py release.json --policy /etc/akyldash/release-policy.json
```
The policy shape is `{"schema_version":1,"repositories":{"backend":["registry.invalid/akyldash/backend"],"opensearch":["registry.invalid/akyldash/opensearch"]}}`.
These `registry.invalid` names are format examples only; this change does not
create a policy file or approve a registry.
The repository name in the policy must be approved for the registry. No real
registry namespace or root-owned host policy is configured by this change; do
not use the synthetic `registry.invalid` names in tests as production values.
The host integration must load policy only from its fixed root-owned path.
This validator checks format and allowlist only: a manifest can claim a main
ref and commit, but is not proof of source, build provenance, signature, or
trusted publication. CI still has no publish credentials, image publication,
host helper, or deploy trigger.
Compose merges an image override with the existing service `build` entries.
The regression test renders fake digest references and confirms both entries
remain; any future deploy command must use `docker compose up --no-build` so
the host cannot silently build a different image. This does not make digest
rollback available: the current rollback helper expects local tags. A separate
host-side deploy/rollback design, owner-approved registry permissions and
retention, and manual backup/health/rollback end-to-end validation are still
required before publishing or enabling CD.
## CI
`.gitea/workflows/ci.yaml` runs the Python unit suites and builds both
production images on pull requests and pushes to `main`. It does not publish an
image or deploy production. Gitea Actions is enabled at repository level, but a
successful workflow run has not yet confirmed that a compatible runner is
online.
image or deploy production. `.gitea/workflows/runner-smoke.yaml` exercises the
`akyldash-ci` runner on pull requests, pushes to `main`, or manual dispatch.
Both workflows passed on main head `0e276ae6` in runs 75 and 76 respectively.
The repository runner-list endpoint is not available to the current API
principal, so its registration details cannot be independently read through
Gitea API.
+12 -5
View File
@@ -18,11 +18,18 @@ sudo -n /usr/local/sbin/akyldash-rollback \
The helper verifies archive integrity, Compose image references, deployment
file hashes, and the availability of both prior local images. It makes and
verifies a second cold backup, restores only the prior backend/OpenSearch image
tags, waits for Compose healthchecks, and checks HTTPS locally. It refuses to
restore if the current Compose/Caddyfile differs from the saved release. This
does not restore data or automatically undo a release with an unknown schema
change. The application has no formal migration framework; compatibility must
be checked for every change. If it is uncertain, stop and ask the owner.
tags, waits for Compose healthchecks, and checks HTTPS locally. Digest-pinned
references are validated against their recorded local image IDs, but rollback
fails before stopping services or making a backup when either image reference
uses `@sha256:`. The helper cannot safely switch immutable digest references
with the current release-recording flow. Do not retag a digest manually; digest
rollback needs a separately reviewed atomic release-reference mechanism.
It refuses to restore if the current Compose/Caddyfile differs from the saved
release. This does not restore data or automatically undo a release with an
unknown schema change. The application has no formal migration framework;
compatibility must be checked for every change. If it is uncertain, stop and ask
the owner.
The initial owner-run deployment must record its running backend image ID in
`/opt/akyldash/deployed-image.txt` with mode `0600`; the rollback helper updates
@@ -11,8 +11,13 @@
- загрузчик находится в `backend/ingestion/minjust_cbd.py`;
- сырой архив хранится в `data/minjust-cbd` и исключён из Git;
- в манифесте 209 811 успешно загруженных документов;
- 13 кодов остаются в таблице `errors` и отсутствуют в таблице `documents`;
- в read-only проверенном манифесте 209 958 документов;
- один код, `6`, остаётся в таблице `errors` и отсутствует в `documents`;
это не подтверждает отсутствие документа в источнике: при последней
проверке OpenData API возвращал HTTP `404` для списка, `GetDocument(6)`,
контрольного `GetDocument(1096)` и `CheckAvailable`;
- в нормализованном манифесте 209 958 успешных записей, schema `1`,
normalizer `1.0.0`; полная потоковая проверка всех файлов ещё выполняется.
- документ содержит `metadata.json` и каталог `editions`;
- редакция содержит `metadata.json`, `ru.html` и/или `ky.html`, иногда
изображения;
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ $EUID -ne 0 ]]; then
echo "Run this read-only audit as root." >&2
exit 1
fi
echo '== Host =='
hostname -f
grep -E '^(PRETTY_NAME|VERSION_ID)=' /etc/os-release
nproc
free -h
df -h /
echo '== Packages and services =='
dpkg-query -W docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
docker version --format 'Docker Engine {{.Server.Version}}'
docker compose version
containerd --version
runc --version
systemctl is-active docker
python3 -c 'import json; d=json.load(open("/etc/docker/daemon.json")); print("Docker logs:", d.get("log-driver"), d.get("log-opts"))'
echo '== Account and paths =='
id codex
for path in /opt/akyldash /opt/akyldash/current /etc/akyldash /etc/akyldash/production.env /etc/akyldash/review_secret /var/lib/akyldash/data /var/backups/akyldash /usr/local/sbin/akyldash-verify /usr/local/sbin/akyldash-rollback /usr/local/sbin/akyldash-compose /etc/sudoers.d/akyldash-codex; do
if [[ -e $path ]]; then
stat -c '%n owner=%U:%G mode=%a type=%F' "$path"
else
printf '%s absent\n' "$path"
fi
done
echo '== ISPmanager =='
dpkg-query -W -f='${binary:Package} ${Version}\n' 'ispmanager*' 'ispmgr*' 2>/dev/null || true
systemctl list-unit-files --type=service --no-legend | grep -iE 'ispmanager|ispmgr' || echo 'No ISPmanager service unit found.'
echo '== Docker state =='
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
docker volume ls
echo '== Gitea Actions runner (read-only) =='
if id akyldash-runner >/dev/null 2>&1; then
id akyldash-runner
else
echo 'akyldash-runner user absent'
fi
if command -v act_runner >/dev/null 2>&1; then
act_runner --version
else
echo 'act_runner binary absent from PATH'
fi
for path in /etc/akyldash-runner/config.yaml /var/lib/akyldash-runner/.runner /var/lib/akyldash-runner/docker-store.img; do
if [[ -e $path ]]; then
stat -c '%n owner=%U:%G mode=%a type=%F' "$path"
else
printf '%s absent\n' "$path"
fi
done
systemctl status akyldash-act-runner.service --no-pager --full || true
systemctl status akyldash-runner-healthcheck.timer --no-pager --full || true
findmnt /var/lib/akyldash-runner/docker-data || true
echo '== Firewall and listeners (read-only) =='
ufw status verbose
ufw status numbered
ss -lntup
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
source "$script_dir/../test/run-as-unprivileged.sh"
output=$(mktemp)
trap 'rm -f "$output"' EXIT
if run_as_unprivileged bash "$script_dir/audit-host.sh" >"$output" 2>&1; then
echo 'Host audit unexpectedly succeeded as an unprivileged UID.' >&2
exit 1
fi
grep -qx 'Run this read-only audit as root.' "$output"
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
guard=$(mktemp)
output=$(mktemp)
trap 'rm -f "$guard" "$output"' EXIT
awk '
/^if \[\[ \$EUID -ne 0 \]\]; then$/ { capture = 1 }
capture { print }
capture && /^fi$/ { found = 1; exit }
END { if (!found) exit 1 }
' "$script_dir/audit-host.sh" | sed 's/\$EUID/\$TEST_EUID/g' >"$guard" || {
echo 'Could not extract the host audit root guard.' >&2
exit 1
}
if ! grep -Fq 'if [[ $TEST_EUID -ne 0 ]]; then' "$guard"; then
echo 'Extracted guard does not use the simulated effective UID.' >&2
exit 1
fi
if TEST_EUID=0 bash "$guard" >"$output" 2>&1; then
[[ ! -s $output ]] || { cat "$output" >&2; exit 1; }
else
echo 'Host audit root guard rejected simulated UID 0.' >&2
exit 1
fi
if TEST_EUID=65534 bash "$guard" >"$output" 2>&1; then
echo 'Host audit root guard accepted simulated non-root UID.' >&2
exit 1
fi
grep -qx 'Run this read-only audit as root.' "$output"
echo 'PASS host audit root guard accepts UID 0 and rejects non-root UIDs'
+3 -2
View File
@@ -129,13 +129,14 @@ for endpoint in $udp_listeners; do
failed=1
done
expected_sudoers=$'codex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-verify\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-rollback *'
expected_sudoers=$'codex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-verify\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-rollback *\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-compose config\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-compose status'
if [[ -x /usr/local/sbin/akyldash-verify && $(stat -c %U:%a /usr/local/sbin/akyldash-verify) == root:750 && \
-x /usr/local/sbin/akyldash-rollback && $(stat -c %U:%a /usr/local/sbin/akyldash-rollback) == root:750 && \
-x /usr/local/sbin/akyldash-compose && $(stat -c %U:%a /usr/local/sbin/akyldash-compose) == root:750 && \
-f /usr/local/lib/akyldash/archive.py && $(stat -c %U:%a /usr/local/lib/akyldash/archive.py) == root:644 && \
-f /etc/sudoers.d/akyldash-codex && $(stat -c %U:%a /etc/sudoers.d/akyldash-codex) == root:440 && \
$(cat /etc/sudoers.d/akyldash-codex) == "$expected_sudoers" ]]; then
echo 'PASS read-only verifier, rollback helper, and exact sudoers rules present'
echo 'PASS read-only verifier, Compose status/config helper, rollback helper, and exact sudoers rules present'
else
echo 'FAIL restricted helper or sudoers installation missing/different'
failed=1
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env python3
"""Require non-empty completed normalized audits and compare every file row."""
from __future__ import annotations
import argparse
import fcntl
import itertools
import json
import os
import sqlite3
import stat
import sys
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Any
class ComparisonError(ValueError):
pass
@contextmanager
def shared_inventory_lock(result_path: Path, label: str):
lock_path = result_path.absolute().parent / ".inventory.lock"
if not hasattr(os, "O_NOFOLLOW"):
raise ComparisonError(f"{label}: safe inventory lock opens are unsupported")
try:
descriptor = os.open(lock_path, os.O_RDONLY | os.O_NOFOLLOW)
except OSError as error:
raise ComparisonError(f"{label}: cannot open inventory lock: {error}") from error
try:
metadata = os.fstat(descriptor)
if (
not stat.S_ISREG(metadata.st_mode)
or metadata.st_nlink != 1
or stat.S_IMODE(metadata.st_mode) & 0o077
):
raise ComparisonError(f"{label}: inventory lock must be a private regular file")
try:
fcntl.flock(descriptor, fcntl.LOCK_SH | fcntl.LOCK_NB)
except BlockingIOError as error:
raise ComparisonError(f"{label}: inventory is active (lock busy)") from error
yield
finally:
os.close(descriptor)
def read_audit(result_path: Path, label: str) -> tuple[sqlite3.Connection, dict[str, Any]]:
result_path = result_path.absolute()
try:
metadata = result_path.lstat()
if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode):
raise ComparisonError(f"{label}: result.json must be a regular file")
result = json.loads(result_path.read_text(encoding="utf-8"))
except (OSError, UnicodeError, json.JSONDecodeError) as error:
raise ComparisonError(f"{label}: cannot read result.json: {error}") from error
if not isinstance(result, dict):
raise ComparisonError(f"{label}: result.json must contain an object")
if result.get("status") != "complete" or result.get("verification_status") != "passed":
raise ComparisonError(f"{label}: inventory must be complete and passed")
verifier_sha256 = result.get("verifier_sha256")
if not isinstance(verifier_sha256, str) or len(verifier_sha256) != 64:
raise ComparisonError(f"{label}: verifier SHA-256 is missing")
normalized_root = result.get("roots", {}).get("normalized")
if not isinstance(normalized_root, str):
raise ComparisonError(f"{label}: normalized root is missing")
summary = result.get("manifest_summary", {}).get("normalized")
if not isinstance(summary, dict):
raise ComparisonError(f"{label}: normalized manifest summary is missing")
for key in ("documents", "fragments"):
if not isinstance(summary.get(key), int) or summary[key] <= 0:
raise ComparisonError(f"{label}: normalized {key} must be greater than zero")
database_path = result_path.with_name("inventory.sqlite3")
try:
db_metadata = database_path.lstat()
if (
stat.S_ISLNK(db_metadata.st_mode)
or not stat.S_ISREG(db_metadata.st_mode)
or db_metadata.st_nlink != 1
):
raise ComparisonError(f"{label}: inventory.sqlite3 must be a regular unlinked file")
for suffix in ("-wal", "-shm", "-journal"):
if Path(f"{database_path}{suffix}").exists():
raise ComparisonError(f"{label}: SQLite sidecar files are present")
connection = sqlite3.connect(database_path.resolve().as_uri() + "?mode=ro&immutable=1", uri=True)
connection.row_factory = sqlite3.Row
if connection.execute("PRAGMA quick_check").fetchone()[0] != "ok":
raise ComparisonError(f"{label}: inventory database failed quick_check")
manifest_documents = connection.execute(
"SELECT count(*) FROM manifest_docs WHERE root_id='normalized'"
).fetchone()[0]
if manifest_documents != summary["documents"]:
raise ComparisonError(f"{label}: normalized document count does not match inventory")
files, byte_count, unfinished = connection.execute(
"SELECT count(*),coalesce(sum(size),0),"
"sum(CASE WHEN status!='hashed' OR size IS NULL OR sha256 IS NULL THEN 1 ELSE 0 END) "
"FROM files WHERE root_id='normalized'"
).fetchone()
progress = result.get("progress", {}).get("normalized", {})
if unfinished or files != progress.get("files") or byte_count != progress.get("bytes"):
raise ComparisonError(f"{label}: inventory rows do not match completed scan counters")
return connection, {
"root": normalized_root,
"documents": summary["documents"],
"fragments": summary["fragments"],
"files": files,
"bytes": byte_count,
"verifier_sha256": verifier_sha256,
"schema_version": result.get("schema_version"),
}
except sqlite3.Error as error:
raise ComparisonError(f"{label}: cannot validate inventory database: {error}") from error
def compare_inventories(source_result: Path, staging_result: Path) -> dict[str, int]:
with ExitStack() as locks:
locks.enter_context(shared_inventory_lock(source_result, "source"))
locks.enter_context(shared_inventory_lock(staging_result, "staging"))
source_db, source = read_audit(source_result, "source")
try:
staging_db, staging = read_audit(staging_result, "staging")
except Exception:
source_db.close()
raise
try:
if source["root"] == staging["root"]:
raise ComparisonError("source and staging must be different normalized roots")
if source["verifier_sha256"] != staging["verifier_sha256"] or source["schema_version"] != staging["schema_version"]:
raise ComparisonError("source and staging audits must use the same verifier and schema")
if (source["documents"], source["fragments"]) != (staging["documents"], staging["fragments"]):
raise ComparisonError("source and staging document or fragment counts differ")
query = (
"SELECT relative_path,size,sha256 FROM files "
"WHERE root_id='normalized' ORDER BY relative_path"
)
left = iter(source_db.execute(query))
right = iter(staging_db.execute(query))
mismatch_count = 0
for source_row, staging_row in itertools.zip_longest(left, right):
if source_row != staging_row:
mismatch_count += 1
if mismatch_count:
raise ComparisonError(f"source and staging differ in {mismatch_count} inventory rows")
return {"documents": source["documents"], "fragments": source["fragments"], "files": source["files"], "bytes": source["bytes"]}
finally:
source_db.close()
staging_db.close()
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("source_result", type=Path)
parser.add_argument("staging_result", type=Path)
arguments = parser.parse_args()
try:
summary = compare_inventories(arguments.source_result, arguments.staging_result)
except (ComparisonError, OSError) as error:
print(f"FAIL: {error}", file=sys.stderr)
return 1
print(
"PASS: matched "
f"{summary['documents']} documents, {summary['fragments']} fragments, "
f"{summary['files']} files, {summary['bytes']} bytes"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -ne 1 || ( $1 != config && $1 != status ) ]]; then
echo 'Usage: akyldash-compose {config|status}' >&2
exit 2
fi
if [[ $EUID -ne 0 ]]; then
echo 'Run this helper through its fixed sudoers command.' >&2
exit 1
fi
checkout=/opt/akyldash/current
if [[ ! -d $checkout/.git || -L $checkout || $(stat -c %U "$checkout") != root ]]; then
echo 'Refusing Compose access: approved root-owned checkout is unavailable.' >&2
exit 1
fi
unsafe=$(find "$checkout" -xdev \( ! -user root -o -perm /022 \) -print -quit)
if [[ -n $unsafe ]]; then
echo 'Refusing Compose access: checkout contains a writable or non-root-owned path.' >&2
exit 1
fi
compose=(docker compose --env-file /etc/akyldash/production.env -f "$checkout/deploy/production/compose.yaml")
case $1 in
config) "${compose[@]}" config --quiet ;;
status) "${compose[@]}" ps ;;
esac
+6 -4
View File
@@ -10,7 +10,8 @@ checkout=/opt/akyldash/current
rollback_source=$checkout/infra/deploy/rollback.py
backup_source=$checkout/infra/backup/archive.py
verify_source=$checkout/infra/bootstrap/verify-helper.sh
if [[ ! -d $checkout/.git || ! -f $rollback_source || ! -f $backup_source || ! -f $verify_source ]]; then
compose_source=$checkout/infra/deploy/compose-helper.sh
if [[ ! -d $checkout/.git || ! -f $rollback_source || ! -f $backup_source || ! -f $verify_source || ! -f $compose_source ]]; then
echo "Install the approved repository checkout at $checkout first." >&2
exit 1
fi
@@ -43,7 +44,7 @@ if ! command -v visudo >/dev/null 2>&1; then
exit 1
fi
for path in /usr/local/sbin/akyldash-verify /usr/local/sbin/akyldash-rollback /usr/local/lib/akyldash/archive.py /etc/sudoers.d/akyldash-codex; do
for path in /usr/local/sbin/akyldash-verify /usr/local/sbin/akyldash-rollback /usr/local/sbin/akyldash-compose /usr/local/lib/akyldash/archive.py /etc/sudoers.d/akyldash-codex; do
if [[ -L $path ]]; then
echo "Refusing symlink at $path." >&2
exit 1
@@ -54,9 +55,10 @@ install -d -o root -g root -m 0755 /usr/local/lib/akyldash
install -o root -g root -m 0644 "$backup_source" /usr/local/lib/akyldash/archive.py
install -o root -g root -m 0750 "$verify_source" /usr/local/sbin/akyldash-verify
install -o root -g root -m 0750 "$rollback_source" /usr/local/sbin/akyldash-rollback
install -o root -g root -m 0750 "$compose_source" /usr/local/sbin/akyldash-compose
sudoers_file=/etc/sudoers.d/akyldash-codex
expected=$'codex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-verify\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-rollback *'
expected=$'codex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-verify\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-rollback *\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-compose config\ncodex ALL=(root) NOPASSWD: /usr/local/sbin/akyldash-compose status'
created=false
if [[ -e $sudoers_file ]]; then
if [[ $(stat -c %U:%a "$sudoers_file") != root:440 || $(cat "$sudoers_file") != "$expected" ]]; then
@@ -85,4 +87,4 @@ if ! visudo -c -f /etc/sudoers; then
echo "Sudoers validation failed; the newly added rule was removed." >&2
exit 1
fi
echo "Installed read-only verification and guarded rollback helpers; codex receives no general sudo or Docker access."
echo "Installed fixed read-only verification/Compose commands and guarded rollback; codex receives no general sudo or Docker access."
+50
View File
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ $EUID -ne 0 ]]; then
echo 'Run this configuration step as root.' >&2
exit 1
fi
checkout=/opt/akyldash/current
example=$checkout/deploy/production/.env.example
config_dir=/etc/akyldash
env_file=$config_dir/production.env
secret_file=$config_dir/review_secret
if [[ ! -d $checkout/.git || -L $checkout || $(stat -c %U "$checkout") != root || ! -f $example ]]; then
echo 'Refusing configuration: approved root-owned checkout is unavailable.' >&2
exit 1
fi
unsafe=$(find "$checkout" -xdev \( ! -user root -o -perm /022 \) -print -quit)
if [[ -n $unsafe ]] || ! git -C "$checkout" diff --quiet HEAD -- deploy/production/.env.example; then
echo 'Refusing configuration: checkout is writable or .env.example differs from Git.' >&2
exit 1
fi
if [[ ! -d $config_dir || $(stat -c %U:%G:%a "$config_dir") != root:root:700 ]]; then
echo 'Expected /etc/akyldash to be root-owned mode 0700.' >&2
exit 1
fi
if [[ -e $env_file || -L $env_file || -e $secret_file || -L $secret_file ]]; then
echo 'Production env or review secret already exists; refusing to overwrite either file.' >&2
exit 1
fi
env_tmp=$(mktemp "$config_dir/.production.env.XXXXXX")
secret_tmp=$(mktemp "$config_dir/.review_secret.XXXXXX")
trap 'rm -f "$env_tmp" "$secret_tmp"' EXIT
install -o root -g root -m 0600 "$example" "$env_tmp"
openssl rand -hex 32 > "$secret_tmp"
chown root:root "$secret_tmp"
chmod 0600 "$secret_tmp"
mv -n "$env_tmp" "$env_file"
if [[ -e $env_tmp ]]; then
echo 'Production env already appeared; refusing to continue.' >&2
exit 1
fi
mv -n "$secret_tmp" "$secret_file"
if [[ -e $secret_tmp ]]; then
rm -f "$env_file"
echo 'Review secret already appeared; removed the new env file and stopped.' >&2
exit 1
fi
echo 'Created root-owned production.env and review_secret with mode 0600. No services or volumes were changed.'
+45 -8
View File
@@ -19,6 +19,12 @@ ENV_FILE = Path("/etc/akyldash/production.env")
BACKUP_DIR = Path("/var/backups/akyldash")
BACKUP_TOOL = Path("/usr/local/lib/akyldash/archive.py")
APP_SERVICES = {"backend", "opensearch"}
IMAGE_TAG_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]*")
IMAGE_DIGEST_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]*@sha256:[0-9a-f]{64}")
def is_digest_reference(reference: str) -> bool:
return IMAGE_DIGEST_RE.fullmatch(reference) is not None
def image_restore_plan(manifest: dict, configured: dict[str, str]) -> dict[str, tuple[str, str]]:
@@ -34,14 +40,48 @@ def image_restore_plan(manifest: dict, configured: dict[str, str]) -> dict[str,
image_id = image.get("image_id")
if reference != configured.get(service):
raise ValueError(f"Compose image reference changed for {service}; refusing rollback")
if not isinstance(reference, str) or not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._:/-]*", reference):
raise ValueError(f"{service} image reference is not a local tag")
if not isinstance(reference, str) or not (
IMAGE_TAG_RE.fullmatch(reference) or is_digest_reference(reference)
):
raise ValueError(f"invalid image reference for {service}")
if not isinstance(image_id, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id):
raise ValueError(f"invalid previous image ID for {service}")
plan[service] = (image_id, reference)
return plan
def verify_previous_images(plan: dict[str, tuple[str, str]]) -> None:
for image_id, reference in plan.values():
try:
actual = run(["docker", "image", "inspect", "--format", "{{.Id}}", image_id], capture=True)
except subprocess.CalledProcessError as error:
raise ValueError(f"previous image is unavailable: {image_id}") from error
if actual != image_id:
raise ValueError(f"previous image is unavailable: {image_id}")
if is_digest_reference(reference):
try:
referenced_id = run(
["docker", "image", "inspect", "--format", "{{.Id}}", reference], capture=True
)
except subprocess.CalledProcessError as error:
raise ValueError(f"previous digest reference is unavailable: {reference}") from error
if referenced_id != image_id:
raise ValueError(f"previous digest reference does not match its recorded image: {reference}")
def require_tag_rollback(plan: dict[str, tuple[str, str]]) -> None:
if any(is_digest_reference(reference) for _image_id, reference in plan.values()):
raise ValueError(
"rollback of digest-pinned images is not supported safely; no services or image references were changed"
)
def tag_previous_images(plan: dict[str, tuple[str, str]]) -> None:
require_tag_rollback(plan)
for image_id, reference in plan.values():
run(["docker", "image", "tag", image_id, reference])
def run(command: list[str], *, capture: bool = False) -> str:
result = subprocess.run(command, check=True, text=True, capture_output=capture)
return result.stdout.strip() if capture else ""
@@ -96,10 +136,8 @@ def rollback(backup: Path, data_compatible: bool) -> None:
rendered = json.loads(compose("config", "--format", "json", capture=True))
configured = {service: rendered["services"][service]["image"] for service in APP_SERVICES}
plan = image_restore_plan(manifest, configured)
for image_id, _reference in plan.values():
actual = run(["docker", "image", "inspect", "--format", "{{.Id}}", image_id], capture=True)
if actual != image_id:
raise ValueError(f"previous image is unavailable: {image_id}")
verify_previous_images(plan)
require_tag_rollback(plan)
backup_name = f"akyldash-pre-rollback-{dt.datetime.now(dt.timezone.utc):%Y%m%dT%H%M%SZ}.tar.gz"
backup_output = BACKUP_DIR / backup_name
@@ -123,8 +161,7 @@ def rollback(backup: Path, data_compatible: bool) -> None:
compose("up", "--detach")
raise
for image_id, reference in plan.values():
run(["docker", "image", "tag", image_id, reference])
tag_previous_images(plan)
compose("up", "--detach", "--no-build", "--wait", "--wait-timeout", "300")
run(
[
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -euo pipefail
helper=$(dirname "$0")/compose-helper.sh
for action in deploy up down docker; do
if bash "$helper" "$action" >/dev/null 2>&1; then
printf 'FAIL Compose helper accepted %s\n' "$action" >&2
exit 1
fi
done
echo 'PASS Compose helper rejects operations outside its read-only allowlist'
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
source "$script_dir/../test/run-as-unprivileged.sh"
output=$(mktemp)
trap 'rm -f "$output"' EXIT
if run_as_unprivileged bash "$script_dir/prepare-production-config.sh" >"$output" 2>&1; then
echo 'Production config setup unexpectedly succeeded as an unprivileged UID.' >&2
exit 1
fi
grep -qx 'Run this configuration step as root.' "$output"
@@ -0,0 +1,132 @@
import fcntl
import json
import os
import sqlite3
import tempfile
import unittest
from pathlib import Path
from compare_corpus_inventories import ComparisonError, compare_inventories
class CompareCorpusInventoryTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.base = Path(self.temp.name)
self.source = self._artifact("source", "/data/source")
self.staging = self._artifact("staging", "/data/staging")
def tearDown(self):
self.temp.cleanup()
def _artifact(self, name, root, documents=1, fragments=2, rows=None):
directory = self.base / name
directory.mkdir(mode=0o700)
(directory / ".inventory.lock").touch(mode=0o600)
database = directory / "inventory.sqlite3"
connection = sqlite3.connect(database)
connection.execute(
"CREATE TABLE files (root_id TEXT, relative_path TEXT, size INTEGER, sha256 TEXT, status TEXT)"
)
connection.execute("CREATE TABLE manifest_docs (root_id TEXT, code TEXT)")
connection.executemany(
"INSERT INTO manifest_docs VALUES ('normalized',?)",
[(str(code),) for code in range(documents)],
)
if rows is None:
rows = [
("manifest.sqlite3", 16, "a" * 64),
("documents/1/document.json", 24, "b" * 64),
]
connection.executemany(
"INSERT INTO files VALUES ('normalized',?,?,?,'hashed')", rows
)
connection.commit()
connection.close()
result = {
"status": "complete",
"verification_status": "passed",
"schema_version": 2,
"verifier_sha256": "c" * 64,
"roots": {"normalized": root},
"manifest_summary": {
"normalized": {"documents": documents, "fragments": fragments}
},
"progress": {
"normalized": {
"files": len(rows),
"bytes": sum(row[1] for row in rows),
}
},
}
result_path = directory / "result.json"
result_path.write_text(json.dumps(result), encoding="utf-8")
return result_path
def test_nonempty_source_and_staging_inventories_match(self):
self.assertEqual(
compare_inventories(self.source, self.staging),
{"documents": 1, "fragments": 2, "files": 2, "bytes": 40},
)
def test_empty_source_is_rejected(self):
empty = self._artifact("empty-source", "/data/empty-source", documents=0, fragments=0)
with self.assertRaisesRegex(ComparisonError, "source: normalized documents"):
compare_inventories(empty, self.staging)
def test_empty_staging_is_rejected(self):
empty = self._artifact("empty-staging", "/data/empty-staging", documents=0, fragments=0)
with self.assertRaisesRegex(ComparisonError, "staging: normalized documents"):
compare_inventories(self.source, empty)
def test_zero_fragments_are_rejected_even_with_documents(self):
empty = self._artifact("no-fragments", "/data/no-fragments", fragments=0)
with self.assertRaisesRegex(ComparisonError, "normalized fragments"):
compare_inventories(self.source, empty)
def test_different_file_sha_is_rejected(self):
changed = self._artifact(
"changed", "/data/changed",
rows=[("manifest.sqlite3", 16, "a" * 64), ("documents/1/document.json", 24, "d" * 64)],
)
with self.assertRaisesRegex(ComparisonError, "differ in 1 inventory rows"):
compare_inventories(self.source, changed)
def test_busy_source_lock_is_rejected(self):
self._assert_busy_lock(self.source, "source")
def test_busy_staging_lock_is_rejected(self):
self._assert_busy_lock(self.staging, "staging")
def _assert_busy_lock(self, result_path, label):
descriptor = os.open(result_path.parent / ".inventory.lock", os.O_RDWR)
try:
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
with self.assertRaisesRegex(ComparisonError, rf"{label}: inventory is active"):
compare_inventories(self.source, self.staging)
finally:
os.close(descriptor)
def test_missing_lock_is_rejected_without_creating_it(self):
lock = self.source.parent / ".inventory.lock"
lock.unlink()
with self.assertRaisesRegex(ComparisonError, "source: cannot open inventory lock"):
compare_inventories(self.source, self.staging)
self.assertFalse(lock.exists())
def test_symlink_lock_is_rejected(self):
lock = self.source.parent / ".inventory.lock"
lock.unlink()
target = self.base / "external-lock"
target.touch(mode=0o600)
lock.symlink_to(target)
with self.assertRaisesRegex(ComparisonError, "source: cannot open inventory lock"):
compare_inventories(self.source, self.staging)
def test_read_only_lock_can_be_shared(self):
(self.source.parent / ".inventory.lock").chmod(0o400)
self.assertEqual(compare_inventories(self.source, self.staging)["documents"], 1)
if __name__ == "__main__":
unittest.main()
+131
View File
@@ -0,0 +1,131 @@
import json
import os
import shutil
import subprocess
import tempfile
import unittest
from pathlib import Path
from validate_release import validate_manifest
ROOT = Path(__file__).resolve().parents[2]
REPOSITORIES = {
"backend": ["registry.invalid/akyldash/backend"],
"opensearch": ["registry.invalid/akyldash/opensearch"],
}
class ReleaseManifestTests(unittest.TestCase):
def setUp(self):
self.policy = {"schema_version": 1, "repositories": REPOSITORIES}
self.manifest = {
"schema_version": 1,
"source_ref": "refs/heads/main",
"commit": "a" * 40,
"images": {
"backend": "registry.invalid/akyldash/backend@sha256:" + "b" * 64,
"opensearch": "registry.invalid/akyldash/opensearch@sha256:" + "c" * 64,
},
}
def test_accepts_both_policy_pinned_images(self):
self.assertEqual(validate_manifest(self.manifest, self.policy), self.manifest)
def test_rejects_untrusted_ref(self):
self.manifest["source_ref"] = "refs/pull/12/head"
with self.assertRaisesRegex(ValueError, "source_ref"):
validate_manifest(self.manifest, self.policy)
def test_rejects_tag_instead_of_digest(self):
self.manifest["images"]["backend"] = "registry.invalid/akyldash/backend:latest"
with self.assertRaisesRegex(ValueError, "sha256"):
validate_manifest(self.manifest, self.policy)
def test_rejects_empty_image_reference(self):
self.manifest["images"]["backend"] = ""
with self.assertRaisesRegex(ValueError, "sha256"):
validate_manifest(self.manifest, self.policy)
def test_rejects_invalid_commit(self):
self.manifest["commit"] = "not-a-commit"
with self.assertRaisesRegex(ValueError, "commit"):
validate_manifest(self.manifest, self.policy)
def test_rejects_repository_outside_policy(self):
self.manifest["images"]["backend"] = "registry.invalid/other/backend@sha256:" + "b" * 64
with self.assertRaisesRegex(ValueError, "not allowed"):
validate_manifest(self.manifest, self.policy)
def test_rejects_missing_or_extra_images(self):
self.manifest["images"].pop("opensearch")
with self.assertRaisesRegex(ValueError, "exactly backend and opensearch"):
validate_manifest(self.manifest, self.policy)
self.manifest["images"]["opensearch"] = "registry.invalid/akyldash/opensearch@sha256:" + "c" * 64
self.manifest["images"]["extra"] = "registry.invalid/akyldash/extra@sha256:" + "d" * 64
with self.assertRaisesRegex(ValueError, "exactly backend and opensearch"):
validate_manifest(self.manifest, self.policy)
def test_rejects_duplicate_json_keys(self):
with tempfile.TemporaryDirectory() as directory:
manifest_path = Path(directory) / "manifest.json"
policy_path = Path(directory) / "policy.json"
manifest_path.write_text(
'{"schema_version":1,"source_ref":"refs/heads/main","commit":"' + "a" * 40
+ '","images":{"backend":"first","backend":"second","opensearch":"third"}}',
encoding="utf-8",
)
policy_path.write_text(json.dumps(self.policy), encoding="utf-8")
command = [
"python3", str(ROOT / "infra/deploy/validate_release.py"), str(manifest_path),
"--policy", str(policy_path),
]
result = subprocess.run(command, text=True, capture_output=True)
self.assertEqual(result.returncode, 1)
self.assertIn("duplicate JSON key: backend", result.stderr)
def test_rejects_malformed_json(self):
with tempfile.TemporaryDirectory() as directory:
manifest_path = Path(directory) / "manifest.json"
policy_path = Path(directory) / "policy.json"
manifest_path.write_text('{"schema_version":', encoding="utf-8")
policy_path.write_text(json.dumps(self.policy), encoding="utf-8")
result = subprocess.run(
["python3", str(ROOT / "infra/deploy/validate_release.py"), str(manifest_path), "--policy", str(policy_path)],
text=True,
capture_output=True,
)
self.assertEqual(result.returncode, 1)
self.assertIn("cannot read valid JSON", result.stderr)
@unittest.skipUnless(shutil.which("docker"), "Docker CLI is unavailable")
def test_compose_override_renders_digest_images_and_preserves_build(self):
backend = "registry.invalid/akyldash/backend@sha256:" + "b" * 64
opensearch = "registry.invalid/akyldash/opensearch@sha256:" + "c" * 64
with tempfile.TemporaryDirectory() as directory:
temp = Path(directory)
secret = temp / "review_secret"
secret.touch()
override = temp / "release-override.json"
override.write_text(json.dumps({"services": {
"backend": {"image": backend},
"opensearch": {"image": opensearch},
}}), encoding="utf-8")
command = [
"docker", "compose", "--env-file", str(ROOT / "deploy/production/.env.example"),
"-f", str(ROOT / "deploy/production/compose.yaml"), "-f", str(override),
"config", "--format", "json",
]
result = subprocess.run(
command, text=True, capture_output=True,
env={**os.environ, "REVIEW_SECRET_FILE": str(secret)},
)
self.assertEqual(result.returncode, 0, result.stderr)
services = json.loads(result.stdout)["services"]
self.assertEqual(services["backend"]["image"], backend)
self.assertEqual(services["opensearch"]["image"], opensearch)
self.assertIn("build", services["backend"])
self.assertIn("build", services["opensearch"])
if __name__ == "__main__":
unittest.main()
+95 -2
View File
@@ -1,6 +1,12 @@
import json
import subprocess
import sys
import unittest
from types import SimpleNamespace
from unittest.mock import patch
from rollback import image_restore_plan
import rollback
from rollback import image_restore_plan, require_tag_rollback, tag_previous_images, verify_previous_images
class RollbackPlanTests(unittest.TestCase):
@@ -28,9 +34,96 @@ class RollbackPlanTests(unittest.TestCase):
def test_refuses_digest_reference_that_cannot_be_retagged(self):
self.manifest["images"]["backend"]["configured"] = "registry/backend@sha256:" + "c" * 64
self.configured["backend"] = self.manifest["images"]["backend"]["configured"]
with self.assertRaisesRegex(ValueError, "local tag"):
plan = image_restore_plan(self.manifest, self.configured)
self.assertEqual(plan["backend"], ("sha256:" + "a" * 64, self.configured["backend"]))
with self.assertRaisesRegex(ValueError, "not supported safely"):
require_tag_rollback(plan)
def test_refuses_malformed_digest_reference(self):
self.manifest["images"]["backend"]["configured"] = "registry/backend@sha256:xyz"
self.configured["backend"] = self.manifest["images"]["backend"]["configured"]
with self.assertRaisesRegex(ValueError, "invalid image reference"):
image_restore_plan(self.manifest, self.configured)
def test_refuses_invalid_recorded_image_id(self):
self.manifest["images"]["backend"]["image_id"] = "sha256:not-a-digest"
with self.assertRaisesRegex(ValueError, "invalid previous image ID"):
image_restore_plan(self.manifest, self.configured)
def test_digest_reference_must_resolve_to_recorded_image_id(self):
digest = "registry/backend@sha256:" + "c" * 64
self.manifest["images"]["backend"]["configured"] = digest
self.configured["backend"] = digest
plan = image_restore_plan(self.manifest, self.configured)
with patch("rollback.run", side_effect=["sha256:" + "a" * 64, "sha256:" + "a" * 64,
"sha256:" + "b" * 64]):
verify_previous_images(plan)
with patch("rollback.run", side_effect=["sha256:" + "a" * 64, "sha256:" + "d" * 64]):
with self.assertRaisesRegex(ValueError, "does not match its recorded image"):
verify_previous_images(plan)
def test_missing_previous_image_fails_closed(self):
plan = image_restore_plan(self.manifest, self.configured)
with patch("rollback.run", side_effect=subprocess.CalledProcessError(1, "docker")):
with self.assertRaisesRegex(ValueError, "previous image is unavailable"):
verify_previous_images(plan)
def test_digest_reference_is_never_tagged(self):
digest = "registry/backend@sha256:" + "c" * 64
plan = {"backend": ("sha256:" + "a" * 64, digest)}
with patch("rollback.run") as run_mock:
with self.assertRaisesRegex(ValueError, "not supported safely"):
tag_previous_images(plan)
run_mock.assert_not_called()
def test_digest_rollback_fails_before_stopping_or_changing_references(self):
backend_id = "sha256:" + "a" * 64
opensearch_id = "sha256:" + "b" * 64
digest = "registry/backend@sha256:" + "c" * 64
manifest = {
"images": {
"backend": {"configured": digest, "image_id": backend_id},
"opensearch": self.manifest["images"]["opensearch"],
},
"deployment_files": {
"deploy/production/compose.yaml": "compose-hash",
"deploy/production/Caddyfile": "caddy-hash",
},
}
rendered = {
"services": {
"backend": {"image": digest},
"opensearch": {"image": self.configured["opensearch"]},
}
}
fake_archive = SimpleNamespace(
digest=lambda path: "compose-hash" if path.name == "compose.yaml" else "caddy-hash",
verify_archive=lambda _backup: manifest,
)
backup = rollback.BACKUP_DIR / "synthetic.tar.gz"
with (
patch("rollback.Path.is_file", return_value=True),
patch("rollback.Path.is_symlink", return_value=False),
patch("rollback.Path.stat", return_value=SimpleNamespace(st_uid=0, st_mode=0o100600)),
patch("rollback.compose", return_value=json.dumps(rendered)) as compose_mock,
patch("rollback.run", side_effect=[backend_id, backend_id, opensearch_id]) as run_mock,
patch.dict(sys.modules, {"archive": fake_archive}),
):
with self.assertRaisesRegex(ValueError, "not supported safely"):
rollback.rollback(backup, data_compatible=True)
compose_mock.assert_called_once_with("config", "--format", "json", capture=True)
self.assertEqual(run_mock.call_count, 3)
self.assertTrue(all(call.args[0][:3] == ["docker", "image", "inspect"]
for call in run_mock.call_args_list))
def test_tag_rollback_keeps_existing_image_tag_operation(self):
plan = image_restore_plan(self.manifest, self.configured)
with patch("rollback.run") as run_mock:
tag_previous_images(plan)
self.assertEqual(run_mock.call_count, 2)
run_mock.assert_any_call(["docker", "image", "tag", "sha256:" + "a" * 64,
"akyldash/backend:local"])
if __name__ == "__main__":
unittest.main()
+328
View File
@@ -0,0 +1,328 @@
import hashlib
import json
import os
import fcntl
import sqlite3
import tempfile
import unittest
from pathlib import Path
from verify_corpus import (
InventoryError,
database,
insert_or_update_file,
prefix_upper_bound,
raw_file_rows,
combined_source_hash,
safe_relative_path,
validate_fragment_file,
verify_corpus,
)
class CorpusInventoryTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.base = Path(self.temp.name)
self.raw = self.base / "raw"
self.raw.mkdir()
self._manifest()
def tearDown(self):
self.temp.cleanup()
def _manifest(self):
connection = sqlite3.connect(self.raw / "manifest.sqlite3")
connection.executescript(
"CREATE TABLE documents (code INTEGER PRIMARY KEY, source_sha256 TEXT, fetched_at TEXT);"
"CREATE TABLE errors (code INTEGER PRIMARY KEY, message TEXT, failed_at TEXT);"
)
connection.close()
def _nonempty_corpus(self):
document = self.raw / "documents/1"
edition = document / "editions/2"
edition.mkdir(parents=True)
(document / "metadata.json").write_text('{"Code":"1"}\n', encoding="utf-8")
(edition / "metadata.json").write_text('{"Code":"2"}\n', encoding="utf-8")
html = edition / "ru.html"
html.write_text("<p>Фрагмент</p>\n", encoding="utf-8")
source_files = []
for path in sorted(document.rglob("*")):
if path.is_file():
digest = hashlib.sha256(path.read_bytes()).hexdigest()
source_files.append({"path": path.relative_to(self.raw).as_posix(), "sha256": digest})
source_sha = combined_source_hash(source_files)
raw_manifest = sqlite3.connect(self.raw / "manifest.sqlite3")
raw_manifest.execute("INSERT INTO documents VALUES (1, ?, 'now')", (source_sha,))
raw_manifest.commit()
raw_manifest.close()
normalized = self.base / "normalized"
language = normalized / "documents/1/editions/2/ru"
language.mkdir(parents=True)
fragment_text = "Фрагмент"
fragments = [{
"id": "document:1:edition:2:lang:ru:fragment:1",
"document_code": "1",
"edition_code": "2",
"language": "ru",
"position": 1,
"text": fragment_text,
"text_sha256": hashlib.sha256(fragment_text.encode("utf-8")).hexdigest(),
"source_path": "documents/1/editions/2/ru.html",
"source_sha256": next(item["sha256"] for item in source_files if item["path"].endswith("/ru.html")),
}]
(language / "fragments.json").write_text(json.dumps(fragments, ensure_ascii=False), encoding="utf-8")
(language / "content.txt").write_text(fragment_text + "\n", encoding="utf-8")
(language / "content.html").write_text("<p>Фрагмент</p>\n", encoding="utf-8")
document_json = {
"schema_version": "1",
"source_code": "1",
"normalizer": {"version": "1.0.0"},
"source": {"path": "documents/1", "files": source_files, "sha256": source_sha},
"editions": [{"source_code": "2"}],
}
(normalized / "documents/1/document.json").write_text(json.dumps(document_json), encoding="utf-8")
edition_json = {
"source_code": "2",
"available_languages": ["ru"],
"source": {
"path": "documents/1/editions/2",
"files": [item for item in source_files if item["path"].startswith("documents/1/editions/2/")],
},
"images": [],
}
(normalized / "documents/1/editions/2/edition.json").write_text(json.dumps(edition_json), encoding="utf-8")
normalized_manifest = sqlite3.connect(normalized / "manifest.sqlite3")
normalized_manifest.execute(
"CREATE TABLE documents (code TEXT PRIMARY KEY, source_sha256 TEXT, schema_version TEXT, normalizer_version TEXT, state TEXT)"
)
normalized_manifest.execute("INSERT INTO documents VALUES ('1', ?, '1', '1.0.0', 'success')", (source_sha,))
normalized_manifest.commit()
normalized_manifest.close()
return normalized, source_sha
def test_output_inside_source_is_rejected_without_creating_it(self):
output = self.raw / "artifacts"
before_mode = self.raw.stat().st_mode
with self.assertRaisesRegex(InventoryError, "separate"):
verify_corpus(self.raw, None, output)
self.assertFalse(output.exists())
self.assertEqual(self.raw.stat().st_mode, before_mode)
self.assertEqual({path.name for path in self.raw.iterdir()}, {"manifest.sqlite3"})
def test_symlink_output_is_rejected_without_changing_target(self):
target = self.base / "target"
target.mkdir(mode=0o755)
link = self.base / "output-link"
try:
link.symlink_to(target, target_is_directory=True)
except (NotImplementedError, OSError):
self.skipTest("directory symlinks are unavailable")
before_mode = target.stat().st_mode
with self.assertRaisesRegex(InventoryError, "link or file"):
verify_corpus(self.raw, None, link)
self.assertEqual(target.stat().st_mode, before_mode)
self.assertEqual(list(target.iterdir()), [])
def test_public_existing_output_is_rejected_without_chmod(self):
output = self.base / "public-output"
output.mkdir(mode=0o755)
output.chmod(0o755)
before_mode = output.stat().st_mode
with self.assertRaisesRegex(InventoryError, "must be private"):
verify_corpus(self.raw, None, output)
self.assertEqual(output.stat().st_mode, before_mode)
self.assertEqual(list(output.iterdir()), [])
def test_resume_rejects_symlink_inventory_without_touching_target(self):
output = self.base / "artifacts"
output.mkdir(mode=0o700)
target = self.base / "do-not-touch.sqlite3"
target.write_bytes(b"sentinel")
link = output / "inventory.sqlite3"
try:
link.symlink_to(target)
except (NotImplementedError, OSError):
self.skipTest("file symlinks are unavailable")
with self.assertRaisesRegex(InventoryError, "private regular file"):
verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(target.read_bytes(), b"sentinel")
def test_rejects_unsafe_cross_platform_relative_paths(self):
for value in ("../outside", "C:\\outside", "folder\\file", "//server/share"):
with self.subTest(value=value), self.assertRaises(InventoryError):
safe_relative_path(value)
def test_interrupted_inventory_resumes_and_detects_changed_input(self):
payload = self.raw / "payload.txt"
payload.write_text("before", encoding="utf-8")
output = self.base / "artifacts"
stopped = verify_corpus(self.raw, None, output, stop_after_entries=1)
self.assertEqual(stopped["status"], "incomplete")
resumed = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(resumed["status"], "complete")
self.assertEqual(resumed["input_changes"]["raw"]["added"], 1)
payload.write_text("after", encoding="utf-8")
changed = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(changed["status"], "complete")
self.assertEqual(changed["input_changes"]["raw"]["changed"], 1)
rows = sqlite3.connect(output / "inventory.sqlite3").execute(
"SELECT sha256 FROM files WHERE relative_path='payload.txt'"
).fetchone()
self.assertEqual(rows[0], hashlib.sha256(b"after").hexdigest())
def test_large_filesystem_identity_values_are_stored_without_integer_overflow(self):
connection = database(self.base / "identity.sqlite3")
insert_or_update_file(
connection, "run", "raw", "payload", 1, "a" * 64,
(2**100, 2**120, 1, 2**110, 2**115, 1), {"raw": {"added": 0, "changed": 0}},
)
device, inode, mtime_ns, ctime_ns = connection.execute(
"SELECT device,inode,mtime_ns,ctime_ns FROM files"
).fetchone()
self.assertEqual((device, inode, mtime_ns, ctime_ns), tuple(map(str, (2**100, 2**120, 2**110, 2**115))))
connection.close()
def test_document_prefix_uses_primary_key_range_and_excludes_neighbor_codes(self):
connection = database(self.base / "paths.sqlite3")
prefix = "documents/1/"
upper = prefix_upper_bound(prefix)
plan = connection.execute(
"EXPLAIN QUERY PLAN SELECT relative_path FROM files WHERE root_id='raw' AND relative_path>=? AND relative_path<?",
(prefix, upper),
).fetchone()[3]
self.assertIn("USING COVERING INDEX sqlite_autoindex_files_1", plan)
connection.executemany(
"INSERT INTO files(root_id,relative_path,status,last_seen_run) VALUES('raw',?,'hashed','run')",
[(f"documents/{code}/a",) for code in range(20000)]
+ [("documents/1x/a",)],
)
self.assertEqual([item["path"] for item in raw_file_rows(connection, "1")], ["documents/1/a"])
connection.close()
def test_inventory_rejects_a_concurrent_writer_and_records_verifier_hash(self):
output = self.base / "artifacts"
output.mkdir(mode=0o700)
lock_path = output / ".inventory.lock"
descriptor = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
try:
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
with self.assertRaisesRegex(InventoryError, "another inventory run"):
verify_corpus(self.raw, None, output)
finally:
os.close(descriptor)
result = verify_corpus(self.raw, None, output)
self.assertEqual(result["status"], "complete")
self.assertRegex(result["verifier_sha256"], r"^[a-f0-9]{64}$")
def test_removed_file_is_reported_on_resume(self):
payload = self.raw / "payload.txt"
payload.write_text("payload", encoding="utf-8")
output = self.base / "artifacts"
self.assertEqual(verify_corpus(self.raw, None, output)["status"], "complete")
payload.unlink()
result = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(result["input_changes"]["raw"]["removed"], 1)
self.assertIn("removed_file", {item["kind"] for item in result["finding_examples"]})
def test_added_file_is_reported_on_resume(self):
output = self.base / "artifacts"
self.assertEqual(verify_corpus(self.raw, None, output)["status"], "complete")
(self.raw / "extra.txt").write_text("extra", encoding="utf-8")
result = verify_corpus(self.raw, None, output, resume=True)
self.assertEqual(result["input_changes"]["raw"]["added"], 1)
def test_manifest_error_missing_document_and_extra_directory_are_reported(self):
connection = sqlite3.connect(self.raw / "manifest.sqlite3")
connection.execute("INSERT INTO documents VALUES (1, ?, 'now')", ("a" * 64,))
connection.execute("INSERT INTO errors VALUES (6, 'fetch failed', 'now')")
connection.commit()
connection.close()
(self.raw / "documents" / "999").mkdir(parents=True)
result = verify_corpus(self.raw, None, self.base / "artifacts")
kinds = {finding["kind"] for finding in result["finding_examples"]}
self.assertEqual(result["status"], "failed")
self.assertIn("raw_manifest_error", kinds)
self.assertIn("missing_raw_document_directory", kinds)
self.assertIn("extra_raw_document_directory", kinds)
def test_empty_raw_and_normalized_manifests_complete_successfully(self):
normalized = self.base / "normalized"
normalized.mkdir()
connection = sqlite3.connect(normalized / "manifest.sqlite3")
connection.execute(
"CREATE TABLE documents (code TEXT PRIMARY KEY, source_sha256 TEXT, schema_version TEXT, normalizer_version TEXT, state TEXT)"
)
connection.commit()
connection.close()
result = verify_corpus(self.raw, normalized, self.base / "artifacts")
self.assertEqual(result["status"], "complete", result["finding_examples"])
self.assertEqual(result["manifest_summary"]["raw"]["documents"], 0)
self.assertEqual(result["manifest_summary"]["normalized"]["documents"], 0)
def test_nonempty_raw_and_normalized_manifests_cross_validate_fragment_sources(self):
normalized, _ = self._nonempty_corpus()
result = verify_corpus(self.raw, normalized, self.base / "artifacts")
self.assertEqual(result["status"], "complete", result["finding_examples"])
self.assertEqual(result["manifest_summary"]["raw"]["documents"], 1)
self.assertEqual(result["manifest_summary"]["normalized"]["fragments"], 1)
def test_raw_normalized_source_mismatch_is_reported(self):
normalized, _ = self._nonempty_corpus()
manifest = sqlite3.connect(normalized / "manifest.sqlite3")
manifest.execute("UPDATE documents SET source_sha256=? WHERE code='1'", ("0" * 64,))
manifest.commit()
manifest.close()
result = verify_corpus(self.raw, normalized, self.base / "artifacts")
kinds = {item["kind"] for item in result["finding_examples"]}
self.assertEqual(result["status"], "failed")
self.assertIn("normalized_source_hash_mismatch", kinds)
self.assertIn("document_source_manifest_mismatch", kinds)
def test_fragment_text_hash_mismatch_is_reported(self):
normalized = self.base / "normalized"
fragment_rel = "documents/1/editions/2/ru/fragments.json"
fragment_path = normalized / fragment_rel
fragment_path.parent.mkdir(parents=True)
fragment = {
"id": "document:1:edition:2:lang:ru:fragment:1",
"document_code": "1",
"edition_code": "2",
"language": "ru",
"position": 1,
"text": "text",
"text_sha256": "0" * 64,
"source_path": "documents/1/source.txt",
"source_sha256": "1" * 64,
}
fragment_path.write_text(json.dumps([fragment]), encoding="utf-8")
content_path = fragment_path.with_name("content.txt")
content_path.write_text("text\n", encoding="utf-8")
connection = database(self.base / "test.sqlite3")
content_digest = hashlib.sha256(b"text\n").hexdigest()
connection.execute(
"INSERT INTO files(root_id,relative_path,sha256,status,last_seen_run) VALUES('normalized',?,?, 'hashed','run')",
(content_path.relative_to(normalized).as_posix(), content_digest),
)
validate_fragment_file(
connection,
"run",
normalized,
fragment_path,
fragment_rel,
hashlib.sha256(fragment_path.read_bytes()).hexdigest(),
{"documents/1/source.txt": "1" * 64},
)
kinds = {row[0] for row in connection.execute("SELECT kind FROM findings WHERE run_id='run'")}
connection.close()
self.assertIn("fragment_text_checksum_mismatch", kinds)
if __name__ == "__main__":
unittest.main()
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Validate the shape and repository allowlist of a prepared release manifest."""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
SERVICES = {"backend", "opensearch"}
COMMIT_RE = re.compile(r"[0-9a-f]{40}\Z")
REPOSITORY_RE = re.compile(
r"[a-z0-9]+(?:[._-][a-z0-9]+)*(?::[0-9]+)?"
r"(?:/[a-z0-9]+(?:[._-][a-z0-9]+)*)*\Z"
)
IMAGE_RE = re.compile(r"(.+)@sha256:([0-9a-f]{64})\Z")
def unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]:
result: dict[str, object] = {}
for key, value in pairs:
if key in result:
raise ValueError(f"duplicate JSON key: {key}")
result[key] = value
return result
def read_json(path: Path) -> object:
try:
return json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=unique_object)
except (OSError, UnicodeError, json.JSONDecodeError) as error:
raise ValueError(f"cannot read valid JSON from {path}: {error}") from error
def validate_policy(policy: object) -> dict[str, list[str]]:
if not isinstance(policy, dict) or set(policy) != {"schema_version", "repositories"}:
raise ValueError("policy must contain exactly schema_version and repositories")
if type(policy["schema_version"]) is not int or policy["schema_version"] != 1:
raise ValueError("unsupported policy schema_version")
repositories = policy["repositories"]
if not isinstance(repositories, dict) or set(repositories) != SERVICES:
raise ValueError("policy repositories must contain exactly backend and opensearch")
result: dict[str, list[str]] = {}
for service in sorted(SERVICES):
allowed = repositories[service]
if not isinstance(allowed, list) or not allowed:
raise ValueError(f"policy repository allowlist for {service} must be a non-empty list")
if any(not isinstance(repository, str) or not REPOSITORY_RE.fullmatch(repository) for repository in allowed):
raise ValueError(f"policy contains an invalid repository for {service}")
if len(set(allowed)) != len(allowed):
raise ValueError(f"policy repository allowlist for {service} contains duplicates")
result[service] = allowed
return result
def validate_manifest(manifest: object, policy: object) -> dict[str, object]:
allowed_repositories = validate_policy(policy)
if not isinstance(manifest, dict) or set(manifest) != {
"schema_version", "source_ref", "commit", "images"
}:
raise ValueError("manifest must contain exactly schema_version, source_ref, commit, and images")
if type(manifest["schema_version"]) is not int or manifest["schema_version"] != 1:
raise ValueError("unsupported manifest schema_version")
if manifest["source_ref"] != "refs/heads/main":
raise ValueError("manifest source_ref must be refs/heads/main")
commit = manifest["commit"]
if not isinstance(commit, str) or not COMMIT_RE.fullmatch(commit):
raise ValueError("manifest commit must be a 40-character lowercase Git SHA-1")
images = manifest["images"]
if not isinstance(images, dict) or set(images) != SERVICES:
raise ValueError("manifest images must contain exactly backend and opensearch")
for service in sorted(SERVICES):
image = images[service]
match = IMAGE_RE.fullmatch(image) if isinstance(image, str) else None
if not match:
raise ValueError(f"{service} image must use repository@sha256:<64 lowercase hex>")
repository = match.group(1)
if not REPOSITORY_RE.fullmatch(repository):
raise ValueError(f"{service} image has an invalid OCI repository")
if repository not in allowed_repositories[service]:
raise ValueError(f"{service} image repository is not allowed by policy")
return manifest
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("manifest", type=Path)
parser.add_argument("--policy", required=True, type=Path)
args = parser.parse_args()
try:
manifest = validate_manifest(read_json(args.manifest), read_json(args.policy))
except ValueError as error:
print(f"ERROR: {error}", file=sys.stderr)
return 1
images = manifest["images"]
print(
"PASS release manifest "
f"commit={manifest['commit']} backend={images['backend']} opensearch={images['opensearch']}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
File diff suppressed because it is too large. Load diff
+53 -9
View File
@@ -15,7 +15,7 @@ facts:
memory_gib: 7.7
swap_gib: 3.8
disk_gib: 99
disk_free_gib: 93
disk_free_gib: 92
network_interface: ens3
ipv4: 206.223.249.118/32
domain_resolution: akyldash.kg resolves to Cloudflare IPv4 and IPv6 edge addresses
@@ -26,12 +26,37 @@ facts:
- OpenSSH server
- systemd-resolved
- unattended-upgrades
absent:
- Docker Engine
- Docker Compose plugin
- Docker CE and CLI 29.9.0
- Docker Compose plugin 5.6.0
- containerd 2.4.1
- runc 1.5.2
- Git
- reverse proxy
access_audit_date: 2026-10-11
absent:
- ISPmanager package/service (unprivileged package and unit checks)
- root-owned application checkout
- restricted verifier and rollback helper
- non-loopback listeners on TCP 80, 443, 1500
access_audit_date: 2026-10-12
docker_service_active: true
docker_api_accessible_to_codex: false
docker_daemon_log_config:
driver: local
max_size: 10m
max_file: 3
bootstrap_paths:
/opt/akyldash: root:root 0755
/etc/akyldash: root:root 0700
/var/lib/akyldash/data: root:root 0750
/var/backups/akyldash: root:root 0700
unprivileged_nonloopback_tcp_listeners: [22]
owner_reported_ufw_allowed_tcp: [22, 80, 443, 1500]
port_1500_listener: false
rootless_docker_command_present: /usr/bin/dockerd-rootless.sh
runner_gitea_host_resolution: git.vakanaut.ru resolves to 195.209.48.28
runner_gitea_https_probe: /api/v1/version returned Gitea 1.25.5 (Actions protocol unverified)
cloudflare_edge_status:
http: 521
https: 521
target:
compose_file_in_repository: deploy/production/compose.yaml
compose_file_on_host: /opt/akyldash/current/deploy/production/compose.yaml
@@ -58,7 +83,7 @@ target:
persistent_data: named Compose volume opensearch-data
reverse_proxy:
target: Caddy in Compose; Caddyfile proxies akyldash.kg to backend:8080
status: repository configuration exists; not installed or running on VPS
status: no listener observed on 80/443; container state needs root Docker audit
external_ports_target: [22, 80, 443]
compose_networks:
proxy: caddy and backend
@@ -70,15 +95,31 @@ target:
restricted_helper:
path: /usr/local/sbin/akyldash-rollback
status: repository code and owner-run installer exist; not installed on VPS
compose_read_only_helper:
path: /usr/local/sbin/akyldash-compose
allowed_operations: [config, status]
status: repository code and owner-run installer exist; not installed on VPS
read_only_verifier:
path: /usr/local/sbin/akyldash-verify
status: repository code and owner-run installer exist; not installed on VPS
gitea_actions_runner:
repository: admin/akyldash
registration_scope: repository
implementation: official act_runner 0.2.13; pinned binary SHA-256 in infra/runner/install-runner.sh
label: akyldash-ci
capacity: 1
job_limits: {cpu: 2, memory_gib: 3, pids: 512}
account_slice_limits: {cpu: 2.5, memory_gib: 4, tasks: 2048}
docker_store_limit_gib: 15
docker_boundary: rootless dedicated daemon; no production socket; job socket injection disabled
service_status: configuration prepared, not installed or registered; Online unverified
owner_action: run the root installer from the reviewed checkout; register through repository runner settings
checks:
- infra/bootstrap/verify.sh akyldash_vps
- docker compose -f deploy/production/compose.yaml ps
- curl -fsS https://akyldash.kg/review
- python3 infra/backup/archive.py verify /var/backups/akyldash/<archive>.tar.gz
backup_path: /var/backups/akyldash (target; archive script in infra/backup/archive.py is not installed or run)
backup_path: /var/backups/akyldash (directory exists; archive script in infra/backup/archive.py is not installed or run)
deployed_backend_image_record: /opt/akyldash/deployed-image.txt (required by drift verifier; not present before first deployment)
versions:
application:
@@ -87,4 +128,7 @@ versions:
telegram_bot: 0.2.2
infrastructure: unknown; no production install or release exists
caddy_image: 2.11.6-alpine pinned by manifest digest in Compose
docker_engine: not installed
docker_engine: 29.9.0 (daemon active; server API unverified as codex)
docker_compose: 5.6.0
containerd: 2.4.1
runc: 1.5.2
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
set -Eeuo pipefail
systemctl is-active --quiet akyldash-act-runner.service
runner_uid=$(id -u akyldash-runner)
runuser -u akyldash-runner -- env \
HOME=/var/lib/akyldash-runner/.config/act-runner \
XDG_CONFIG_HOME=/var/lib/akyldash-runner/.config \
XDG_DATA_HOME=/var/lib/akyldash-runner/.config/act-runner/.local/share \
DOCKER_HOST="unix:///run/user/$runner_uid/docker.sock" \
/usr/bin/docker info --format '{{.ServerVersion}}' >/dev/null
@@ -0,0 +1,7 @@
[Unit]
Description=Check AkylDash Gitea Actions runner health
OnFailure=akyldash-runner-restart.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/akyldash-runner-healthcheck
@@ -0,0 +1,10 @@
[Unit]
Description=Periodically check AkylDash Gitea Actions runner health
[Timer]
OnBootSec=2m
OnUnitActiveSec=1m
Unit=akyldash-runner-healthcheck.service
[Install]
WantedBy=timers.target
@@ -0,0 +1,6 @@
[Unit]
Description=Restart AkylDash Gitea Actions runner after a failed health check
[Service]
Type=oneshot
ExecStart=/bin/systemctl restart akyldash-act-runner.service
+15
View File
@@ -0,0 +1,15 @@
log:
level: info
runner:
file: /var/lib/akyldash-runner/.config/act-runner/.runner
capacity: 1
labels:
- "akyldash-ci:docker://docker.gitea.com/runner-images:ubuntu-24.04"
container:
privileged: false
options: --cpus=2 --memory=3g --pids-limit=512
valid_volumes: []
docker_host: "-"
force_pull: true
cache:
enabled: false
+22
View File
@@ -0,0 +1,22 @@
BEGIN { FS = ":" }
/^[[:space:]]*$/ || /^[[:space:]]*#/ { next }
NF != 3 || $2 !~ /^[0-9]+$/ || $3 !~ /^[0-9]+$/ || $3 < 1 { invalid = 1; next }
{
start[FILENAME, ++count[FILENAME]] = $2
end[FILENAME, count[FILENAME]] = $2 + $3 - 1
files[FILENAME] = 1
}
END {
if (invalid) exit 2
for (base = 100000; base + 65535 <= 4294901760; base += 65536) {
overlap = 0
for (file in files)
for (i = 1; i <= count[file]; i++)
if (start[file, i] <= base + 65535 && end[file, i] >= base) overlap = 1
if (!overlap) {
print base "-" base + 65535
exit
}
}
exit 1
}
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env bash
runner_die() { printf 'ERROR: %s\n' "$*" >&2; return 1; }
runner_ensure_account() {
local user=$1 home=$2 entry uid passwd_home shell groups uid_min
if ! entry=$(getent passwd "$user"); then
[[ ! -e $home && ! -L $home ]] || { runner_die "runner home already exists without its account: $home"; return 1; }
useradd --system --create-home --home-dir "$home" --shell /usr/sbin/nologin "$user" || return 1
return
fi
IFS=: read -r _ _ uid _ _ passwd_home shell <<<"$entry"
uid_min=$(awk '$1 == "UID_MIN" && $2 ~ /^[0-9]+$/ {print $2; exit}' /etc/login.defs)
[[ $uid =~ ^[0-9]+$ && $uid -gt 0 && $uid_min =~ ^[0-9]+$ && $uid -lt $uid_min && $passwd_home == "$home" && $shell == /usr/sbin/nologin ]] || { runner_die "existing $user account has an unexpected UID, home, or shell"; return 1; }
[[ $(id -gn "$user") == "$user" ]] || { runner_die "existing $user account has an unexpected primary group"; return 1; }
groups=$(id -nG "$user")
[[ $groups == "$user" ]] || { runner_die "existing $user account has unexpected supplementary groups: $groups"; return 1; }
[[ -d $home && ! -L $home ]] || { runner_die "existing $user home is missing or has unexpected type"; return 1; }
case $(stat -c %U "$home") in
"$user") ;;
root) [[ $(stat -c '%G:%a' "$home") == root:711 ]] || { runner_die "root-owned $user home must be mode 0711"; return 1; } ;;
*) runner_die "existing $user home has unexpected ownership"; return 1 ;;
esac
}
runner_lock_home() {
local home=$1 parent_mode
[[ -d $home && ! -L $home && $(stat -c %U /var/lib) == root ]] || { runner_die "runner home or its parent is unsafe: $home"; return 1; }
parent_mode=$(stat -c %a /var/lib)
(( (8#$parent_mode & 022) == 0 )) || { runner_die '/var/lib must not be group/world-writable'; return 1; }
chown root:root "$home" && chmod 0711 "$home" || return 1
[[ $(stat -c '%U:%G:%a' "$home") == root:root:711 ]] || { runner_die 'runner home could not be locked from runner writes'; return 1; }
}
runner_ensure_image() {
local image=$1 bytes=$2 parent_uid=$3 parent_mode=$4
python3 "$(dirname "${BASH_SOURCE[0]}")/secure-runner-image.py" "$image" "$bytes" "$parent_uid" "$parent_mode" || { runner_die "runner image validation or initialization failed: $image"; return 1; }
}
runner_install_file() {
local source=$1 destination=$2 owner=$3 group=$4 mode=${5#0}
if [[ -e $destination || -L $destination ]]; then
[[ -f $destination && ! -L $destination ]] && cmp -s "$source" "$destination" || { runner_die "existing runner file differs from reviewed configuration: $destination"; return 1; }
[[ $(stat -c '%U:%G:%a' "$destination") == "$owner:$group:$mode" ]] || { runner_die "existing runner file has unexpected ownership or permissions: $destination"; return 1; }
return
fi
install -o "$owner" -g "$group" -m "0$mode" "$source" "$destination"
}
runner_install_content() {
local destination=$1 owner=$2 group=$3 mode=$4 temporary
temporary=$(mktemp)
cat >"$temporary"
if ! runner_install_file "$temporary" "$destination" "$owner" "$group" "$mode"; then
rm -f "$temporary"
return 1
fi
rm -f "$temporary"
}
runner_ensure_private_directory() {
local directory=$1 owner=$2 group=$3 current= component index
local -a components
[[ $directory == /* ]] || { runner_die "runner directory path is not absolute: $directory"; return 1; }
IFS=/ read -r -a components <<<"${directory#/}"
for index in "${!components[@]}"; do
component=${components[index]}
[[ -n $component && $component != . && $component != .. ]] || { runner_die "runner directory path has an unsafe component: $directory"; return 1; }
current+="/$component"
[[ ! -L $current ]] || { runner_die "runner directory path contains a symbolic link: $current"; return 1; }
if [[ ! -e $current ]]; then
((index == ${#components[@]} - 1)) || { runner_die "runner directory parent is missing: $current"; return 1; }
break
fi
[[ -d $current ]] || { runner_die "runner directory path component is not a directory: $current"; return 1; }
if [[ $current == /var/lib/akyldash-runner ]]; then
[[ $(stat -c '%U:%G:%a' "$current") == root:root:711 ]] || { runner_die 'runner home must be root-owned mode 0711'; return 1; }
elif [[ $current == /var/lib/akyldash-runner/* ]]; then
[[ $(stat -c '%U:%G' "$current") == "$owner:$group" ]] || { runner_die "runner directory ancestor has unexpected ownership: $current"; return 1; }
fi
done
# These paths are runner-writable; make changes as that user so a path swap
# cannot turn a root operation into a write outside its home.
if [[ $directory == /var/lib/akyldash-runner/.config ]]; then
install -d -o "$owner" -g "$group" -m 0700 "$directory" || return 1
else
runuser -u "$owner" -- install -d -m 0700 "$directory" || return 1
fi
[[ -d $directory && ! -L $directory && $(stat -c '%U:%G:%a' "$directory") == "$owner:$group:700" ]] || { runner_die "runner directory failed validation: $directory"; return 1; }
}
runner_install_user_content() {
local destination=$1 owner=$2 group=$3 mode=${4#0} temporary
temporary=$(mktemp)
cat >"$temporary"
if [[ -e $destination || -L $destination ]]; then
[[ -f $destination && ! -L $destination ]] && cmp -s "$temporary" "$destination" || { rm -f "$temporary"; runner_die "existing runner file differs from reviewed configuration: $destination"; return 1; }
[[ $(stat -c '%U:%G:%a' "$destination") == "$owner:$group:$mode" ]] || { rm -f "$temporary"; runner_die "existing runner file has unexpected ownership or permissions: $destination"; return 1; }
rm -f "$temporary"
return
fi
chown "$owner:$group" "$temporary" && chmod "0$mode" "$temporary" || { rm -f "$temporary"; return 1; }
local result=0
runuser -u "$owner" -- install -m "0$mode" "$temporary" "$destination" || result=$?
rm -f "$temporary"
((result == 0)) || return 1
[[ -f $destination && ! -L $destination && $(stat -c '%U:%G:%a' "$destination") == "$owner:$group:$mode" ]] || { runner_die "runner file failed validation: $destination"; return 1; }
}
runner_validate_mount() {
local image=$1 store=$2 target filesystem source options backing mount_info
mount_info=$(findmnt --noheadings --output TARGET,FSTYPE,SOURCE,OPTIONS --mountpoint "$store") || { runner_die 'runner store is not mounted'; return 1; }
read -r target filesystem source options <<<"$mount_info" || { runner_die 'runner store mount information is empty'; return 1; }
[[ $target == "$store" && $filesystem == ext4 && $source == /dev/loop* ]] || { runner_die 'runner store is mounted from an unexpected target, filesystem, or device'; return 1; }
case ",$options," in *,nosuid,*nodev,*|*,nodev,*nosuid,*) ;; *) runner_die 'runner store mount is missing nosuid or nodev'; return 1 ;; esac
backing=$(losetup --noheadings --output BACK-FILE "$source") || { runner_die 'cannot identify the runner loop backing file'; return 1; }
[[ $(realpath -e "$backing") == "$(realpath -e "$image")" ]] || { runner_die 'runner store is backed by a different image'; return 1; }
}
runner_ensure_mount() {
local image=$1 store=$2 unit_name=$3 unit_file=$4 unit_tmp mounted=0
[[ -d $store && ! -L $store ]] || { runner_die "runner mount point is missing or is not a directory: $store"; return 1; }
if mountpoint -q "$store"; then
runner_validate_mount "$image" "$store" || return 1
mounted=1
fi
unit_tmp=$(mktemp)
if ! cat >"$unit_tmp" <<EOF
[Unit]
Description=Bounded storage for the AkylDash CI runner
Before=local-fs.target
[Mount]
What=$image
Where=$store
Type=ext4
Options=loop,nosuid,nodev
[Install]
WantedBy=local-fs.target
EOF
then
rm -f "$unit_tmp"
runner_die 'failed to prepare runner mount unit'
return 1
fi
if [[ -e $unit_file || -L $unit_file ]]; then
if [[ ! -f $unit_file || -L $unit_file ]] || ! cmp -s "$unit_tmp" "$unit_file"; then
rm -f "$unit_tmp"
runner_die "existing runner mount unit differs from the reviewed configuration: $unit_file"
return 1
fi
[[ $(stat -c '%u:%g:%a' "$unit_file") == "$(id -u):$(id -g):644" ]] || { rm -f "$unit_tmp"; runner_die 'existing runner mount unit has unexpected ownership or permissions'; return 1; }
else
install -m 0644 "$unit_tmp" "$unit_file" || { rm -f "$unit_tmp"; return 1; }
fi
rm -f "$unit_tmp"
systemctl daemon-reload || return 1
if ((mounted)); then
systemctl enable "$unit_name" || return 1
else
systemctl enable "$unit_name" || return 1
systemctl start "$unit_name" || return 1
runner_validate_mount "$image" "$store" || return 1
fi
}
+243
View File
@@ -0,0 +1,243 @@
#!/usr/bin/env bash
set -Eeuo pipefail
readonly runner_user=akyldash-runner
readonly runner_home=/var/lib/akyldash-runner
readonly runner_runtime_home="$runner_home/.config/act-runner"
readonly runner_xdg_config_home="$runner_home/.config"
readonly runner_xdg_data_home="$runner_runtime_home/.local/share"
readonly runner_store="$runner_home/docker-data"
readonly runner_image="$runner_home/docker-store.img"
readonly runner_mount_unit_name="$(systemd-escape --path --suffix=mount /var/lib/akyldash-runner/docker-data)"
readonly runner_mount_unit="/etc/systemd/system/$runner_mount_unit_name"
readonly runner_image_bytes=16106127360
readonly runner_version=0.2.13
readonly runner_sha256=3acac8b506ac8cadc88a55155b5d6378f0fab0b8f62d1e0c0450f4ccd69733e2
readonly runner_url="https://dl.gitea.com/gitea-runner/$runner_version/act_runner-$runner_version-linux-amd64"
readonly checkout=/opt/akyldash/current
die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
[[ $EUID -eq 0 ]] || die 'run this script as root'
[[ -r /etc/os-release ]] || die 'cannot identify operating system'
. /etc/os-release
[[ ${ID:-} == ubuntu && ${VERSION_ID:-} == 24.04 ]] || die 'Ubuntu 24.04 is required'
[[ $(uname -m) == x86_64 ]] || die 'this pinned binary is for x86_64 only'
[[ $(stat -fc %T /sys/fs/cgroup) == cgroup2fs ]] || die 'rootless Docker requires cgroup v2'
[[ $(findmnt -n -o FSTYPE /) == ext4 ]] || die 'the bounded runner store currently requires ext4 on /'
[[ -d $checkout/.git && ! -L $checkout && $(stat -c %U "$checkout") == root ]] || die 'install the reviewed root-owned checkout at /opt/akyldash/current first'
for directory in /opt /opt/akyldash; do
[[ $(stat -c %U "$directory") == root ]] || die "$directory must be owned by root"
mode=$(stat -c %a "$directory")
(( (8#$mode & 022) == 0 )) || die "$directory must not be group/world-writable"
done
unsafe=$(find "$checkout" -xdev \( ! -user root -o -perm /022 \) -print -quit)
[[ -z $unsafe ]] || die 'the repository checkout contains files writable by non-root users'
git -C "$checkout" diff --quiet HEAD -- infra/runner || die 'runner files differ from the reviewed Git revision'
source "$checkout/infra/runner/install-runner-lib.sh"
if systemctl is-active --quiet akyldash-act-runner.service; then
die 'stop akyldash-act-runner.service after all jobs finish before rerunning this installer'
fi
[[ -r /etc/subuid && -r /etc/subgid ]] || die 'subordinate ID allocation files are unavailable'
if getent passwd "$runner_user" >/dev/null; then
runner_ensure_account "$runner_user" "$runner_home" || die 'runner account validation failed'
runner_lock_home "$runner_home" || die 'runner home protection failed'
else
[[ ! -e $runner_home && ! -L $runner_home ]] || die 'runner home exists without its account; inspect it before continuing'
fi
if [[ -e $runner_image || -L $runner_image ]]; then
runner_ensure_image "$runner_image" "$runner_image_bytes" 0 711 || die 'runner image validation failed'
else
[[ $(df -Pk / | awk 'NR==2 {print $4}') -ge 25000000 ]] || die 'need at least 25 GiB free before reserving 15 GiB for runner data'
fi
for file in /etc/subuid /etc/subgid; do
awk -F: '!/^[[:space:]]*#/ && !/^[[:space:]]*$/ && (NF != 3 || $2 !~ /^[0-9]+$/ || $3 !~ /^[0-9]+$/ || $3 < 1) { bad=1 } END { exit bad }' "$file" || die "$file contains an invalid subordinate ID entry"
done
subuid_pair=$(awk -F: -v user="$runner_user" '$1 == user { count++; pair=$2 ":" $3 } END { if (count > 1) exit 2; if (count == 1) print pair }' /etc/subuid) || die 'runner has multiple subuid allocations; inspect them before continuing'
subgid_pair=$(awk -F: -v user="$runner_user" '$1 == user { count++; pair=$2 ":" $3 } END { if (count > 1) exit 2; if (count == 1) print pair }' /etc/subgid) || die 'runner has multiple subgid allocations; inspect them before continuing'
for pair in "$subuid_pair" "$subgid_pair"; do
[[ -z $pair || $pair =~ ^[0-9]+:65536$ ]] || die 'existing runner subordinate ID allocation is not a single 65536-ID range'
done
if [[ -n $subuid_pair && -n $subgid_pair && $subuid_pair != "$subgid_pair" ]]; then
die 'runner subuid and subgid allocations differ; inspect them before continuing'
fi
existing_pair=${subuid_pair:-$subgid_pair}
if [[ -n $existing_pair ]]; then
subid_start=${existing_pair%%:*}
[[ $subid_start =~ ^[1-9][0-9]*$ && $subid_start -ge 100000 && $subid_start -le 4294901760 ]] || die 'existing runner subordinate ID range is outside the supported range'
subid_end=$((subid_start + 65535))
subid_range="$subid_start-$subid_end"
else
subid_range=$(awk -f "$(dirname "$0")/find-free-subid-range.awk" /etc/subuid /etc/subgid) || die 'could not find a free 65536-ID range shared by /etc/subuid and /etc/subgid'
subid_start=${subid_range%-*}
subid_end=${subid_range#*-}
fi
for file in /etc/subuid /etc/subgid; do
awk -F: -v user="$runner_user" -v start="$subid_start" -v end="$subid_end" '$1 != user && $2 <= end && $2 + $3 - 1 >= start { conflict=1 } END { exit conflict }' "$file" || die "runner subordinate ID range conflicts with another entry in $file"
done
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y uidmap dbus-user-session curl logrotate docker-ce-rootless-extras
runner_ensure_account "$runner_user" "$runner_home" || die 'runner account creation failed'
runner_lock_home "$runner_home" || die 'runner home protection failed'
subid_args=()
[[ -n $subuid_pair ]] || subid_args+=(--add-subuids "$subid_range")
[[ -n $subgid_pair ]] || subid_args+=(--add-subgids "$subid_range")
if ((${#subid_args[@]})); then
usermod "${subid_args[@]}" "$runner_user"
fi
runner_uid=$(id -u "$runner_user")
runner_ensure_private_directory "$runner_home/.config" "$runner_user" "$runner_user" || die 'runner config directory failed validation'
runner_ensure_private_directory "$runner_home/.config/docker" "$runner_user" "$runner_user" || die 'runner Docker config directory failed validation'
runner_ensure_private_directory "$runner_home/.config/systemd" "$runner_user" "$runner_user" || die 'runner systemd config directory failed validation'
runner_ensure_private_directory "$runner_home/.config/systemd/user" "$runner_user" "$runner_user" || die 'runner user systemd directory failed validation'
runner_ensure_private_directory "$runner_home/.config/systemd/user/docker.service.d" "$runner_user" "$runner_user" || die 'runner Docker override directory failed validation'
runner_ensure_private_directory "$runner_runtime_home" "$runner_user" "$runner_user" || die 'runner runtime home failed directory validation'
runner_ensure_private_directory "$runner_runtime_home/.local" "$runner_user" "$runner_user" || die 'runner XDG data parent failed directory validation'
runner_ensure_private_directory "$runner_xdg_data_home" "$runner_user" "$runner_user" || die 'runner XDG data directory failed directory validation'
if [[ -e $runner_home/.runner || -L $runner_home/.runner ]]; then
[[ -f $runner_home/.runner && ! -L $runner_home/.runner ]] || die 'legacy runner registration state is not a regular file'
if [[ ! -e $runner_runtime_home/.runner && ! -L $runner_runtime_home/.runner ]]; then
runner_install_user_content "$runner_runtime_home/.runner" "$runner_user" "$runner_user" 0600 <"$runner_home/.runner" || die 'legacy runner registration state could not be preserved'
fi
fi
if [[ ! -e $runner_store && ! -L $runner_store ]]; then
install -d -o root -g root -m 0755 "$runner_store"
fi
[[ -d $runner_store && ! -L $runner_store ]] || die 'runner mount path is not a directory'
install -d -o root -g "$runner_user" -m 0750 /etc/akyldash-runner
install -d -o "$runner_user" -g "$runner_user" -m 0750 /var/log/akyldash-runner
runner_ensure_image "$runner_image" "$runner_image_bytes" 0 711 || die 'runner image initialization failed'
runner_ensure_mount "$runner_image" "$runner_store" "$runner_mount_unit_name" "$runner_mount_unit" || die 'runner mount activation failed'
chown "$runner_user:$runner_user" "$runner_store"
chmod 0700 "$runner_store"
runner_install_file "$(dirname "$0")/act-runner-healthcheck.sh" /usr/local/sbin/akyldash-runner-healthcheck root root 0755 || die 'healthcheck command differs from reviewed configuration'
runner_install_file "$(dirname "$0")/config.yaml" /etc/akyldash-runner/config.yaml root "$runner_user" 0640 || die 'runner config differs from reviewed configuration'
if [[ -e /usr/local/bin/act_runner || -L /usr/local/bin/act_runner ]]; then
[[ -f /usr/local/bin/act_runner && ! -L /usr/local/bin/act_runner ]] || die 'existing act_runner path is not a regular file'
printf '%s %s\n' "$runner_sha256" /usr/local/bin/act_runner | sha256sum --check --status || die 'existing act_runner binary does not match the pinned release'
else
tmp_binary=$(mktemp)
trap 'rm -f "$tmp_binary"' EXIT
curl --fail --location --silent --show-error "$runner_url" -o "$tmp_binary"
printf '%s %s\n' "$runner_sha256" "$tmp_binary" | sha256sum --check --status || die 'official act_runner checksum mismatch'
install -o root -g root -m 0755 "$tmp_binary" /usr/local/bin/act_runner
rm -f "$tmp_binary"
trap - EXIT
fi
runner_install_user_content "$runner_home/.config/docker/daemon.json" "$runner_user" "$runner_user" 0600 <<EOF
{
"data-root": "$runner_store",
"log-driver": "local",
"log-opts": {"max-size": "10m", "max-file": "2"}
}
EOF
runner_install_user_content "$runner_home/.config/systemd/user/docker.service.d/limits.conf" "$runner_user" "$runner_user" 0644 <<'EOF'
[Service]
Delegate=yes
CPUQuota=200%
MemoryMax=4G
TasksMax=1024
Restart=always
RestartSec=5
EOF
runner_install_user_content "$runner_home/.config/systemd/user/docker.service.d/environment.conf" "$runner_user" "$runner_user" 0644 <<EOF
[Service]
Environment=HOME=$runner_runtime_home
Environment=XDG_CONFIG_HOME=$runner_xdg_config_home
Environment=XDG_DATA_HOME=$runner_xdg_data_home
EOF
runner_install_content /etc/systemd/system/akyldash-act-runner.service root root 0644 <<EOF
[Unit]
Description=AkylDash repository Gitea Actions runner
After=network-online.target local-fs.target
Wants=network-online.target
RequiresMountsFor=$runner_store
[Service]
Type=simple
Slice=user-$runner_uid.slice
User=$runner_user
Group=$runner_user
Environment=HOME=$runner_runtime_home
Environment=XDG_CONFIG_HOME=$runner_xdg_config_home
Environment=XDG_DATA_HOME=$runner_xdg_data_home
WorkingDirectory=$runner_runtime_home
Environment=XDG_RUNTIME_DIR=/run/user/$runner_uid
Environment=DOCKER_HOST=unix:///run/user/$runner_uid/docker.sock
ExecStartPre=/bin/sh -c 'for i in \$(seq 1 60); do /usr/bin/docker info >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1'
ExecStart=/usr/local/bin/act_runner --config /etc/akyldash-runner/config.yaml daemon
Restart=always
RestartSec=10
CPUQuota=50%
MemoryMax=768M
TasksMax=256
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
RestrictSUIDSGID=yes
ReadWritePaths=$runner_runtime_home $runner_xdg_config_home /var/log/akyldash-runner /run/user/$runner_uid
StandardOutput=append:/var/log/akyldash-runner/act_runner.log
StandardError=append:/var/log/akyldash-runner/act_runner.log
[Install]
WantedBy=multi-user.target
EOF
install -d -o root -g root -m 0755 "/etc/systemd/system/user-$runner_uid.slice.d"
runner_install_content "/etc/systemd/system/user-$runner_uid.slice.d/limits.conf" root root 0644 <<EOF
[Slice]
CPUQuota=250%
MemoryMax=4G
TasksMax=2048
EOF
runner_install_content /etc/logrotate.d/akyldash-runner root root 0644 <<'EOF'
/var/log/akyldash-runner/act_runner.log {
daily
rotate 7
size 10M
compress
missingok
notifempty
copytruncate
}
EOF
runner_install_file "$(dirname "$0")/akyldash-runner-healthcheck.service" /etc/systemd/system/akyldash-runner-healthcheck.service root root 0644
runner_install_file "$(dirname "$0")/akyldash-runner-healthcheck.timer" /etc/systemd/system/akyldash-runner-healthcheck.timer root root 0644
runner_install_file "$(dirname "$0")/akyldash-runner-restart.service" /etc/systemd/system/akyldash-runner-restart.service root root 0644
loginctl enable-linger "$runner_user"
systemctl daemon-reload
systemctl start "user@$runner_uid.service"
rootless_docker_unit="$runner_xdg_config_home/systemd/user/docker.service"
rootless_env=(HOME="$runner_runtime_home" XDG_CONFIG_HOME="$runner_xdg_config_home" XDG_DATA_HOME="$runner_xdg_data_home" XDG_RUNTIME_DIR="/run/user/$runner_uid" DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$runner_uid/bus")
runuser -u "$runner_user" -- env "${rootless_env[@]}" sh -c 'test -w "$HOME" && test -w "$XDG_CONFIG_HOME" && test -w "$XDG_DATA_HOME"' || die 'rootless Docker HOME and XDG directories must be writable by the runner'
if [[ -e $rootless_docker_unit || -L $rootless_docker_unit ]]; then
[[ -f $rootless_docker_unit && ! -L $rootless_docker_unit && $(stat -c %U "$rootless_docker_unit") == "$runner_user" ]] || die 'existing rootless Docker unit has an unexpected type or owner'
grep -Fq 'dockerd-rootless.sh' "$rootless_docker_unit" || die 'existing rootless Docker unit differs from the expected service'
else
runuser -u "$runner_user" -- env "${rootless_env[@]}" dockerd-rootless-setuptool.sh install
fi
runuser -u "$runner_user" -- env "${rootless_env[@]}" systemctl --user daemon-reload
runuser -u "$runner_user" -- env "${rootless_env[@]}" systemctl --user enable docker.service
if runuser -u "$runner_user" -- env "${rootless_env[@]}" systemctl --user is-active --quiet docker.service; then
runuser -u "$runner_user" -- env "${rootless_env[@]}" systemctl --user restart docker.service
else
runuser -u "$runner_user" -- env "${rootless_env[@]}" systemctl --user start docker.service
fi
docker_pid=$(runuser -u "$runner_user" -- env "${rootless_env[@]}" systemctl --user show docker.service --property=MainPID --value)
[[ $docker_pid =~ ^[1-9][0-9]*$ && -r /proc/$docker_pid/environ ]] || die 'rootless Docker process environment is unavailable'
docker_environment=$(tr '\0' '\n' <"/proc/$docker_pid/environ") || die 'could not read rootless Docker process environment'
for expected in "HOME=$runner_runtime_home" "XDG_CONFIG_HOME=$runner_xdg_config_home" "XDG_DATA_HOME=$runner_xdg_data_home"; do
grep -Fqx -- "$expected" <<<"$docker_environment" || die "running rootless Docker process is missing $expected"
done
printf '\nRunner runtime installed. Register it interactively, then enable its services as described in docs/operations/CI_RUNNER.md.\n'
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
"""Create or validate the runner image without following replaceable paths."""
import os
import stat
import subprocess
import sys
def open_directory(path: str) -> int:
current = os.open("/", os.O_RDONLY | os.O_DIRECTORY)
try:
for component in path.strip("/").split("/"):
if not component:
continue
next_fd = os.open(
component,
os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW,
dir_fd=current,
)
os.close(current)
current = next_fd
return current
except BaseException:
os.close(current)
raise
def filesystem_type(fd: int) -> str:
result = subprocess.run(
["blkid", "-p", "-s", "TYPE", "-o", "value", f"/proc/self/fd/{fd}"],
check=False,
capture_output=True,
text=True,
pass_fds=(fd,),
)
return result.stdout.strip() if result.returncode == 0 else ""
def main() -> int:
image, bytes_text, parent_uid_text, parent_mode_text = sys.argv[1:]
image = os.path.abspath(image)
parent_path, name = os.path.split(image)
if not name or name in {".", ".."}:
raise ValueError("invalid image filename")
parent_fd = open_directory(parent_path)
image_fd = -1
try:
parent_stat = os.fstat(parent_fd)
if parent_stat.st_uid != int(parent_uid_text) or stat.S_IMODE(parent_stat.st_mode) != int(parent_mode_text, 8):
raise RuntimeError("image parent has unexpected owner or permissions")
created = False
try:
image_fd = os.open(
name,
os.O_RDWR | os.O_NOFOLLOW | os.O_CLOEXEC,
dir_fd=parent_fd,
)
except FileNotFoundError:
image_fd = os.open(
name,
os.O_RDWR | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC,
0o600,
dir_fd=parent_fd,
)
created = True
image_stat = os.fstat(image_fd)
if not stat.S_ISREG(image_stat.st_mode) or image_stat.st_uid != os.geteuid() or image_stat.st_nlink != 1:
raise RuntimeError("image must be a singly-linked regular file owned by the installer")
if not created and image_stat.st_mode & 0o022:
raise RuntimeError("existing image is writable by group or other users")
if created:
os.posix_fallocate(image_fd, 0, int(bytes_text))
subprocess.run(
["mkfs.ext4", "-F", "-q", f"/proc/self/fd/{image_fd}"],
check=True,
pass_fds=(image_fd,),
)
image_stat = os.fstat(image_fd)
if image_stat.st_size != int(bytes_text):
raise RuntimeError(f"image size is {image_stat.st_size} bytes, expected {bytes_text}; refusing to resize or format")
if filesystem_type(image_fd) != "ext4":
raise RuntimeError("image does not contain ext4; refusing to format")
path_stat = os.stat(name, dir_fd=parent_fd, follow_symlinks=False)
if not stat.S_ISREG(path_stat.st_mode) or (path_stat.st_dev, path_stat.st_ino) != (image_stat.st_dev, image_stat.st_ino):
raise RuntimeError("image path changed during validation")
return 0
finally:
if image_fd >= 0:
os.close(image_fd)
os.close(parent_fd)
if __name__ == "__main__":
try:
raise SystemExit(main())
except (OSError, RuntimeError, ValueError, subprocess.CalledProcessError) as error:
print(f"ERROR: {error}", file=sys.stderr)
raise SystemExit(1)
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
set -euo pipefail
workflow=.gitea/workflows/ci.yaml
kaniko_action=.gitea/actions/kaniko-build/action.yml
caddy_action=.gitea/actions/caddy-validate/action.yml
grep -Fq 'runs-on: akyldash-ci' "$workflow"
grep -Fq 'uses: ./.gitea/actions/caddy-validate' "$workflow"
[[ $(grep -Fc 'uses: ./.gitea/actions/kaniko-build' "$workflow") -eq 2 ]]
check_workspace_wiring() {
local workflow_file=$1 caddy_file=${2:-$caddy_action} kaniko_file=${3:-$kaniko_action}
[[ $(grep -Fc 'workspace: ${{ github.workspace }}' "$workflow_file") -eq 3 ]] &&
grep -Fxq ' - ${{ inputs.workspace }}/deploy/production/Caddyfile' "$caddy_file" &&
grep -Fxq ' - --context=dir://${{ inputs.workspace }}' "$kaniko_file" &&
grep -Fxq ' - --dockerfile=${{ inputs.workspace }}/${{ inputs.dockerfile }}' "$kaniko_file" &&
! grep -Fq '/github/workspace' "$caddy_file" "$kaniko_file"
}
if ! check_workspace_wiring "$workflow"; then
echo 'Container actions must receive github.workspace and use it for repository paths.' >&2
exit 1
fi
grep -Fq 'dockerfile: deploy/production/Dockerfile.backend' "$workflow"
grep -Fq 'dockerfile: deploy/production/Dockerfile.opensearch' "$workflow"
grep -Fq 'docker compose --env-file deploy/production/.env.example -f deploy/production/compose.yaml config --quiet' "$workflow"
grep -Fxq ' entrypoint: /usr/bin/caddy' "$caddy_action"
grep -Fxq ' - validate' "$caddy_action"
grep -Fxq ' - --config' "$caddy_action"
grep -Fxq ' - ${{ inputs.workspace }}/deploy/production/Caddyfile' "$caddy_action"
grep -Fxq ' - --adapter' "$caddy_action"
grep -Fxq ' - caddyfile' "$caddy_action"
if grep -Eq 'docker (build|run)([[:space:]]|$)' "$workflow"; then
echo 'workflow must not invoke docker build/run in a job container' >&2
exit 1
fi
check_kaniko_action() {
grep -Fxq ' entrypoint: /kaniko/executor' "$1" &&
grep -Fxq ' - --context=dir://${{ inputs.workspace }}' "$1" &&
grep -Fxq ' - --dockerfile=${{ inputs.workspace }}/${{ inputs.dockerfile }}' "$1" &&
grep -Fxq ' - --destination=${{ inputs.destination }}' "$1" &&
grep -Fxq -- ' - --no-push' "$1" &&
grep -Fxq -- ' - --no-push-cache' "$1"
}
if ! check_kaniko_action "$kaniko_action"; then
echo 'Kaniko action must define its entrypoint, build arguments, and no-push flags.' >&2
exit 1
fi
without_entrypoint=$(mktemp)
without_no_push=$(mktemp)
without_workspace=$(mktemp)
wrong_workspace=$(mktemp)
trap 'rm -f "$without_entrypoint" "$without_no_push" "$without_workspace" "$wrong_workspace"' EXIT
sed '/^ entrypoint: \/kaniko\/executor$/d' "$kaniko_action" >"$without_entrypoint"
if check_kaniko_action "$without_entrypoint"; then
echo 'CI isolation test accepted a Kaniko action without its executor entrypoint.' >&2
exit 1
fi
sed '/^ - --no-push$/d' "$kaniko_action" >"$without_no_push"
if check_kaniko_action "$without_no_push"; then
echo 'CI isolation test accepted --no-push-cache without standalone --no-push.' >&2
exit 1
fi
sed '/workspace: \${{ github.workspace }}/d' "$workflow" >"$without_workspace"
if check_workspace_wiring "$without_workspace"; then
echo 'CI isolation test accepted container actions without workflow workspace inputs.' >&2
exit 1
fi
sed 's#${{ inputs.workspace }}#/github/workspace#g' "$kaniko_action" >"$wrong_workspace"
if check_workspace_wiring "$workflow" "$caddy_action" "$wrong_workspace"; then
echo 'CI isolation test accepted a fixed Kaniko workspace root.' >&2
exit 1
fi
grep -Fq 'FROM ghcr.io/osscontainertools/kaniko:v1.28.5-debug@sha256:c73ca67a3b494bce79a8fb6090c4f73c18997ab272d80bffc460a62343aeb412' .gitea/actions/kaniko-build/Dockerfile
grep -Fq 'FROM caddy:2.11.6-alpine@sha256:881bbc60f9986d5ab8e7cfd6cf7e4ef3c9c0439fef2429d035d065577882f028' .gitea/actions/caddy-validate/Dockerfile
if grep -REn 'docker.sock|privileged:|container.options:|^[[:space:]]*volumes:' "$workflow" .gitea/actions/kaniko-build .gitea/actions/caddy-validate; then
echo 'workflow and container actions must not request sockets, privileges, or additional host mounts' >&2
exit 1
fi
echo 'PASS CI container actions use the supplied workspace and remain isolated without socket or publish access'
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -Eeuo pipefail
script_dir=$(dirname "${BASH_SOURCE[0]}")
source "$script_dir/../test/run-as-unprivileged.sh"
output=$(mktemp)
trap 'rm -f "$output"' EXIT
if run_as_unprivileged bash "$script_dir/install-runner.sh" >"$output" 2>&1; then
echo 'Runner installer unexpectedly succeeded as an unprivileged UID.' >&2
exit 1
fi
grep -qx 'ERROR: run this script as root' "$output"
@@ -0,0 +1,251 @@
#!/usr/bin/env bash
set -Eeuo pipefail
runner_dir=$(cd "$(dirname "$0")" && pwd)
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
mkdir -p "$tmp/bin" "$tmp/store" "$tmp/partial-store"
log=$tmp/commands.log
export RUNNER_TEST_LOG=$log
cat >"$tmp/bin/mkfs.ext4" <<'EOF'
#!/usr/bin/env bash
printf 'mkfs\n' >>"$RUNNER_TEST_LOG"
if [[ -n ${RUNNER_TEST_REPLACE_IMAGE:-} ]]; then
rm -f "$RUNNER_TEST_IMAGE"
ln -s "$RUNNER_TEST_REPLACE_IMAGE" "$RUNNER_TEST_IMAGE"
fi
EOF
cat >"$tmp/bin/blkid" <<'EOF'
#!/usr/bin/env bash
printf '%s\n' "${RUNNER_TEST_FS:-ext4}"
EOF
cat >"$tmp/bin/systemctl" <<'EOF'
#!/usr/bin/env bash
printf '%s\n' "$*" >>"$RUNNER_TEST_LOG"
if [[ $1 == start ]]; then
[[ ${RUNNER_TEST_FAIL_START:-0} == 0 ]] || exit 1
: >"$RUNNER_TEST_MOUNTED"
fi
EOF
cat >"$tmp/bin/mountpoint" <<'EOF'
#!/usr/bin/env bash
[[ -e $RUNNER_TEST_MOUNTED ]]
EOF
cat >"$tmp/bin/findmnt" <<'EOF'
#!/usr/bin/env bash
printf '%s ext4 /dev/loop0 %s\n' "$RUNNER_TEST_STORE" "${RUNNER_TEST_OPTIONS:-rw,nosuid,nodev,relatime}"
EOF
cat >"$tmp/bin/losetup" <<'EOF'
#!/usr/bin/env bash
printf '%s\n' "$RUNNER_TEST_BACKING"
EOF
chmod +x "$tmp/bin/"*
PATH="$tmp/bin:$PATH"
export PATH
source "$runner_dir/install-runner-lib.sh"
runuser() {
shift 2
[[ $1 == -- ]] || return 2
shift
"$@"
}
getent() {
[[ $1 == passwd ]] || return 2
if [[ -n ${RUNNER_TEST_PASSWD:-} ]]; then
printf '%s\n' "$RUNNER_TEST_PASSWD"
elif [[ -r ${RUNNER_TEST_ACCOUNT_STATE:-} ]]; then
cat "$RUNNER_TEST_ACCOUNT_STATE"
else
return 2
fi
}
id() {
case $1 in
-gn) printf '%s\n' "${RUNNER_TEST_PRIMARY_GROUP:-akyldash-runner}" ;;
-nG) [[ ${TEST_EXTRA_GROUP:-0} == 1 ]] && printf 'akyldash-runner docker\n' || printf 'akyldash-runner\n' ;;
*) command id "$@" ;;
esac
}
stat() {
if [[ $1 == -c && $2 == %U && $3 == ${RUNNER_TEST_HOME:-} ]]; then
printf '%s\n' "${RUNNER_TEST_HOME_OWNER:-akyldash-runner}"
elif [[ $1 == -c && $2 == %G:%a && $3 == ${RUNNER_TEST_HOME:-} ]]; then
printf '%s:%s\n' "${RUNNER_TEST_HOME_GROUP:-root}" "${RUNNER_TEST_HOME_MODE:-711}"
else
command stat "$@"
fi
}
useradd() {
printf 'useradd\n' >>"$RUNNER_TEST_LOG"
local user=${@: -1} home=
while (($#)); do
if [[ $1 == --home-dir ]]; then home=$2; shift 2; else shift; fi
done
mkdir -p "$home"
printf '%s:x:990:990::%s:/usr/sbin/nologin\n' "$user" "$home" >"$RUNNER_TEST_ACCOUNT_STATE"
}
export RUNNER_TEST_HOME="$tmp/account-home"
export RUNNER_TEST_ACCOUNT_STATE="$tmp/account.passwd"
mkdir -p "$RUNNER_TEST_HOME"
export RUNNER_TEST_PASSWD="akyldash-runner:x:990:990::${RUNNER_TEST_HOME}:/usr/sbin/nologin"
runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME"
RUNNER_TEST_HOME_OWNER=root runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME"
if RUNNER_TEST_HOME_OWNER=root RUNNER_TEST_HOME_MODE=755 runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME" 2>/dev/null; then
echo 'Writable root-owned runner home was accepted' >&2
exit 1
fi
TEST_EXTRA_GROUP=1
if runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME" 2>/dev/null; then
echo 'Runner account with extra group membership was accepted' >&2
exit 1
fi
unset TEST_EXTRA_GROUP
if RUNNER_TEST_PASSWD="akyldash-runner:x:990:990::${tmp}/wrong-home:/usr/sbin/nologin" runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME" 2>/dev/null; then
echo 'Runner account with a different home was accepted' >&2
exit 1
fi
if RUNNER_TEST_PASSWD="akyldash-runner:x:0:990::${RUNNER_TEST_HOME}:/usr/sbin/nologin" runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME" 2>/dev/null; then
echo 'Runner account with UID 0 was accepted' >&2
exit 1
fi
if RUNNER_TEST_PASSWD="akyldash-runner:x:1000:990::${RUNNER_TEST_HOME}:/usr/sbin/nologin" runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME" 2>/dev/null; then
echo 'Runner account with a regular-user UID was accepted' >&2
exit 1
fi
RUNNER_TEST_PRIMARY_GROUP=unexpected
if runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME" 2>/dev/null; then
echo 'Runner account with an unexpected primary group was accepted' >&2
exit 1
fi
unset RUNNER_TEST_PRIMARY_GROUP
unset RUNNER_TEST_PASSWD
export RUNNER_TEST_HOME="$tmp/new-account-home"
runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME"
runner_ensure_account akyldash-runner "$RUNNER_TEST_HOME"
[[ $(grep -c '^useradd$' "$log") == 1 ]]
private_root=$tmp/private
mkdir -m 0700 "$private_root"
runner_owner=$(command id -un)
runner_group=$(command id -gn)
runner_ensure_private_directory "$private_root/config" "$runner_owner" "$runner_group"
runner_ensure_private_directory "$private_root/config" "$runner_owner" "$runner_group"
mkdir -m 0700 "$tmp/outside"
ln -s "$tmp/outside" "$private_root/linked"
if runner_ensure_private_directory "$private_root/linked/docker" "$runner_owner" "$runner_group" 2>/dev/null; then
echo 'Runner config symlink was accepted' >&2
exit 1
fi
[[ ! -e $tmp/outside/docker ]]
legacy_root=$tmp/legacy
mkdir -p "$legacy_root/.config/systemd/user/docker.service.d"
chmod 0755 "$legacy_root" "$legacy_root/.config" "$legacy_root/.config/systemd" "$legacy_root/.config/systemd/user" "$legacy_root/.config/systemd/user/docker.service.d"
for legacy_dir in "$legacy_root" "$legacy_root/.config" "$legacy_root/.config/systemd" "$legacy_root/.config/systemd/user" "$legacy_root/.config/systemd/user/docker.service.d"; do
runner_ensure_private_directory "$legacy_dir" "$runner_owner" "$runner_group"
[[ $(stat -c %a "$legacy_dir") == 700 ]]
done
user_config=$private_root/daemon.json
runner_install_user_content "$user_config" "$runner_owner" "$runner_group" 0600 <<'EOF'
{"data-root":"/private/runner"}
EOF
runner_install_user_content "$user_config" "$runner_owner" "$runner_group" 0600 <<'EOF'
{"data-root":"/private/runner"}
EOF
if runner_install_user_content "$user_config" "$runner_owner" "$runner_group" 0600 <<'EOF' 2>/dev/null
{"data-root":"/unexpected"}
EOF
then
echo 'Conflicting runner user config was accepted' >&2
exit 1
fi
image=$tmp/docker-store.img
runner_ensure_image "$image" 1048576 "$(command id -u)" 700
runner_ensure_image "$image" 1048576 "$(command id -u)" 700
[[ $(grep -c '^mkfs$' "$log") == 1 ]]
truncate -s 1048575 "$image"
if runner_ensure_image "$image" 1048576 "$(command id -u)" 700 2>/dev/null; then
echo 'Mismatched image size was accepted' >&2
exit 1
fi
[[ $(grep -c '^mkfs$' "$log") == 1 ]]
truncate -s 1048576 "$image"
if RUNNER_TEST_FS=ext3 runner_ensure_image "$image" 1048576 "$(command id -u)" 700 2>/dev/null; then
echo 'Existing non-ext4 image was accepted' >&2
exit 1
fi
[[ $(grep -c '^mkfs$' "$log") == 1 ]]
external=$tmp/external-file
printf 'preserve this file\n' >"$external"
race_image=$tmp/race.img
ln -s "$external" "$race_image"
if runner_ensure_image "$race_image" 1048576 "$(command id -u)" 700 2>/dev/null; then
echo 'Symlink runner image was accepted' >&2
exit 1
fi
[[ $(cat "$external") == 'preserve this file' ]]
rm -f "$race_image"
race_image=$tmp/race-during-format.img
export RUNNER_TEST_IMAGE="$race_image" RUNNER_TEST_REPLACE_IMAGE="$external"
if runner_ensure_image "$race_image" 1048576 "$(command id -u)" 700 2>/dev/null; then
echo 'Image path replacement during formatting was accepted' >&2
exit 1
fi
[[ $(cat "$external") == 'preserve this file' ]]
rm -f "$race_image"
unset RUNNER_TEST_IMAGE RUNNER_TEST_REPLACE_IMAGE
[[ $(grep -c '^mkfs$' "$log") == 2 ]]
unit_name='var-lib-akyldash\x2drunner-docker\x2ddata.mount'
unit_file=$tmp/systemd/$unit_name
mkdir -p "$(dirname "$unit_file")"
export RUNNER_TEST_STORE="$tmp/store" RUNNER_TEST_BACKING="$image"
export RUNNER_TEST_MOUNTED="$tmp/mounted"
runner_ensure_mount "$image" "$RUNNER_TEST_STORE" "$unit_name" "$unit_file"
[[ $(grep -c '^start ' "$log") == 1 ]]
grep -Fq "enable $unit_name" "$log"
grep -Fq "start $unit_name" "$log"
# A second run with the image mounted must reuse it and avoid another start.
runner_ensure_image "$image" 1048576 "$(command id -u)" 700
runner_ensure_mount "$image" "$RUNNER_TEST_STORE" "$unit_name" "$unit_file"
[[ $(grep -c '^mkfs$' "$log") == 2 ]]
[[ $(grep -c '^start ' "$log") == 1 ]]
chmod 0664 "$unit_file"
if runner_ensure_mount "$image" "$RUNNER_TEST_STORE" "$unit_name" "$unit_file" 2>/dev/null; then
echo 'Writable systemd mount unit was accepted' >&2
exit 1
fi
chmod 0644 "$unit_file"
export RUNNER_TEST_BACKING="$tmp/unrelated.img"
touch "$RUNNER_TEST_BACKING"
if runner_ensure_mount "$image" "$RUNNER_TEST_STORE" "$unit_name" "$unit_file" 2>/dev/null; then
echo 'Active mount backed by a different image was accepted' >&2
exit 1
fi
[[ $(grep -c '^start ' "$log") == 1 ]]
export RUNNER_TEST_BACKING="$image"
# Simulate interruption after unit installation/enable, before mount activation.
export RUNNER_TEST_STORE="$tmp/partial-store" RUNNER_TEST_MOUNTED="$tmp/partial-mounted"
export RUNNER_TEST_FAIL_START=1
if runner_ensure_mount "$image" "$RUNNER_TEST_STORE" "$unit_name" "$tmp/systemd/partial.mount" 2>/dev/null; then
echo 'Injected mount interruption was not observed' >&2
exit 1
fi
export RUNNER_TEST_FAIL_START=0
runner_ensure_image "$image" 1048576 "$(command id -u)" 700
runner_ensure_mount "$image" "$RUNNER_TEST_STORE" "$unit_name" "$tmp/systemd/partial.mount"
[[ $(grep -c '^mkfs$' "$log") == 2 ]]
[[ $(grep -c '^start ' "$log") == 3 ]]
export RUNNER_TEST_STORE="$tmp/store" RUNNER_TEST_MOUNTED="$tmp/mounted" RUNNER_TEST_OPTIONS=rw,relatime
if runner_validate_mount "$image" "$RUNNER_TEST_STORE" 2>/dev/null; then
echo 'Runner mount missing nosuid/nodev was accepted' >&2
exit 1
fi
unset RUNNER_TEST_OPTIONS
echo 'runner installer idempotency checks passed'
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env bash
set -Eeuo pipefail
config=infra/runner/config.yaml
installer=infra/runner/install-runner.sh
assert_absent() {
local pattern=$1 file=$2
if grep -Eq "$pattern" "$file"; then
echo "Forbidden content matched in $file" >&2
return 1
fi
}
grep -Eq '^ capacity: 1$' "$config"
grep -Fq 'akyldash-ci:docker://docker.gitea.com/runner-images:ubuntu-24.04' "$config"
grep -Fq 'docker_host: "-"' "$config"
grep -Eq '^ privileged: false$' "$config"
grep -Eq '^ valid_volumes: \[\]$' "$config"
grep -Fq -- '--cpus=2 --memory=3g --pids-limit=512' "$config"
assert_absent 'var/run/docker.sock|metrics:|health_check:|production.env|review_secret' "$config"
grep -Fq 'docker-ce-rootless-extras' "$installer"
grep -Fq 'runner_ensure_image "$runner_image" "$runner_image_bytes" 0 711' "$installer"
grep -Fq 'runner_lock_home "$runner_home"' "$installer"
grep -Fq 'test -w "$HOME" && test -w "$XDG_CONFIG_HOME" && test -w "$XDG_DATA_HOME"' "$installer"
grep -Fq 'rootless_env=(HOME="$runner_runtime_home" XDG_CONFIG_HOME="$runner_xdg_config_home" XDG_DATA_HOME="$runner_xdg_data_home"' "$installer"
grep -Fq 'Environment=XDG_CONFIG_HOME=$runner_xdg_config_home' "$installer"
grep -Fq 'Environment=XDG_DATA_HOME=$runner_xdg_data_home' "$installer"
grep -Fq 'runner_home/.config/systemd/user/docker.service.d/environment.conf' "$installer"
grep -Fq 'Environment=HOME=$runner_runtime_home' "$installer"
grep -Fq 'systemctl --user restart docker.service' "$installer"
grep -Fq 'systemctl --user show docker.service --property=MainPID --value' "$installer"
grep -Fq 'tr '\''\0'\'' '\''\n'\'' <"/proc/$docker_pid/environ"' "$installer"
grep -Fq 'grep -Fqx -- "$expected"' "$installer"
if grep -Fq 'systemctl --user set-environment' "$installer"; then
echo 'Rootless Docker environment must persist in its user unit drop-in' >&2
exit 1
fi
grep -Fq 'HOME=/var/lib/akyldash-runner/.config/act-runner' infra/runner/act-runner-healthcheck.sh
grep -Fq 'XDG_CONFIG_HOME=/var/lib/akyldash-runner/.config' infra/runner/act-runner-healthcheck.sh
grep -Fq 'XDG_DATA_HOME=/var/lib/akyldash-runner/.config/act-runner/.local/share' infra/runner/act-runner-healthcheck.sh
if grep -Fq 'env HOME="$runner_home"' "$installer"; then
echo 'Protected runner home must not be passed as rootless Docker HOME' >&2
exit 1
fi
grep -Fq 'die '\''stop akyldash-act-runner.service after all jobs finish before rerunning this installer' "$installer"
grep -Fq 'ReadWritePaths=$runner_runtime_home $runner_xdg_config_home /var/log/akyldash-runner /run/user/$runner_uid' "$installer"
grep -Fq 'os.O_NOFOLLOW | os.O_CLOEXEC' infra/runner/secure-runner-image.py
grep -Fq 'os.posix_fallocate(image_fd, 0, int(bytes_text))' infra/runner/secure-runner-image.py
grep -Fq 'pass_fds=(image_fd,)' infra/runner/secure-runner-image.py
grep -Fq 'systemctl enable "$unit_name"' infra/runner/install-runner-lib.sh
grep -Fq 'systemctl start "$unit_name"' infra/runner/install-runner-lib.sh
grep -Fq 'runuser -u "$owner" -- install -d -m 0700 "$directory"' infra/runner/install-runner-lib.sh
grep -Fq 'runner_install_user_content "$runner_home/.config/docker/daemon.json"' "$installer"
assert_absent 'basename.*runner_mount_unit|enable --now.*runner_mount_unit' "$installer"
grep -Fq 'groups=$(id -nG "$user")' infra/runner/install-runner-lib.sh
grep -Fq 'id -gn "$user"' infra/runner/install-runner-lib.sh
assert_absent 'dockerd-rootless-setuptool.sh install --force' "$installer"
grep -Fq 'runner_sha256=3acac8b506ac8cadc88a55155b5d6378f0fab0b8f62d1e0c0450f4ccd69733e2' "$installer"
grep -Fq 'find-free-subid-range.awk' "$installer"
grep -Fq 'usermod "${subid_args[@]}" "$runner_user"' "$installer"
grep -Fq 'subid_args+=(--add-subuids "$subid_range")' "$installer"
assert_absent 'usermod.*(sudo|docker)|/var/run/docker.sock|/etc/sudoers' "$installer"
grep -Fq 'runs-on: akyldash-ci' .gitea/workflows/runner-smoke.yaml
grep -Fq 'test ! -S /var/run/docker.sock' .gitea/workflows/runner-smoke.yaml
grep -Fq 'test ! -e /etc/akyldash/production.env' .gitea/workflows/runner-smoke.yaml
grep -Fq 'test ! -e /etc/akyldash/review_secret' .gitea/workflows/runner-smoke.yaml
assert_absent '(^|[[:space:]])(docker|docker-compose|deploy)([[:space:]]|$)|production.env=|review_secret=' .gitea/workflows/runner-smoke.yaml
fixture=$(mktemp)
fixture_g=$(mktemp)
trap 'rm -f "$fixture" "$fixture_g"' EXIT
printf 'production.env\n' >"$fixture"
if assert_absent 'deploy|production.env|review_secret|docker.sock' "$fixture" 2>/dev/null; then
echo 'Negative guard regression: prohibited fixture was accepted' >&2
exit 1
fi
printf 'codex:100000:65536\n' >"$fixture"
printf 'codex:165000:1000\n' >"$fixture_g"
range=$(awk -f infra/runner/find-free-subid-range.awk "$fixture" "$fixture_g")
[[ $range == 231072-296607 ]]
: >"$fixture_g"
range=$(awk -f infra/runner/find-free-subid-range.awk "$fixture" "$fixture_g")
[[ $range == 165536-231071 ]]
: >"$fixture"
range=$(awk -f infra/runner/find-free-subid-range.awk "$fixture" "$fixture_g")
[[ $range == 100000-165535 ]]
printf 'broken\n' >"$fixture_g"
if awk -f infra/runner/find-free-subid-range.awk "$fixture" "$fixture_g" >/dev/null 2>&1; then
echo 'Malformed subordinate ID allocation was accepted' >&2
exit 1
fi
echo 'runner configuration guards passed'
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
run_as_unprivileged() {
if [[ $EUID -eq 0 ]]; then
if ! command -v setpriv >/dev/null 2>&1; then
echo 'setpriv is required to run guard tests as an unprivileged UID.' >&2
return 127
fi
setpriv --reuid=65534 --regid=65534 --clear-groups -- "$@"
else
"$@"
fi
}