Guard volume gaps and back up initialized production

deploy/production/README.md

@@ -43,10 +43,13 @@ containers exist. Continue without a backup in that case. If any persistent
 data exists, create and verify a backup before changing schema, images, or
 starting a new application version. This also applies when containers are
 absent, including an initial installation where the normalized corpus was
-preloaded under `DATA_ROOT`; create the archive before the first `up`. Missing
-named volumes are recorded as empty backup components when Docker confirms
-they do not exist. The archive tool refuses ambiguous storage state, running
-containers, and incomplete Compose service containers. See
+preloaded under `DATA_ROOT`; create the archive before the first `up`. When
+named volumes do not exist, pass `--initial-deployment`; this is accepted only
+without Compose containers and when `DATA_ROOT` contains only
+`minjust-normalized/`. During upgrades, missing volumes are an error because
+they can indicate lost production data. The archive tool also refuses
+ambiguous storage state, running containers, and incomplete Compose service
+containers. See
 [`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md) for the exact
 commands. Do not remove an older usable backup to make room; retain the
 pre-change archive until the new deployment and a newer verified backup are
@@ -76,6 +79,13 @@ the `data` backup. Admin password reset is also interactive:
 sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --reset-admin-password
 ```
 
+After the initial OpenSearch index is loaded and validated, the `admin` account
+is created, and `/review` works over HTTPS, create and verify the required
+post-setup baseline backup before treating deployment as complete. It captures
+the initialized index, account database, and Caddy certificate data. Follow
+the exact stop/create/verify/start sequence in
+[`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md#initial-post-setup-baseline).
+
 ## Backup, restore, and rollback
 
 Backups are cold and require downtime. Stop services, create the archive, then

docs/operations/BACKUP.md

@@ -24,12 +24,28 @@ and `/var/backups/akyldash` remains a planned root-only location.
 ## Clean first deployment
 
 No backup is needed only when Compose has no containers and all four
-persistent mounts are confirmed empty. Named volumes not created yet count as
-empty on a clean first deployment. If data has already been placed in a bind
-mount, create the archive before the first `up`; missing named volumes are
-recorded as empty components. Missing mount definitions, inaccessible Docker
-state, or partial service containers are errors and must stop deployment. Do
-not create placeholder containers to make backup pass.
+persistent mounts are confirmed empty. For a first install with a preloaded
+corpus and not-yet-created named volumes, use `--initial-deployment` to create
+the pre-start backup. This flag is accepted only with no Compose containers and
+when `DATA_ROOT` contains exactly `minjust-normalized/`; it archives absent
+named volumes as empty components. In every normal backup, an absent volume is
+an error because it could mean that a previous installation lost data.
+Missing mount definitions, inaccessible Docker state, or partial service
+containers are also errors and must stop deployment. Do not create placeholder
+containers to make backup pass.
+
+For the corpus-only initial case, run this before the first `up` from the
+approved checkout. Replace the timestamp with one real UTC timestamp:
+
+```bash
+cd /opt/akyldash/current
+sudo python3 infra/backup/archive.py create \
+  --compose-file deploy/production/compose.yaml \
+  --env-file /etc/akyldash/production.env \
+  --initial-deployment \
+  --output /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
+sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz
+```
 
 ## Cold backup procedure
 
@@ -48,9 +64,11 @@ sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYM
 
 Replace the timestamp placeholder with one real UTC timestamp before running.
 On a clean first deployment, `create` exits with a clear message that backup
-can be skipped. With any existing persistent data it archives the available
-storage even when no Compose containers exist; partial service containers or
-running services are rejected as unsafe. The create command archives the application data,
+can be skipped. With existing data and all expected volumes present, it
+archives the storage even when no Compose containers exist. A preloaded corpus
+with no named volumes requires the explicit `--initial-deployment` mode;
+otherwise missing volumes are rejected. Partial service containers or running
+services are rejected as unsafe. The create command archives the application data,
 OpenSearch volume, Caddy data/config volumes, production environment file, and
 review secret. It records the checked-out Git revision, deployment-file hashes,
 and image references/IDs for each stopped Compose service, writes a SHA-256
@@ -62,6 +80,30 @@ cd /opt/akyldash/current
 sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
 ```
 
+## Initial post-setup baseline
+
+After the first index has been loaded and validated, the initial `admin` account
+has been created, and `/review` responds over HTTPS, create a second backup.
+This one captures the initialized index, account database, and newly created
+Caddy/OpenSearch volumes. Stop all services, create and verify the archive,
+then start the stack again:
+
+```bash
+cd /opt/akyldash/current
+sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop
+sudo python3 infra/backup/archive.py create \
+  --compose-file deploy/production/compose.yaml \
+  --env-file /etc/akyldash/production.env \
+  --output /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
+sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz
+sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d
+```
+
+Use one real UTC timestamp in both archive paths. Do not treat initial
+production setup as complete until this baseline archive verifies. Keep the
+pre-start corpus archive as well until the baseline is verified and copied to
+separately managed encrypted storage.
+
 Keep the local archive root-only, then copy it to separately managed encrypted
 storage. A local archive alone does not protect against loss of the VPS disk.
 Restore extracts only to a new `akyldash-restore-*` directory under `/tmp` or

docs/operations/CURRENT_STATE.md

@@ -51,8 +51,10 @@ from planned configuration.
   tests use synthetic data; neither tool has run against production. The
   archive preflight now permits a clean first deployment to skip backup,
   requires backup for existing persistent data even without containers,
-  records not-yet-created named volumes as empty, and rejects inaccessible or
-  ambiguous mounts and incomplete/running service containers.
+  allows absent named volumes only in explicit corpus-only initial-deployment
+  mode, and rejects inaccessible/ambiguous mounts and incomplete/running
+  service containers. Initial deployment docs require a second verified backup
+  after index and admin initialization.
 - Recovery bootstrap has an explicit `--recovery-target` mode that refuses the
   inventoried production hostname/IP; its host-identity guard has shell tests.
   The mode has not been run on a VPS.

docs/operations/DEPLOYMENT.md

@@ -35,10 +35,14 @@ proxy/firewall configuration, and a backup and restore have been tested.
    verification.
 5. For a clean first deployment, skip backup only when Compose confirms there
    are no containers and all persistent mounts are empty. If any data exists,
-   create and verify a backup before starting a new version. An uncertain or
-   partial state stops deployment. Then start services, load and validate the
-   OpenSearch index, create the initial admin interactively, and verify
-   `/review` over HTTPS. Record the backend image ID for drift checks:
+   create and verify a backup before starting a new version. For a preloaded
+   corpus with absent named volumes, use `--initial-deployment`; it is limited
+   to a container-free host whose `DATA_ROOT` contains only
+   `minjust-normalized/`. Normal backups reject absent volumes. An uncertain
+   or partial state stops deployment. Then start services, load and validate
+   the OpenSearch index, create the initial admin interactively, verify
+   `/review` over HTTPS, and create the required post-setup baseline backup
+   described in `BACKUP.md`. Record the backend image ID for drift checks:
 
    ```bash
    sudo sh -c 'docker compose --env-file /etc/akyldash/production.env -f /opt/akyldash/current/deploy/production/compose.yaml ps --quiet backend | xargs -r docker inspect --format "{{.Image}}" > /opt/akyldash/deployed-image.txt'

infra/backup/archive.py

@@ -172,9 +172,16 @@ def has_persistent_data(sources: dict[str, Path | None]) -> bool:
     return any(any(path.iterdir()) for path in sources.values() if path is not None)
 
 
+def is_preloaded_corpus_only(data: Path | None) -> bool:
+    if data is None:
+        return False
+    entries = list(data.iterdir())
+    return len(entries) == 1 and entries[0].name == "minjust-normalized" and entries[0].is_dir() and not entries[0].is_symlink()
+
+
 def compose_sources(
-    compose_file: Path, env_file: Path
-) -> tuple[dict[str, Path], str, dict[str, dict[str, str]], dict[str, str]]:
+    compose_file: Path, env_file: Path, initial_deployment: bool = False
+) -> tuple[dict[str, Path | None], str, dict[str, dict[str, str]], dict[str, str]]:
     command = ["docker", "compose", "--env-file", str(env_file), "-f", str(compose_file)]
     rendered = subprocess.run(
         [*command, "config", "--format", "json"], check=True, capture_output=True, text=True
@@ -188,8 +195,13 @@ def compose_sources(
     persistent_data = has_persistent_data(storage)
     if not containers and not persistent_data:
         raise ValueError("no backup is required for a clean first deployment; skip backup and start the initial stack")
-    if containers and any(path is None for path in storage.values()):
-        raise ValueError("persistent Compose volumes are missing while data exists; refusing an incomplete backup")
+    missing_volumes = any(path is None for path in storage.values())
+    if missing_volumes and containers:
+        raise ValueError("persistent Compose volumes are missing while containers exist; refusing an incomplete backup")
+    if missing_volumes and not initial_deployment:
+        raise ValueError("persistent Compose volumes are missing; refusing an incomplete backup (use --initial-deployment only for a new host with a preloaded corpus)")
+    if missing_volumes and not is_preloaded_corpus_only(storage["data"]):
+        raise ValueError("initial deployment with missing volumes requires DATA_ROOT to contain only minjust-normalized/; refusing an incomplete backup")
     images: dict[str, dict[str, str]] = {}
     for container in containers:
         result = subprocess.run(
@@ -350,6 +362,7 @@ def main() -> int:
     create.add_argument("--output", required=True, type=Path)
     create.add_argument("--compose-file", required=True, type=Path)
     create.add_argument("--env-file", required=True, type=Path)
+    create.add_argument("--initial-deployment", action="store_true", help="allow absent named volumes only for a new host containing only the preloaded corpus")
     verify = subparsers.add_parser("verify", help="verify archive checksum and contents")
     verify.add_argument("archive", type=Path)
     restore = subparsers.add_parser("restore", help="extract into a new staging directory")
@@ -360,7 +373,9 @@ def main() -> int:
         if args.command in {"create", "restore"} and os.geteuid() != 0:
             raise ValueError("create and restore must run as root to protect production secrets and data")
         if args.command == "create":
-            sources, revision, images, deployment_files = compose_sources(args.compose_file, args.env_file)
+            sources, revision, images, deployment_files = compose_sources(
+                args.compose_file, args.env_file, args.initial_deployment
+            )
             create_backup(args.output, sources, revision, images, deployment_files)
         elif args.command == "verify":
             manifest = verify_archive(args.archive)

infra/backup/test_archive.py

@@ -238,12 +238,43 @@ class BackupArchiveTests(unittest.TestCase):
             with self.assertRaisesRegex(ValueError, "not all configured Compose services"):
                 compose_sources(Path.cwd() / "deploy/production/compose.yaml", env_file)
 
-    def test_existing_data_with_uncreated_named_volume_is_backed_up_as_empty(self):
+    def test_missing_named_volume_with_existing_data_requires_explicit_initial_mode(self):
         config, volume_paths, env_file, _secret = self.compose_fixture()
         (self.root / "data/search-users.sqlite3").unlink()
         (self.root / "data/existing.json").write_text("data", encoding="utf-8")
         available = set(volume_paths) - {"akyldash_os-data"}
 
+        def run(command, **_kwargs):
+            if "config" in command:
+                return SimpleNamespace(stdout=json.dumps(config))
+            if "ps" in command:
+                return SimpleNamespace(stdout="")
+            if command[1] == "volume" and "ls" in command:
+                return SimpleNamespace(stdout="\n".join(available) + "\n")
+            if command[1] == "volume":
+                return SimpleNamespace(stdout=f"{volume_paths[command[-1]]}\n")
+            if "status" in command:
+                return SimpleNamespace(stdout="")
+            if "rev-parse" in command:
+                return SimpleNamespace(stdout="abc123\n")
+            raise AssertionError(command)
+
+        with patch("archive.subprocess.run", side_effect=run):
+            with self.assertRaisesRegex(ValueError, "use --initial-deployment"):
+                compose_sources(Path.cwd() / "deploy/production/compose.yaml", env_file)
+            with self.assertRaisesRegex(ValueError, "contain only minjust-normalized"):
+                compose_sources(
+                    Path.cwd() / "deploy/production/compose.yaml", env_file, initial_deployment=True
+                )
+
+    def test_initial_mode_backs_up_only_preloaded_corpus_when_volumes_are_absent(self):
+        config, volume_paths, env_file, _secret = self.compose_fixture()
+        (self.root / "data/search-users.sqlite3").unlink()
+        corpus = self.root / "data/minjust-normalized"
+        corpus.mkdir()
+        (corpus / "document.json").write_text("corpus", encoding="utf-8")
+        available = set(volume_paths) - {"akyldash_os-data"}
+
         def run(command, **_kwargs):
             if "config" in command:
                 return SimpleNamespace(stdout=json.dumps(config))
@@ -261,15 +292,16 @@ class BackupArchiveTests(unittest.TestCase):
 
         with patch("archive.subprocess.run", side_effect=run):
             sources, revision, images, deployment_files = compose_sources(
-                Path.cwd() / "deploy/production/compose.yaml", env_file
+                Path.cwd() / "deploy/production/compose.yaml", env_file, initial_deployment=True
             )
         self.assertIsNone(sources["opensearch"])
-        archive = self.private / "bind-only-backup.tar.gz"
+        archive = self.private / "initial-corpus-backup.tar.gz"
         create_backup(archive, sources, revision, images, deployment_files)
         manifest = verify_archive(archive)
         self.assertEqual(manifest["git_revision"], "abc123")
         with tarfile.open(archive, "r:gz") as saved:
             self.assertTrue(saved.getmember("opensearch/").isdir())
+            self.assertTrue(saved.getmember("data/minjust-normalized/document.json").isfile())
 
     def test_compose_backup_records_sources_and_image_ids(self):
         data = self.root / "production-data"