diff --git a/deploy/production/README.md b/deploy/production/README.md index a24e153..ec0b0d3 100644 --- a/deploy/production/README.md +++ b/deploy/production/README.md @@ -43,10 +43,13 @@ containers exist. Continue without a backup in that case. If any persistent data exists, create and verify a backup before changing schema, images, or starting a new application version. This also applies when containers are absent, including an initial installation where the normalized corpus was -preloaded under `DATA_ROOT`; create the archive before the first `up`. Missing -named volumes are recorded as empty backup components when Docker confirms -they do not exist. The archive tool refuses ambiguous storage state, running -containers, and incomplete Compose service containers. See +preloaded under `DATA_ROOT`; create the archive before the first `up`. When +named volumes do not exist, pass `--initial-deployment`; this is accepted only +without Compose containers and when `DATA_ROOT` contains only +`minjust-normalized/`. During upgrades, missing volumes are an error because +they can indicate lost production data. The archive tool also refuses +ambiguous storage state, running containers, and incomplete Compose service +containers. See [`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md) for the exact commands. Do not remove an older usable backup to make room; retain the pre-change archive until the new deployment and a newer verified backup are @@ -76,6 +79,13 @@ the `data` backup. Admin password reset is also interactive: sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml exec backend python -m search.api --reset-admin-password ``` +After the initial OpenSearch index is loaded and validated, the `admin` account +is created, and `/review` works over HTTPS, create and verify the required +post-setup baseline backup before treating deployment as complete. It captures +the initialized index, account database, and Caddy certificate data. Follow +the exact stop/create/verify/start sequence in +[`docs/operations/BACKUP.md`](../../docs/operations/BACKUP.md#initial-post-setup-baseline). + ## Backup, restore, and rollback Backups are cold and require downtime. Stop services, create the archive, then diff --git a/docs/operations/BACKUP.md b/docs/operations/BACKUP.md index 2f77037..2a85e2e 100644 --- a/docs/operations/BACKUP.md +++ b/docs/operations/BACKUP.md @@ -24,12 +24,28 @@ and `/var/backups/akyldash` remains a planned root-only location. ## Clean first deployment No backup is needed only when Compose has no containers and all four -persistent mounts are confirmed empty. Named volumes not created yet count as -empty on a clean first deployment. If data has already been placed in a bind -mount, create the archive before the first `up`; missing named volumes are -recorded as empty components. Missing mount definitions, inaccessible Docker -state, or partial service containers are errors and must stop deployment. Do -not create placeholder containers to make backup pass. +persistent mounts are confirmed empty. For a first install with a preloaded +corpus and not-yet-created named volumes, use `--initial-deployment` to create +the pre-start backup. This flag is accepted only with no Compose containers and +when `DATA_ROOT` contains exactly `minjust-normalized/`; it archives absent +named volumes as empty components. In every normal backup, an absent volume is +an error because it could mean that a previous installation lost data. +Missing mount definitions, inaccessible Docker state, or partial service +containers are also errors and must stop deployment. Do not create placeholder +containers to make backup pass. + +For the corpus-only initial case, run this before the first `up` from the +approved checkout. Replace the timestamp with one real UTC timestamp: + +```bash +cd /opt/akyldash/current +sudo python3 infra/backup/archive.py create \ + --compose-file deploy/production/compose.yaml \ + --env-file /etc/akyldash/production.env \ + --initial-deployment \ + --output /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz +sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-corpus-YYYYMMDDTHHMMSSZ.tar.gz +``` ## Cold backup procedure @@ -48,9 +64,11 @@ sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-YYYYM Replace the timestamp placeholder with one real UTC timestamp before running. On a clean first deployment, `create` exits with a clear message that backup -can be skipped. With any existing persistent data it archives the available -storage even when no Compose containers exist; partial service containers or -running services are rejected as unsafe. The create command archives the application data, +can be skipped. With existing data and all expected volumes present, it +archives the storage even when no Compose containers exist. A preloaded corpus +with no named volumes requires the explicit `--initial-deployment` mode; +otherwise missing volumes are rejected. Partial service containers or running +services are rejected as unsafe. The create command archives the application data, OpenSearch volume, Caddy data/config volumes, production environment file, and review secret. It records the checked-out Git revision, deployment-file hashes, and image references/IDs for each stopped Compose service, writes a SHA-256 @@ -62,6 +80,30 @@ cd /opt/akyldash/current sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d ``` +## Initial post-setup baseline + +After the first index has been loaded and validated, the initial `admin` account +has been created, and `/review` responds over HTTPS, create a second backup. +This one captures the initialized index, account database, and newly created +Caddy/OpenSearch volumes. Stop all services, create and verify the archive, +then start the stack again: + +```bash +cd /opt/akyldash/current +sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml stop +sudo python3 infra/backup/archive.py create \ + --compose-file deploy/production/compose.yaml \ + --env-file /etc/akyldash/production.env \ + --output /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz +sudo python3 infra/backup/archive.py verify /var/backups/akyldash/akyldash-initial-baseline-YYYYMMDDTHHMMSSZ.tar.gz +sudo docker compose --env-file /etc/akyldash/production.env -f deploy/production/compose.yaml up -d +``` + +Use one real UTC timestamp in both archive paths. Do not treat initial +production setup as complete until this baseline archive verifies. Keep the +pre-start corpus archive as well until the baseline is verified and copied to +separately managed encrypted storage. + Keep the local archive root-only, then copy it to separately managed encrypted storage. A local archive alone does not protect against loss of the VPS disk. Restore extracts only to a new `akyldash-restore-*` directory under `/tmp` or diff --git a/docs/operations/CURRENT_STATE.md b/docs/operations/CURRENT_STATE.md index 0d4d69c..5aaf858 100644 --- a/docs/operations/CURRENT_STATE.md +++ b/docs/operations/CURRENT_STATE.md @@ -51,8 +51,10 @@ from planned configuration. tests use synthetic data; neither tool has run against production. The archive preflight now permits a clean first deployment to skip backup, requires backup for existing persistent data even without containers, - records not-yet-created named volumes as empty, and rejects inaccessible or - ambiguous mounts and incomplete/running service containers. + allows absent named volumes only in explicit corpus-only initial-deployment + mode, and rejects inaccessible/ambiguous mounts and incomplete/running + service containers. Initial deployment docs require a second verified backup + after index and admin initialization. - Recovery bootstrap has an explicit `--recovery-target` mode that refuses the inventoried production hostname/IP; its host-identity guard has shell tests. The mode has not been run on a VPS. diff --git a/docs/operations/DEPLOYMENT.md b/docs/operations/DEPLOYMENT.md index c43c9a4..5ec86dd 100644 --- a/docs/operations/DEPLOYMENT.md +++ b/docs/operations/DEPLOYMENT.md @@ -35,10 +35,14 @@ proxy/firewall configuration, and a backup and restore have been tested. verification. 5. For a clean first deployment, skip backup only when Compose confirms there are no containers and all persistent mounts are empty. If any data exists, - create and verify a backup before starting a new version. An uncertain or - partial state stops deployment. Then start services, load and validate the - OpenSearch index, create the initial admin interactively, and verify - `/review` over HTTPS. Record the backend image ID for drift checks: + create and verify a backup before starting a new version. For a preloaded + corpus with absent named volumes, use `--initial-deployment`; it is limited + to a container-free host whose `DATA_ROOT` contains only + `minjust-normalized/`. Normal backups reject absent volumes. An uncertain + or partial state stops deployment. Then start services, load and validate + the OpenSearch index, create the initial admin interactively, verify + `/review` over HTTPS, and create the required post-setup baseline backup + described in `BACKUP.md`. Record the backend image ID for drift checks: ```bash sudo sh -c 'docker compose --env-file /etc/akyldash/production.env -f /opt/akyldash/current/deploy/production/compose.yaml ps --quiet backend | xargs -r docker inspect --format "{{.Image}}" > /opt/akyldash/deployed-image.txt' diff --git a/infra/backup/archive.py b/infra/backup/archive.py index 4c720e2..8175fcc 100644 --- a/infra/backup/archive.py +++ b/infra/backup/archive.py @@ -172,9 +172,16 @@ def has_persistent_data(sources: dict[str, Path | None]) -> bool: return any(any(path.iterdir()) for path in sources.values() if path is not None) +def is_preloaded_corpus_only(data: Path | None) -> bool: + if data is None: + return False + entries = list(data.iterdir()) + return len(entries) == 1 and entries[0].name == "minjust-normalized" and entries[0].is_dir() and not entries[0].is_symlink() + + def compose_sources( - compose_file: Path, env_file: Path -) -> tuple[dict[str, Path], str, dict[str, dict[str, str]], dict[str, str]]: + compose_file: Path, env_file: Path, initial_deployment: bool = False +) -> tuple[dict[str, Path | None], str, dict[str, dict[str, str]], dict[str, str]]: command = ["docker", "compose", "--env-file", str(env_file), "-f", str(compose_file)] rendered = subprocess.run( [*command, "config", "--format", "json"], check=True, capture_output=True, text=True @@ -188,8 +195,13 @@ def compose_sources( persistent_data = has_persistent_data(storage) if not containers and not persistent_data: raise ValueError("no backup is required for a clean first deployment; skip backup and start the initial stack") - if containers and any(path is None for path in storage.values()): - raise ValueError("persistent Compose volumes are missing while data exists; refusing an incomplete backup") + missing_volumes = any(path is None for path in storage.values()) + if missing_volumes and containers: + raise ValueError("persistent Compose volumes are missing while containers exist; refusing an incomplete backup") + if missing_volumes and not initial_deployment: + raise ValueError("persistent Compose volumes are missing; refusing an incomplete backup (use --initial-deployment only for a new host with a preloaded corpus)") + if missing_volumes and not is_preloaded_corpus_only(storage["data"]): + raise ValueError("initial deployment with missing volumes requires DATA_ROOT to contain only minjust-normalized/; refusing an incomplete backup") images: dict[str, dict[str, str]] = {} for container in containers: result = subprocess.run( @@ -350,6 +362,7 @@ def main() -> int: create.add_argument("--output", required=True, type=Path) create.add_argument("--compose-file", required=True, type=Path) create.add_argument("--env-file", required=True, type=Path) + create.add_argument("--initial-deployment", action="store_true", help="allow absent named volumes only for a new host containing only the preloaded corpus") verify = subparsers.add_parser("verify", help="verify archive checksum and contents") verify.add_argument("archive", type=Path) restore = subparsers.add_parser("restore", help="extract into a new staging directory") @@ -360,7 +373,9 @@ def main() -> int: if args.command in {"create", "restore"} and os.geteuid() != 0: raise ValueError("create and restore must run as root to protect production secrets and data") if args.command == "create": - sources, revision, images, deployment_files = compose_sources(args.compose_file, args.env_file) + sources, revision, images, deployment_files = compose_sources( + args.compose_file, args.env_file, args.initial_deployment + ) create_backup(args.output, sources, revision, images, deployment_files) elif args.command == "verify": manifest = verify_archive(args.archive) diff --git a/infra/backup/test_archive.py b/infra/backup/test_archive.py index 388a693..837c698 100644 --- a/infra/backup/test_archive.py +++ b/infra/backup/test_archive.py @@ -238,12 +238,43 @@ class BackupArchiveTests(unittest.TestCase): with self.assertRaisesRegex(ValueError, "not all configured Compose services"): compose_sources(Path.cwd() / "deploy/production/compose.yaml", env_file) - def test_existing_data_with_uncreated_named_volume_is_backed_up_as_empty(self): + def test_missing_named_volume_with_existing_data_requires_explicit_initial_mode(self): config, volume_paths, env_file, _secret = self.compose_fixture() (self.root / "data/search-users.sqlite3").unlink() (self.root / "data/existing.json").write_text("data", encoding="utf-8") available = set(volume_paths) - {"akyldash_os-data"} + def run(command, **_kwargs): + if "config" in command: + return SimpleNamespace(stdout=json.dumps(config)) + if "ps" in command: + return SimpleNamespace(stdout="") + if command[1] == "volume" and "ls" in command: + return SimpleNamespace(stdout="\n".join(available) + "\n") + if command[1] == "volume": + return SimpleNamespace(stdout=f"{volume_paths[command[-1]]}\n") + if "status" in command: + return SimpleNamespace(stdout="") + if "rev-parse" in command: + return SimpleNamespace(stdout="abc123\n") + raise AssertionError(command) + + with patch("archive.subprocess.run", side_effect=run): + with self.assertRaisesRegex(ValueError, "use --initial-deployment"): + compose_sources(Path.cwd() / "deploy/production/compose.yaml", env_file) + with self.assertRaisesRegex(ValueError, "contain only minjust-normalized"): + compose_sources( + Path.cwd() / "deploy/production/compose.yaml", env_file, initial_deployment=True + ) + + def test_initial_mode_backs_up_only_preloaded_corpus_when_volumes_are_absent(self): + config, volume_paths, env_file, _secret = self.compose_fixture() + (self.root / "data/search-users.sqlite3").unlink() + corpus = self.root / "data/minjust-normalized" + corpus.mkdir() + (corpus / "document.json").write_text("corpus", encoding="utf-8") + available = set(volume_paths) - {"akyldash_os-data"} + def run(command, **_kwargs): if "config" in command: return SimpleNamespace(stdout=json.dumps(config)) @@ -261,15 +292,16 @@ class BackupArchiveTests(unittest.TestCase): with patch("archive.subprocess.run", side_effect=run): sources, revision, images, deployment_files = compose_sources( - Path.cwd() / "deploy/production/compose.yaml", env_file + Path.cwd() / "deploy/production/compose.yaml", env_file, initial_deployment=True ) self.assertIsNone(sources["opensearch"]) - archive = self.private / "bind-only-backup.tar.gz" + archive = self.private / "initial-corpus-backup.tar.gz" create_backup(archive, sources, revision, images, deployment_files) manifest = verify_archive(archive) self.assertEqual(manifest["git_revision"], "abc123") with tarfile.open(archive, "r:gz") as saved: self.assertTrue(saved.getmember("opensearch/").isdir()) + self.assertTrue(saved.getmember("data/minjust-normalized/document.json").isfile()) def test_compose_backup_records_sources_and_image_ids(self): data = self.root / "production-data"