From 369463db36e40824c5d1a31b093f8aa55c9a7b57 Mon Sep 17 00:00:00 2001 From: Scott Wares Date: Wed, 22 Jul 2026 18:43:40 +0000 Subject: [PATCH] docs: reflect Track 1 cloud backup (Cloudflare R2) as implemented - RUNBOOK.md: add backup-cloud row to backup table; add cloud check commands; fix storage tiers paragraph (bulk offsite is Track 2 TBD, not done); remove incorrect 'offsite' from backup-nas description - OPS.md: add backup-cloud.timer to status check; add cloud restic snapshot verification command - GITOPS-STUDY-GUIDE.md: expand backup timer table to include backup-vault and backup-cloud rows with destinations --- docs/GITOPS-STUDY-GUIDE.md | 10 ++++++---- docs/OPS.md | 13 ++++++++++--- docs/RUNBOOK.md | 16 +++++++++++----- 3 files changed, 27 insertions(+), 12 deletions(-) diff --git a/docs/GITOPS-STUDY-GUIDE.md b/docs/GITOPS-STUDY-GUIDE.md index 2d0391c..8011e02 100644 --- a/docs/GITOPS-STUDY-GUIDE.md +++ b/docs/GITOPS-STUDY-GUIDE.md @@ -337,10 +337,12 @@ The lab runs two storage tiers with independent failure domains for data safety. **Backup timers (systemd, never disabled):** -| Timer | What it backs up | Destination | -|---|---|---| -| `backup-nas` | NAS data on NVMe | Cold RAID tier | -| `backup-etcd` | k3s etcd snapshot | Cold RAID tier | +| Timer | When | What it backs up | Destination | +|---|---|---|---| +| `backup-nas` | 01:30 | NAS data (`/srv/nas`, VMs, Immich library) | Cold RAID tier (md1 primary → md0 copy) | +| `backup-etcd` | daily | k3s etcd snapshots | Cold RAID tier | +| `backup-vault` | 02:30 | Vault raft snapshot | Cold RAID tier | +| `backup-cloud` | 03:00 | **Offsite.** etcd snapshots + lldap + Vault snapshots + Postgres dumps (Authelia, Immich, Semaphore) | Cloudflare R2 (free 10 GB tier) | Before any storage change on the hot tier, the last backup timestamp is confirmed. This is the one manual gate in an otherwise automated system — it protects against the scenario diff --git a/docs/OPS.md b/docs/OPS.md index 538b4ed..2bef3a1 100644 --- a/docs/OPS.md +++ b/docs/OPS.md @@ -542,11 +542,18 @@ restic check # verify repo integrity ### Backup timer status ```bash -systemctl status backup-nas.timer -systemctl status backup-etcd.timer +systemctl status backup-nas.timer backup-etcd.timer backup-cloud.timer -# When did it last run? +# When did each last run? journalctl -u backup-nas.service --since "7 days ago" | tail -30 +journalctl -u backup-cloud.service --since "7 days ago" | tail -20 +``` + +### Check cloud (offsite) backup + +```bash +# Verify snapshots exist in Cloudflare R2 +sudo bash -c 'set -a; source /etc/restic/cloud.env; set +a; restic snapshots' ``` ### Confirm last backup succeeded before any storage operation diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 3323ba8..4f7d2c1 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -118,13 +118,13 @@ Never `kubectl apply` directly against main — it drifts and ArgoCD reverts it. | Stream | When | What | Downtime | |--------|------|------|----------| -| `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec + offsite | none | +| `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec | none | | `backup-etcd` | daily | k3s SQLite state → `/mnt/cold-8t/k3s-etcd-snapshots/`, 7 copies retained | none | | `backup-vault` | daily 02:30 | Vault raft snapshot → `/mnt/cold-8t/vault-snapshots/`, 30-day retention | none | -| `backup-lldap` | — | **ldap-1 VM decommissioned 2026-07-04.** lldap now runs as a k3s Deployment in the `lldap` namespace; SQLite data is on a `local-path` PVC. To back it up, add a k8s CronJob that copies the SQLite file from the PVC mount. | — | +| `backup-cloud` | daily 03:00 | **Offsite (Track 1).** restic → Cloudflare R2 `homelab-backup` bucket. Sources: etcd snapshots + lldap snapshots + Vault snapshots + Postgres dumps (Authelia, Immich, Semaphore via `kubectl exec`). Credentials in Vault at `secret/lab/cloudflare-r2`. Playbook: `ansible/playbooks/backup-cloud.yml`. | none | | Immich DB dump | daily 01:30 | `pg_dump` via k8s CronJob → `/mnt/cold-8t/immich/backups/` (captured by restic above) | none | -Check: +Check cold-tier backups: export VAULT_ADDR=http://192.168.1.128:8200 export RESTIC_PASSWORD=$(vault kv get -field=password secret/lab/restic) @@ -132,6 +132,11 @@ Check: restic snapshots journalctl -u backup-nas.service --no-pager | tail -30 +Check cloud backup: + + sudo bash -c 'set -a; source /etc/restic/cloud.env; set +a; restic snapshots' + journalctl -u backup-cloud.service --no-pager | tail -20 + **Never run `restic forget` or `restic prune` manually** — retention is managed by timers only. --- @@ -276,8 +281,9 @@ Until it passes all three, keep backups landing on the secondary — don't promo set (DB dumps, configs, irreplaceable originals). - **Hot — NVMe 4 TB.** Live NAS (`lv_nas`) + k8s PVs (`local-path` StorageClass). -Because the secondary can't hold the entire multi-TB library, the bulk library's redundant -copy is the offsite restic + md1's own mirror. The secondary covers the small critical set. +Because the secondary can't hold the entire multi-TB library, md1's RAID mirror is the +only redundant copy of the bulk photo/video data (Track 2 offsite TBD). The secondary +covers the small critical set; `backup-cloud` covers critical cluster state offsite. ### Migrate the ex-Synology secondary to a clean mirror