swares · swares · Jul 22, 2026 · Jul 22, 2026
diff --git a/docs/GITOPS-STUDY-GUIDE.md b/docs/GITOPS-STUDY-GUIDE.md
@@ -337,10 +337,12 @@ The lab runs two storage tiers with independent failure domains for data safety.
 
 **Backup timers (systemd, never disabled):**
 
-| Timer | What it backs up | Destination |
-|---|---|---|
-| `backup-nas` | NAS data on NVMe | Cold RAID tier |
-| `backup-etcd` | k3s etcd snapshot | Cold RAID tier |
+| Timer | When | What it backs up | Destination |
+|---|---|---|---|
+| `backup-nas` | 01:30 | NAS data (`/srv/nas`, VMs, Immich library) | Cold RAID tier (md1 primary → md0 copy) |
+| `backup-etcd` | daily | k3s etcd snapshots | Cold RAID tier |
+| `backup-vault` | 02:30 | Vault raft snapshot | Cold RAID tier |
+| `backup-cloud` | 03:00 | **Offsite.** etcd snapshots + lldap + Vault snapshots + Postgres dumps (Authelia, Immich, Semaphore) | Cloudflare R2 (free 10 GB tier) |
 
 Before any storage change on the hot tier, the last backup timestamp is confirmed. This
 is the one manual gate in an otherwise automated system — it protects against the scenario

diff --git a/docs/OPS.md b/docs/OPS.md
@@ -542,11 +542,18 @@ restic check              # verify repo integrity
 ### Backup timer status
 
 ```bash
-systemctl status backup-nas.timer
-systemctl status backup-etcd.timer
+systemctl status backup-nas.timer backup-etcd.timer backup-cloud.timer
 
-# When did it last run?
+# When did each last run?
 journalctl -u backup-nas.service --since "7 days ago" | tail -30
+journalctl -u backup-cloud.service --since "7 days ago" | tail -20
+```
+
+### Check cloud (offsite) backup
+
+```bash
+# Verify snapshots exist in Cloudflare R2
+sudo bash -c 'set -a; source /etc/restic/cloud.env; set +a; restic snapshots'
 ```
 
 ### Confirm last backup succeeded before any storage operation

diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md
@@ -118,22 +118,42 @@ Never `kubectl apply` directly against main — it drifts and ArgoCD reverts it.
 
 | Stream | When | What | Downtime |
 |--------|------|------|----------|
-| `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec + offsite | none |
+| `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec | none |
 | `backup-etcd` | daily | k3s SQLite state → `/mnt/cold-8t/k3s-etcd-snapshots/`, 7 copies retained | none |
 | `backup-vault` | daily 02:30 | Vault raft snapshot → `/mnt/cold-8t/vault-snapshots/`, 30-day retention | none |
-| `backup-lldap` | — | **ldap-1 VM decommissioned 2026-07-04.** lldap now runs as a k3s Deployment in the `lldap` namespace; SQLite data is on a `local-path` PVC. To back it up, add a k8s CronJob that copies the SQLite file from the PVC mount. | — |
+| `backup-cloud` | daily 03:00 | **Offsite (Track 1).** restic → Cloudflare R2 `homelab-backup` bucket. Sources: etcd snapshots + lldap snapshots + Vault snapshots + Postgres dumps (Authelia, Immich, Semaphore via `kubectl exec`). Credentials in Vault at `secret/lab/cloudflare-r2`. Playbook: `ansible/playbooks/backup-cloud.yml`. | none |
 | Immich DB dump | daily 01:30 | `pg_dump` via k8s CronJob → `/mnt/cold-8t/immich/backups/` (captured by restic above) | none |
 
-Check:
+Check cold-tier backups:
 
     export VAULT_ADDR=http://192.168.1.128:8200
     export RESTIC_PASSWORD=$(vault kv get -field=password secret/lab/restic)
     export RESTIC_REPOSITORY=/mnt/cold-8t/restic
     restic snapshots
     journalctl -u backup-nas.service --no-pager | tail -30
 
+Check cloud backup:
+
+    sudo bash -c 'set -a; source /etc/restic/cloud.env; set +a; restic snapshots'
+    journalctl -u backup-cloud.service --no-pager | tail -20
+
 **Never run `restic forget` or `restic prune` manually** — retention is managed by timers only.
 
+### Track 2 — offsite bulk photo/video backup (deferred)
+
+The ~1.5 TB photo/video library (`/mnt/cold-8t/immich` + `/srv/nas` originals) is currently
+protected only by the two on-prem cold RAID mirrors. Offsite backup of this data is **Track 2**
+and has not been implemented.
+
+When ready, options:
+- **Backblaze B2** — cheapest paid tier (~$0.006/GB/month, ~$9/month for 1.5 TB). Restic
+  speaks B2 natively; add a third `ExecStartPost` to `backup-nas.service` pointing to a B2 repo.
+- **Cloudflare R2 paid** — $0.015/GB/month after free 10 GB; zero egress fees.
+
+Decision gate: confirm total photo/video volume with `du -sh /mnt/cold-8t/immich /srv/nas`
+before committing to a provider. The existing `backup-offsite.service` template is already
+wired — set `offsite_restic_repo` in group_vars and provide `/etc/restic/offsite.env`.
+
 ---
 
 ## Vault operations
@@ -276,8 +296,9 @@ Until it passes all three, keep backups landing on the secondary — don't promo
   set (DB dumps, configs, irreplaceable originals).
 - **Hot — NVMe 4 TB.** Live NAS (`lv_nas`) + k8s PVs (`local-path` StorageClass).
 
-Because the secondary can't hold the entire multi-TB library, the bulk library's redundant
-copy is the offsite restic + md1's own mirror. The secondary covers the small critical set.
+Because the secondary can't hold the entire multi-TB library, md1's RAID mirror is the
+only redundant copy of the bulk photo/video data (Track 2 offsite TBD). The secondary
+covers the small critical set; `backup-cloud` covers critical cluster state offsite.
 
 ### Migrate the ex-Synology secondary to a clean mirror
 

diff --git a/docs/STANDUP.md b/docs/STANDUP.md
@@ -171,8 +171,12 @@ Goal: the VM/devops sandbox. This is a **parallel track** — it doesn't block P
    - AI: gateway lists models and answers a prompt.
    - Backups: a restic snapshot exists on md1 (`/mnt/cold-8t`), copied to the secondary.
 2. **Close the gaps** (from [SERVICES.md](SERVICES.md), priority order):
-   - ✅ **Offsite backup** — wired: `backup-offsite.timer` does a nightly `restic copy` to an
-     offsite repo. Set `offsite_restic_repo` + `/etc/restic/offsite.env` (creds from Vault).
+   - ✅ **Offsite backup (Track 1 — cluster state)** — `backup-cloud.timer` runs daily at 03:00,
+     pushing etcd snapshots, lldap/Vault snapshots, and Postgres dumps to Cloudflare R2 free tier.
+     Credentials in Vault at `secret/lab/cloudflare-r2`. Playbook: `ansible/playbooks/backup-cloud.yml`.
+   - 🔵 **Offsite backup (Track 2 — bulk photos/video ~1.5 TB)** — deferred. See RUNBOOK.md for
+     options (Backblaze B2 or R2 paid). The `backup-offsite.service` template is ready; needs
+     `offsite_restic_repo` set and `/etc/restic/offsite.env` populated.
    - ✅ **MQTT (Mosquitto)** — relocated to a lab Zero 2W (`make mqtt`).
    - 🔵 **UPS + NUT** — deferred by choice (future improvement).
    - 🟡 **Reverse proxy + internal CA + SSO** — once several web UIs are live.