Skip to content

docs: reflect Track 1 cloud backup as implemented #276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swares merged 1 commit into from
Jul 22, 2026
Merged

docs: reflect Track 1 cloud backup as implemented #276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions docs/GITOPS-STUDY-GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,10 +337,12 @@ The lab runs two storage tiers with independent failure domains for data safety.

**Backup timers (systemd, never disabled):**

| Timer | What it backs up | Destination |
|---|---|---|
| `backup-nas` | NAS data on NVMe | Cold RAID tier |
| `backup-etcd` | k3s etcd snapshot | Cold RAID tier |
| Timer | When | What it backs up | Destination |
|---|---|---|---|
| `backup-nas` | 01:30 | NAS data (`/srv/nas`, VMs, Immich library) | Cold RAID tier (md1 primary → md0 copy) |
| `backup-etcd` | daily | k3s etcd snapshots | Cold RAID tier |
| `backup-vault` | 02:30 | Vault raft snapshot | Cold RAID tier |
| `backup-cloud` | 03:00 | **Offsite.** etcd snapshots + lldap + Vault snapshots + Postgres dumps (Authelia, Immich, Semaphore) | Cloudflare R2 (free 10 GB tier) |

Before any storage change on the hot tier, the last backup timestamp is confirmed. This
is the one manual gate in an otherwise automated system — it protects against the scenario
Expand Down
13 changes: 10 additions & 3 deletions docs/OPS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -542,11 +542,18 @@ restic check # verify repo integrity
### Backup timer status

```bash
systemctl status backup-nas.timer
systemctl status backup-etcd.timer
systemctl status backup-nas.timer backup-etcd.timer backup-cloud.timer

# When did it last run?
# When did each last run?
journalctl -u backup-nas.service --since "7 days ago" | tail -30
journalctl -u backup-cloud.service --since "7 days ago" | tail -20
```

### Check cloud (offsite) backup

```bash
# Verify snapshots exist in Cloudflare R2
sudo bash -c 'set -a; source /etc/restic/cloud.env; set +a; restic snapshots'
```

### Confirm last backup succeeded before any storage operation
Expand Down
16 changes: 11 additions & 5 deletions docs/RUNBOOK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,20 +118,25 @@ Never `kubectl apply` directly against main — it drifts and ArgoCD reverts it.

| Stream | When | What | Downtime |
|--------|------|------|----------|
| `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec + offsite | none |
| `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec | none |
| `backup-etcd` | daily | k3s SQLite state → `/mnt/cold-8t/k3s-etcd-snapshots/`, 7 copies retained | none |
| `backup-vault` | daily 02:30 | Vault raft snapshot → `/mnt/cold-8t/vault-snapshots/`, 30-day retention | none |
| `backup-lldap` | — | **ldap-1 VM decommissioned 2026-07-04.** lldap now runs as a k3s Deployment in the `lldap` namespace; SQLite data is on a `local-path` PVC. To back it up, add a k8s CronJob that copies the SQLite file from the PVC mount. | — |
| `backup-cloud` | daily 03:00 | **Offsite (Track 1).** restic → Cloudflare R2 `homelab-backup` bucket. Sources: etcd snapshots + lldap snapshots + Vault snapshots + Postgres dumps (Authelia, Immich, Semaphore via `kubectl exec`). Credentials in Vault at `secret/lab/cloudflare-r2`. Playbook: `ansible/playbooks/backup-cloud.yml`. | none |
| Immich DB dump | daily 01:30 | `pg_dump` via k8s CronJob → `/mnt/cold-8t/immich/backups/` (captured by restic above) | none |

Check:
Check cold-tier backups:

export VAULT_ADDR=http://192.168.1.128:8200
export RESTIC_PASSWORD=$(vault kv get -field=password secret/lab/restic)
export RESTIC_REPOSITORY=/mnt/cold-8t/restic
restic snapshots
journalctl -u backup-nas.service --no-pager | tail -30

Check cloud backup:

sudo bash -c 'set -a; source /etc/restic/cloud.env; set +a; restic snapshots'
journalctl -u backup-cloud.service --no-pager | tail -20

**Never run `restic forget` or `restic prune` manually** — retention is managed by timers only.

---
Expand Down Expand Up @@ -276,8 +281,9 @@ Until it passes all three, keep backups landing on the secondary — don't promo
set (DB dumps, configs, irreplaceable originals).
- **Hot — NVMe 4 TB.** Live NAS (`lv_nas`) + k8s PVs (`local-path` StorageClass).

Because the secondary can't hold the entire multi-TB library, the bulk library's redundant
copy is the offsite restic + md1's own mirror. The secondary covers the small critical set.
Because the secondary can't hold the entire multi-TB library, md1's RAID mirror is the
only redundant copy of the bulk photo/video data (Track 2 offsite TBD). The secondary
covers the small critical set; `backup-cloud` covers critical cluster state offsite.

### Migrate the ex-Synology secondary to a clean mirror

Expand Down