Skip to content

fix(admin): allow cold storage worker DB startup - #7770

Merged
ravarora2 merged 2 commits into
mainfrom
codex/storage-worker-startup
Sep 22, 2026
Merged

fix(admin): allow cold storage worker DB startup#7770
ravarora2 merged 2 commits into
mainfrom
codex/storage-worker-startup

Conversation

@ravarora2

@ravarora2 ravarora2 commented Sep 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

The standalone S3 accounting worker can exit before its scan begins because database startup inherits the relay’s three-second connection budget. A cold DNS probe took about five seconds.

Give buzz-admin storage-snapshot its own pool: one connection, no minimum idle connections, and a 30-second acquisition budget. Retry startup timeouts and transport failures at most twice, after two and five seconds. Total startup wait is bounded to 97 seconds. Configuration, authentication, TLS, and protocol errors fail immediately.

Log each startup attempt’s elapsed time and error category without connection credentials. Relay defaults stay unchanged. Retries stop before lock acquisition; the same lock-owning database session still publishes the completed snapshot.

Related issue

Follow-up to #7543. No duplicate startup fix found.

Testing

Archived-command check: built a nextest archive, temporarily moved target/debug/buzz-admin out of the checkout, and ran scripts/postgres-test-run.sh --archive-file <archive>. Both tests failed before the path fix and passed afterward. The tests use nextest’s relocated executable path, with the original Cargo path retained for local runs.

Manually restored the old startup call and ran scripts/postgres-test-run.sh -p buzz-admin --test postgres_storage_snapshot -E 'test(storage_snapshot_retries_then)'. It failed before reaching S3; restoring the fix passed. This confirms the test exercises the actual worker command.

Local just ci passed all stages before mobile tests, which stopped at SDK discovery in the objective_c native hook. The default Xcode tools report an unaccepted license. Two existing PostgreSQL tests hit connection timeouts in the parallel run and passed individually.

After merge, build a worker image and repeat the cold-pod staging smoke test through snapshot publication before changing relay metrics mode.

Generated with Codex

@ravarora2
fix(admin): allow cold storage worker DB startup
0c18585 
Signed-off-by: Ravneet Arora <rarora@squareup.com>
@github-actions

github-actions Bot commented Sep 20, 2026 •
edited
Loading

Copy link
Copy Markdown

🔐 Codex Security Review

Status: review required for the current range.

The current range is 77729abfb692b25a0f4ec4a69add86af2e32c0dd...aa200bb566a4d8a83bf724e9801002d0d8e06062.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review aa200bb566a4d8a83bf724e9801002d0d8e06062 to authorize a new review.
Any previous review applies only to its recorded range.

@ravarora2
test(admin): resolve archived worker executable
aa200bb 
Signed-off-by: Ravneet Arora <rarora@squareup.com>
@ravarora2
ravarora2 marked this pull request as ready for review September 22, 2026 15:41
@ravarora2
ravarora2 requested a review from a team as a code owner September 22, 2026 15:41
@ravarora2

ravarora2 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

@buzz-security-review aa200bb

@ravarora2
ravarora2 merged commit a929532 into main Sep 22, 2026
147 of 150 checks passed
@ravarora2
ravarora2 deleted the codex/storage-worker-startup branch September 22, 2026 17:32
wpfleger96 pushed a commit that referenced this pull request Sep 22, 2026
Duncan
Merge remote-tracking branch 'origin/main' into duncan/canvas-version…
ac21a64 
…-history

* origin/main:
  fix(admin): allow cold storage worker DB startup (#7770)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
wpfleger96 pushed a commit that referenced this pull request Sep 22, 2026
Duncan
Merge remote-tracking branch 'origin/main' into wpfleger/deterministi…
f3581d2 
…c-agent-commit-identity

* origin/main:
  fix(desktop): bound startup request bursts and recover quota refusals (#7790)
  fix(audit): frame hash inputs with TLV (#7492)
  fix(admin): allow cold storage worker DB startup (#7770)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
brow added a commit that referenced this pull request Sep 22, 2026
@brow
Merge remote-tracking branch 'origin/main' into personal-railway-push…
66d9340 
…-dev

* origin/main:
  fix(desktop): bound startup request bursts and recover quota refusals (#7790)
  fix(audit): frame hash inputs with TLV (#7492)
  fix(admin): allow cold storage worker DB startup (#7770)
  feat(relay): add admin HTTP routes for member restriction management (#7302)
  fix(relay): fire kick live side effects at convergence; persist target; fence re-add race with held lock (#7298)
  feat(relay): add atomic complete read-state snapshots (#7572)
  fix(desktop): register macOS badges for new and existing installs (#7783)
  fix(mobile): avoid opening empty threads on message tap (#7756)
  fix(workflows): make deletion persistent and retryable (#7735)
  fix(mobile): preserve thread replies through refresh failures (#7757)
  fix(mobile): keep iOS message menu actions responsive after rebuilds (#7758)
  fix(relay): exclude ephemeral activity from message quota (#7736)
  release: push gateway chart 0.3.1 (#7749)
  fix(push): label plaintext push gateway service as HTTP (#7717)
  Replace personal and internal data in desktop test fixtures (#7748)
  Add mobile VISION (#7710)
  fix(mobile): keep relay sessions stable during push lease updates (#7745)

Signed-off-by: Tom Brow <tomb@block.xyz>
wpfleger96 pushed a commit that referenced this pull request Sep 22, 2026
Duncan
Merge remote-tracking branch 'origin/main' into wpfleger/desktop-admi…
8ed8a4a 
…n-surface

* origin/main:
  docs(nip-fi): clarify federated identity amendments (#7803)
  fix(desktop): refresh channels after access-revoked closure (#7784)
  fix(desktop): bound startup request bursts and recover quota refusals (#7790)
  fix(audit): frame hash inputs with TLV (#7492)
  fix(admin): allow cold storage worker DB startup (#7770)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
wpfleger96 pushed a commit that referenced this pull request Sep 22, 2026
Duncan
Merge remote-tracking branch 'origin/main' into duncan/nip-fi-blossom…
cb7b891 
…-hardening

* origin/main:
  docs(nip-fi): clarify federated identity amendments (#7803)
  fix(desktop): refresh channels after access-revoked closure (#7784)
  fix(desktop): bound startup request bursts and recover quota refusals (#7790)
  fix(audit): frame hash inputs with TLV (#7492)
  fix(admin): allow cold storage worker DB startup (#7770)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
wpfleger96 added a commit that referenced this pull request Sep 22, 2026
Hayt @wpfleger96
feat(nip-fi): S4 deny-until-TTL API — full implementation + witnesses
4f7c8c1 
NIP-FI admin disconnect/deny API implementation, rebased onto
origin/main (c213d90) after main advanced past the original merge-base
(77729ab). No buzz-relay production code changed by main's 3 new
commits (#7770, #7492, #7790) — rebase is conflict-free.

Squash of all commits from 2a42ddf through 360150d97:

Production:
- Deny-map JWT verifier and command installer (S4)
- Disconnect endpoint and route wiring
- Admin HTTP routes for deny/undeny, expiry TTL
- WS connection admission: deny-map check wired in
- Audio pre-writer send bounding (1s timeout policy)
- nip_fi_session.rs: bounded pre-registration pairing sends
- handler.rs MINOR: Ok(Ok(())) success check replaces is_err()

Witnesses (Thufir round-5 required corrections):
- R2: drain_terminal!() race witness (before_not_a_member_drain_terminal
  hook + cancel_race_test_hook; falsification: remove lifecycle_cancel()
  → frame-1 timeout RED)
- F3: real admission → 48101 JOIN wire bytes → handle_huddle_liveness_req
  oracle → authoritative generation == wire generation
- F4: valid future deadline + cancel-while-held (not already-expired);
  falsification: already-expired → hook timeout RED
- F5: {"kinds":[13534]} REQ skips DB lookup, reaches after_req_permit_acquired;
  AUTH OK(true) check; real Postgres+Redis pools; canonical denial frame +
  1008 close; zero global+channel refcounts asserted
- F6: ENV_TEST_MUTEX synchronization for Config::from_env()
- F7: 10s bounded consumer threads; entry-hook Drop guards
- R1: reason-published/before-enqueue witness
- F1/F2: LifecycleClosed transition primitives

Evidence inventory corrections per Thufir's requirement:
root raw cancels follow pairing_deny_terminal/auth_deny_terminal;
audio pre-registration pairing sends direct denial/close;
post-registration deny-set/expired paths send directly then raw-cancel;
wrapper cancellation exists before AND after run(control).await.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96 added a commit that referenced this pull request Sep 22, 2026
Hayt @wpfleger96
feat(nip-fi): S4 deny-until-TTL API — full implementation + witnesses
cc5aa3b 
NIP-FI admin disconnect/deny API implementation, rebased onto
origin/main (c213d90) after main advanced past the original merge-base
(77729ab). No buzz-relay production code changed by main's 3 new
commits (#7770, #7492, #7790) — rebase is conflict-free.

Squash of all commits from 2a42ddf through 360150d97:

Production:
- Deny-map JWT verifier and command installer (S4)
- Disconnect endpoint and route wiring
- Admin HTTP routes for deny/undeny, expiry TTL
- WS connection admission: deny-map check wired in
- Audio pre-writer send bounding (1s timeout policy)
- nip_fi_session.rs: bounded pre-registration pairing sends
- handler.rs MINOR: Ok(Ok(())) success check replaces is_err()

Witnesses (Thufir round-5 required corrections):
- R2: drain_terminal!() race witness (before_not_a_member_drain_terminal
  hook + cancel_race_test_hook; falsification: remove lifecycle_cancel()
  → frame-1 timeout RED)
- F3: real admission → 48101 JOIN wire bytes → handle_huddle_liveness_req
  oracle → authoritative generation == wire generation
- F4: valid future deadline + cancel-while-held (not already-expired);
  falsification: already-expired → hook timeout RED
- F5: {"kinds":[13534]} REQ skips DB lookup, reaches after_req_permit_acquired;
  AUTH OK(true) check; real Postgres+Redis pools; canonical denial frame +
  1008 close; zero global+channel refcounts asserted
- F6: ENV_TEST_MUTEX synchronization for Config::from_env()
- F7: 10s bounded consumer threads; entry-hook Drop guards
- R1: reason-published/before-enqueue witness
- F1/F2: LifecycleClosed transition primitives

Evidence inventory corrections per Thufir's requirement:
root raw cancels follow pairing_deny_terminal/auth_deny_terminal;
audio pre-registration pairing sends direct denial/close;
post-registration deny-set/expired paths send directly then raw-cancel;
wrapper cancellation exists before AND after run(control).await.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96 added a commit that referenced this pull request Sep 22, 2026
Hayt @wpfleger96
feat(nip-fi): S4 deny-until-TTL API — full implementation + witnesses
7dbb43c 
NIP-FI admin disconnect/deny API implementation, rebased onto
origin/main (c213d90) after main advanced past the original merge-base
(77729ab). No buzz-relay production code changed by main's 3 new
commits (#7770, #7492, #7790) — rebase is conflict-free.

Squash of all commits from 2a42ddf through 360150d97:

Production:
- Deny-map JWT verifier and command installer (S4)
- Disconnect endpoint and route wiring
- Admin HTTP routes for deny/undeny, expiry TTL
- WS connection admission: deny-map check wired in
- Audio pre-writer send bounding (1s timeout policy)
- nip_fi_session.rs: bounded pre-registration pairing sends
- handler.rs MINOR: Ok(Ok(())) success check replaces is_err()

Witnesses (Thufir round-5 required corrections):
- R2: drain_terminal!() race witness (before_not_a_member_drain_terminal
  hook + cancel_race_test_hook; falsification: remove lifecycle_cancel()
  → frame-1 timeout RED)
- F3: real admission → 48101 JOIN wire bytes → handle_huddle_liveness_req
  oracle → authoritative generation == wire generation
- F4: valid future deadline + cancel-while-held (not already-expired);
  falsification: already-expired → hook timeout RED
- F5: {"kinds":[13534]} REQ skips DB lookup, reaches after_req_permit_acquired;
  AUTH OK(true) check; real Postgres+Redis pools; canonical denial frame +
  1008 close; zero global+channel refcounts asserted
- F6: ENV_TEST_MUTEX synchronization for Config::from_env()
- F7: 10s bounded consumer threads; entry-hook Drop guards
- R1: reason-published/before-enqueue witness
- F1/F2: LifecycleClosed transition primitives

Evidence inventory corrections per Thufir's requirement:
root raw cancels follow pairing_deny_terminal/auth_deny_terminal;
audio pre-registration pairing sends direct denial/close;
post-registration deny-set/expired paths send directly then raw-cancel;
wrapper cancellation exists before AND after run(control).await.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96 added a commit that referenced this pull request Sep 23, 2026
Hayt @wpfleger96
feat(nip-fi): S4 deny-until-TTL API — full implementation + witnesses
cea34b1 
NIP-FI admin disconnect/deny API implementation, rebased onto
origin/main (c213d90) after main advanced past the original merge-base
(77729ab). No buzz-relay production code changed by main's 3 new
commits (#7770, #7492, #7790) — rebase is conflict-free.

Squash of all commits from 2a42ddf through 360150d97:

Production:
- Deny-map JWT verifier and command installer (S4)
- Disconnect endpoint and route wiring
- Admin HTTP routes for deny/undeny, expiry TTL
- WS connection admission: deny-map check wired in
- Audio pre-writer send bounding (1s timeout policy)
- nip_fi_session.rs: bounded pre-registration pairing sends
- handler.rs MINOR: Ok(Ok(())) success check replaces is_err()

Witnesses (Thufir round-5 required corrections):
- R2: drain_terminal!() race witness (before_not_a_member_drain_terminal
  hook + cancel_race_test_hook; falsification: remove lifecycle_cancel()
  → frame-1 timeout RED)
- F3: real admission → 48101 JOIN wire bytes → handle_huddle_liveness_req
  oracle → authoritative generation == wire generation
- F4: valid future deadline + cancel-while-held (not already-expired);
  falsification: already-expired → hook timeout RED
- F5: {"kinds":[13534]} REQ skips DB lookup, reaches after_req_permit_acquired;
  AUTH OK(true) check; real Postgres+Redis pools; canonical denial frame +
  1008 close; zero global+channel refcounts asserted
- F6: ENV_TEST_MUTEX synchronization for Config::from_env()
- F7: 10s bounded consumer threads; entry-hook Drop guards
- R1: reason-published/before-enqueue witness
- F1/F2: LifecycleClosed transition primitives

Evidence inventory corrections per Thufir's requirement:
root raw cancels follow pairing_deny_terminal/auth_deny_terminal;
audio pre-registration pairing sends direct denial/close;
post-registration deny-set/expired paths send directly then raw-cancel;
wrapper cancellation exists before AND after run(control).await.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bradseiler bradseiler bradseiler approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ravarora2 @bradseiler