obs: port worker observability + buildinfo from api repo (B1) - #8
Merged
Merged
Conversation
Relocates the observability code that was merged into InstaNode-dev/api under api/worker/ (PR #37 there) into its real home in this repo. What ships: - internal/jobs/middleware.go — WithObservability[T] generic River-Worker wrapper that stamps tid/trace_id on ctx and (optionally) opens a New Relic transaction per job. Fail-open on nil nrApp. - internal/jobs/middleware_test.go — 7 tests covering tid stamping, trace_id missing/present, error propagation, nil-NR safety, delegation of NextRetry/Timeout, plus the int64 formatter. - internal/obs/nr.go — InitNewRelic + WaitForConnection helpers. - internal/obs/nr_test.go — 2 tests asserting fail-open contract. - main.go — slog wrapped in logctx.NewHandler, NR init, /healthz now emits commit_id/build_time/version, workers receive nrApp. - internal/jobs/workers.go — StartWorkers gains nrApp parameter; every river.AddWorker call wraps the worker via WithObservability(...). Critical detail: the api/worker/ PR shipped against TEMPORARY stubs at instant.dev/worker/internal/_obs_stubs/{buildinfo,logctx}. This relocate switches both imports to the canonical common packages: - instant.dev/worker/internal/_obs_stubs/buildinfo -> instant.dev/common/buildinfo - instant.dev/worker/internal/_obs_stubs/logctx -> instant.dev/common/logctx That mirrors today's PR #40 fix on the api repo, where the same stub->common substitution was applied. The worker module's existing `replace instant.dev/common => ../common` directive in go.mod makes the canonical import resolve to the sibling checkout. go.mod gains: - github.com/newrelic/go-agent/v3 (direct) Tests: go test ./... -count=1 — all green, 34 PASS in internal/jobs + internal/obs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
go.mod ended up at go 1.25 after go mod tidy resolved newrelic / k8s client-go transitive bounds. Matches the api repo Dockerfile. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mastermanas805
added a commit
that referenced
this pull request
Jun 2, 2026
- orphan_sweep PASS 4: young-namespace (within grace) + age-lookup-error are NOT reaped (#8 grace branches). - emitDeployFailedAudit: dedup-hit skips the INSERT (#15 idempotency branch). - dbGracePeriodOpener.TerminateActiveGracePeriod: UPDATE success + error-wrap. - billing terminal downgrade still succeeds when the grace-close errors (#5 fail-open warn branch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mastermanas805
added a commit
that referenced
this pull request
Jun 2, 2026
…dup, cursor (#78) * fix(worker): bug-bash batch 2 — grace close, namespace reaper grace, autopsy dedup, cursor Four confirmed bugs from the 2026-06-02 platform bug bash: - #5 (P1) billing_reconciler: a terminal Razorpay status downgraded the team but left the active payment_grace_periods row open, so payment_grace_reminder emitted dunning emails forever and the terminator later re-acted on an already-cancelled subscription. Add TerminateActiveGracePeriod to the gracePeriodOpener interface (status→'terminated', terminated_at=now()) and call it in the terminal-downgrade branch (fail-open). - #8 (P1) orphan_sweep PASS 4: the customer-namespace reaper excluded 'pending' resources from the live-token set AND had no creation-grace, so a sweep during two-phase provisioning could DELETE a live, mid-provision namespace. Add 'pending' to fetchLiveResourceTokens and a namespace-age grace check (skip if younger than orphanNoDBRowGrace) mirroring PASS 3. - #15 (P2) deploy_failure_autopsy: emitDeployFailedAudit inserted a new deploy.failed audit row (new id) on every reconciler retry, and the forwarder dedups by audit_id (not deployment) → duplicate failure emails. Make it idempotent: skip the INSERT when a deploy.failed row already exists for the deployment (metadata->>'deploy_id'). Fail-open on probe error. - #18 (P2) billing_reconciler scanChargeUndeliverable: jumping the cursor to now() on an empty window skipped rows that became visible a moment later (clock skew / late commit). Leave the cursor unchanged on count==0 — the 1h look-back re-applies and re-scanning the small indexed window is cheap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(worker): cover bug-bash batch-2 changed lines (100% patch gate) - orphan_sweep PASS 4: young-namespace (within grace) + age-lookup-error are NOT reaped (#8 grace branches). - emitDeployFailedAudit: dedup-hit skips the INSERT (#15 idempotency branch). - dbGracePeriodOpener.TerminateActiveGracePeriod: UPDATE success + error-wrap. - billing terminal downgrade still succeeds when the grace-close errors (#5 fail-open warn branch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Relocates the worker observability code that was merged into the api repo
under
api/worker/(PR #37 there) into its real home in this repo. Switchesfrom the temporary
_obs_stubs/{buildinfo,logctx}packages to the canonicalinstant.dev/common/{buildinfo,logctx}packages.Why
The api repo PR #37 shipped the observability scaffolding for the worker
but in the wrong git history because the orchestrator created the worktree
against
InstaNode-dev/apirather thanInstaNode-dev/worker. Productioninstant-workerk8s Deployment runs an image built from THIS repo, so theobs code never reached the running worker.
Today's PR #40 on the api repo also did the
_obs_stubs -> commonswitch.This PR mirrors that fix for the worker.
What ships
internal/jobs/middleware.go--WithObservability[T]generic River-Workerwrapper. Stamps
tid/trace_idon ctx vialogctx, opens optional NewRelic transaction, logs duration on completion. Fail-open on nil nrApp.
internal/jobs/middleware_test.go-- 7 tests + 1 subtest (8 total):tid stamping, trace_id missing/preserved, error propagation, nil-NR safe
(success + failure), delegation of NextRetry/Timeout, int64 formatter.
internal/obs/nr.go--InitNewRelicreturning(nil, nil)on missingNEW_RELIC_LICENSE_KEY. Never crashes.internal/obs/nr_test.go-- 2 tests (fail-open, nil-safe).main.go-- slog wrapped inlogctx.NewHandler("worker", base); NRinit + Shutdown defer;
/healthznow returnscommit_id/build_time/
versionfrombuildinfo;StartWorkersreceivesnrApp.internal/jobs/workers.go--StartWorkerssignature gainsnrApp *newrelic.Application. Everyriver.AddWorker(...)wraps theworker via
WithObservability(...).Dockerfile-- bumped togolang:1.25-alpineto match go.mod.Imports refactor (stubs -> common)
The
replace instant.dev/common => ../commondirective already in go.modmakes the canonical imports resolve to the sibling checkout.
Note:
instant.dev/common/logctx.NewHandlertakes(service, base)not(service, commit_id, base). The canonical handler reads commit_id fromCOMMIT_IDenv (fallback "dev");buildinfo.GitSHAis read separately forthe
/healthzpayload. To make the log fields agree with /healthz, the k8sDeployment should set
COMMIT_ID=$GIT_SHAenv var (not done in this PR).Tests
34 PASS, 0 FAIL.
Live deploy verification
Image:
ghcr.io/mastermanas805/instant-worker:v1.5.0-obs-20ee825(linux/amd64, 36.9 MB).commit_id=20ee825matches the head SHA -- ldflag injection works end-to-end.Container name in Deployment spec:
worker.Test plan
Pushback
Cluster had a transient DNS flake during the first rollout attempt
(
postgres-platform.instant.svc.cluster.localfailed to resolve, thenresolved on retry). Not caused by this PR -- pre-existing condition.
instant.dev/common/logctx.NewHandlerignoresbuildinfo.GitSHAandreads commit_id from the
COMMIT_IDenv var. Until the k8s manifestsets
COMMIT_ID=$GIT_SHAper deploy, log lines showcommit_id=devwhile /healthz shows the real SHA. Fix: either update the worker k8s
Deployment env, or change
common/logctxto fall back tobuildinfo.GitSHA-- the latter is the cleaner long-term fix andbelongs in a
common-repo PR.go mod tidyagainst the local Go 1.26 toolchain bumpedgo 1.24.0to
go 1.25. Bumped Dockerfile togolang:1.25-alpineto match;verified the build still produces a working binary.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
(Generated with Claude Code)