Skip to content

Report org and app IDs on command stats from token scope - #5134

Merged
clouvet merged 1 commit into
masterfrom
command-stats-org-id
Aug 28, 2026
Merged

Report org and app IDs on command stats from token scope#5134
clouvet merged 1 commit into
masterfrom
command-stats-org-id

Conversation

@clouvet

@clouvet clouvet commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #5130, which added app_id/org_id to the command/stats payload but never populated them.

Approach: derive the IDs from token scope, not from an app lookup

Our tokens are already scoped to an org, and sometimes to an app. flyio.OrganizationScope/flyio.AppScope read that straight out of the macaroon caveats, so this costs no API call and works for commands that never fetch an app. Measured at ~5.8µs per invocation.

The alternative — calling GetApp during command setup — looks appealing but inverts the value:

app-scoped command packages
Total (RequireAppName/LoadAppNameIfPresent*) 128 files / 39 packages
Ever call GetApp 5 (apps, deploy, mcp, registry, secrets)

Those 5 overlap heavily with deploy/launch, which already report both IDs on their own payloads. The other 34 — status, logs, ssh, scale, machine, volumes, ips, certificates, config, console, … — only ever deal in an app name, and command/stats is the only event covering them. So a GetApp-based approach would mostly duplicate IDs we already have, add a network round trip to every invocation of the highest-volume event we emit, and introduce a network failure mode into RequireAppName, which today does no I/O at all.

Expected coverage

org_id is reported when every permission macaroon agrees on one org:

  • CI and deploy tokens (single org-scoped macaroon) — reported
  • Users belonging to one org — reported
  • Users in several orgsnot reported. fetchOrgTokens deliberately holds one macaroon per org, so there is no correct single answer without knowing the app; we emit nothing rather than guess and mis-attribute.
  • OAuth-only tokens — not reported; OAuth tokens are per-user and carry no org.

app_id is only reported by tokens narrowed to exactly one app (fly tokens create deploy -a …), so expect it to be empty far more often than org_id. AppScope already returns nil for a wildcard token, so a non-empty result always names specific apps.

Both fields are omitempty, so an invocation we can't attribute sends exactly what it sends today.

Also: removes the context helpers from #5130

WithAppID/WithOrgID/AppIDFromContext/OrgIDFromContext could never have worked. RecordCommandFinish reads the package-global commandContext, which startMetrics snapshots once — 11th of 12 in commonPreparers, before auth and before any command-specific preparer runs. Contexts are immutable, so ctx = metrics.WithAppID(ctx, …) later produces a different context the global never sees. Replaced with a mutex-guarded package-level setter, matching how metrics.IsUsingGPU is already reported from command.go.

⚠️ Companion change still required

flyctl-metrics PR #70 accepts these fields on command/stats but deliberately does not forward that endpoint to Rudderstack, because the IDs weren't populated yet. Landing this without restoring the go sendToRudderstack("flyctl_command_stats", …) block in flyctl-metrics main.go (it's in commit c500ecc) means the IDs still never reach Snowflake. I have not touched that repo.

Test plan

  • go build ./...
  • go vet ./...
  • go test ./internal/config/... ./internal/command/
  • New TestScopedIDs covering single-org, multi-org, OAuth-only, nil, single-app and multi-app tokens
  • Verified end-to-end against a real FlyV1 fm2_… token: org ID extracted, zero network calls

The command stats payload gained app_id/org_id fields in #5130, but
nothing ever populated them. Populate them from the scope our tokens
already carry, which costs no API call and needs no app lookup.

Tokens hold one macaroon per org the user belongs to, so an org is only
reported when every permission macaroon agrees on one. Single-org tokens
-- CI and deploy tokens, and users in a single org -- therefore report an
org, while a token spanning several reports none rather than guessing.
Apps are only reported by tokens narrowed to exactly one app, so app_id
stays empty far more often than org_id. Both fields are omitempty, so an
invocation we can't attribute sends what it sends today.

Deriving this from the tokens rather than from a fetched app is what
makes it worth doing: only 5 of the 39 app-scoped command packages ever
call GetApp, and those overlap with deploy and launch, which already
report both IDs on their own payloads. The remaining 34 -- status, logs,
ssh, scale, machine, volumes and friends -- only ever deal in an app
name, and command stats is the sole event covering them.

Also replaces the WithAppID/WithOrgID context helpers from #5130. They
could never have worked: RecordCommandFinish reads a package-global
context that startMetrics snapshots before any app is resolved, and a
context derived later is a different value the global never sees. A
package-level setter matches how IsUsingGPU is already reported.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@clouvet
clouvet requested a review from dangra August 27, 2026 11:23
@clouvet
clouvet merged commit 2379cfe into master Aug 28, 2026
42 of 43 checks passed
@clouvet
clouvet deleted the command-stats-org-id branch August 28, 2026 03:59
clouvet added a commit that referenced this pull request Sep 4, 2026
The app_id and org_id added to command stats in #5134 come from token
scope, which only names an org when every permission macaroon we hold
agrees on one. Tokens loaded from the config file fan out to one macaroon
per org the user belongs to (fetchOrgTokens), so an interactive user in
more than one org reports nothing rather than guessing. Since everyone
has a personal org alongside any org they work in, that is nearly every
human at a business -- exactly the population we most want to attribute.

Send the app name instead of trying harder to resolve an ID on the
client. RequireAppName already resolves it from the --app flag, FLY_APP
or fly.toml with no I/O at all, and app names are unique, so a name
identifies an app and its org regardless of how many macaroons we are
holding. The warehouse already resolves names this way for deploys and
launches, against raw_web_pg.apps and base__flyio__organizations, so
this needs no new lookup anywhere.

This complements the token-scoped IDs rather than replacing them. The
IDs stay first-hand and are accurate as of the moment the command ran,
where a name resolves to whichever org owns the app today; they also
cover commands that have no app name at all. Roughly half our command
packages are not app-scoped, and for those an org-scoped token remains
the only attribution available. Resolution should prefer the event's own
id and fall back to the name, which is the pattern the existing staging
models already use.

Note that this is a property of the credential, not the person: a token
supplied through the environment skips the org fan-out entirely, so
tightly scoped tokens used for CI and agentic work already report both
IDs today and are unaffected by this.

The field is omitempty, so a command with no app sends what it sends
today.

Co-authored-by: Sprite <noreply@sprites.dev>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants