Skip to content

[file-diet] File Diet: refactor pkg/cli/token_usage.go (1141 lines) into focused modules #47078

Description

@github-actions

Overview

The file pkg/cli/token_usage.go has grown to 1141 lines, making it difficult to maintain and test. This task involves refactoring it into smaller, focused files with improved test coverage.

Current State

  • File: pkg/cli/token_usage.go
  • Size: 1141 lines
  • Test file: pkg/cli/token_usage_test.go (782 lines, ratio ~0.69)
  • Complexity: High — the file mixes data models, file parsing, subagent attribution, steering-event detection, AIC computation, and summary aggregation
Full File Analysis

Logical Groups Found

Data models / structs (lines 29–131):

  • TokenCoreMetrics, TokenUsageEntry, AmbientContextMetrics, TokenUsageSummary, ModelTokenUsage, ModelTokenUsageRow, SubagentModelRequest, SubagentModelActual, agentUsageEntry, proxyEventsEntry

Token-usage JSONL parsing (lines 136–360):

  • parseTokenUsageFile, extractAmbientContextMetrics, parseTokenUsageTimestamp, findTokenUsageFile

Agent-usage JSON parsing (lines 360–500):

  • findAgentUsageFile, parseAgentUsageFile

AIC (AI Credits) computation (lines 543–777):

  • findUsageJSONLFiles, extractUsageRecord, usageNumericValue, usageStringValue, isFinite, sumAICFromUsageJSONLFiles, processOneUsageJSONLFile, analyzeTokenUsageAICOnly

Steering events detection (lines 778–894):

  • countAPIProxySteeringEvents, findAPIProxyEventsFile, scanSteeringEntries, parseAPIProxySteeringEvents, containsSteeringKeyword, isSteeringEvent

Subagent attribution (lines 896–1072):

  • augmentSubagentModelAttribution, addTokenUsageWarning, extractSubagentModelRequests, findAgentStdioFile, correlateToolCallsWithTokenDelta

Summary methods / AIC population (lines 1079–1141):

  • TotalTokens, AvgDurationMs, ModelRows, populateAIC

Top-level orchestrator (lines 501–542):

  • analyzeTokenUsage

Refactoring Strategy

Proposed File Splits

  1. token_usage_models.go

    • Types: TokenCoreMetrics, TokenUsageEntry, AmbientContextMetrics, TokenUsageSummary, ModelTokenUsage, ModelTokenUsageRow, SubagentModelRequest, SubagentModelActual
    • Methods: TotalTokens, AvgDurationMs, ModelRows
    • Responsibility: Shared data types and summary helpers
    • Estimated LOC: ~200
  2. token_usage_parser.go

    • Functions: parseTokenUsageFile, extractAmbientContextMetrics, parseTokenUsageTimestamp, findTokenUsageFile, parseAgentUsageFile, findAgentUsageFile, agentUsageEntry
    • Responsibility: Parse JSONL/JSON log files into domain structs
    • Estimated LOC: ~300
  3. token_usage_aic.go

    • Functions: findUsageJSONLFiles, extractUsageRecord, usageNumericValue, usageStringValue, isFinite, sumAICFromUsageJSONLFiles, processOneUsageJSONLFile, analyzeTokenUsageAICOnly, populateAIC
    • Responsibility: AI Credits computation from raw usage JSONL
    • Estimated LOC: ~260
  4. token_usage_steering.go

    • Types: proxyEventsEntry
    • Functions: countAPIProxySteeringEvents, findAPIProxyEventsFile, scanSteeringEntries, parseAPIProxySteeringEvents, containsSteeringKeyword, isSteeringEvent
    • Responsibility: Detect and count steering events from proxy event logs
    • Estimated LOC: ~130
  5. token_usage_subagent.go

    • Functions: augmentSubagentModelAttribution, addTokenUsageWarning, extractSubagentModelRequests, findAgentStdioFile, correlateToolCallsWithTokenDelta
    • Responsibility: Attribute token usage to subagents from stdio logs
    • Estimated LOC: ~200
  6. token_usage.go (shrunk to orchestrator + constants)

    • Functions: analyzeTokenUsage
    • Constants and package-level vars
    • Responsibility: Entry point that wires together all sub-packages
    • Estimated LOC: ~80

Shared Utilities

No new utility files needed; helpers (usageNumericValue, usageStringValue, isFinite) should move into token_usage_aic.go alongside their callers.

Interface Abstractions

  • Consider a logFileLocator interface to abstract find*File helpers, making it easier to test with in-memory paths.
Test Coverage Plan

Add or redistribute tests across new files:

  1. token_usage_models_test.go

    • Test TotalTokens, AvgDurationMs, ModelRows with various inputs
    • Target coverage: >80%
  2. token_usage_parser_test.go

    • Test JSONL parsing with valid, malformed, and empty files
    • Test extractAmbientContextMetrics edge cases (missing first entry, etc.)
    • Target coverage: >80%
  3. token_usage_aic_test.go

    • Test sumAICFromUsageJSONLFiles with single and multi-file scenarios
    • Test processOneUsageJSONLFile with various key name variants
    • Target coverage: >80%
  4. token_usage_steering_test.go

    • Test scanSteeringEntries with token and timeout events, noise lines
    • Test isSteeringEvent with all known event names
    • Target coverage: >80%
  5. token_usage_subagent_test.go

    • Test extractSubagentModelRequests with realistic stdio log snippets
    • Test augmentSubagentModelAttribution with mismatch scenarios
    • Target coverage: >80%

Implementation Guidelines

  1. Preserve Behavior: Ensure all existing functionality works identically
  2. Maintain Exports: Keep public API unchanged (exported functions/types)
  3. Add Tests First: Write tests for each new file before refactoring
  4. Incremental Changes: Split one module at a time
  5. Run Tests Frequently: Verify make test-unit passes after each split
  6. Update Imports: Ensure all import paths are correct
  7. Document Changes: Add comments explaining module boundaries

Acceptance Criteria

  • token_usage.go is split into 5–6 focused files
  • Each new file is under 500 lines
  • All tests pass (make test-unit)
  • Test coverage is ≥80% for new files
  • No breaking changes to public API
  • Code passes linting (make lint)
  • Build succeeds (make build)
Additional Context
  • Repository Guidelines: Follow patterns in .github/agents/developer.instructions.agent.md
  • Code Organization: Prefer many small files grouped by functionality
  • Testing: Match existing test patterns in pkg/cli/*_test.go

Priority: Medium
Effort: Medium (well-defined boundaries, no external API changes needed)
Expected Impact: Improved maintainability, easier testing, reduced complexity per file

Generated by 🧹 Daily File Diet · sonnet46 79.5 AIC · ⌖ 7.02 AIC · ⊞ 6.8K ·

  • expires on Jul 23, 2026, 5:11 AM UTC-08:00

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions