Skip to content

feat(pipeline): hardened v5 pipeline with staging+MERGE - #51

Open
thalescb wants to merge 2 commits into
masterfrom
feat/pipeline-v5-hardening
Open

feat(pipeline): hardened v5 pipeline with staging+MERGE#51
thalescb wants to merge 2 commits into
masterfrom
feat/pipeline-v5-hardening

Conversation

@thalescb

@thalescb thalescb commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What

Production-grade onchain analytics ingestion pipeline replacing the prototype v1.

Changes

Pipeline v5 (new, pipeline-v5/):

  • Staging table + MERGE write path -- idempotent, atomic, free (replaces streaming inserts)
  • Block-aligned chunking -- no data loss on mid-block crash
  • IngestionStatus + PipelineRuns BigQuery observability tables
  • Slack alerting on failure and data staleness (>36h threshold)
  • Structured JSON logging with run_id tracing
  • Retry with exponential backoff on HyperSync and BQ transient errors
  • Two CLI modes: daily (append to chain tip) and backfill (with --from/--to)
  • Structured exit codes: 0=success, 1=partial, 2=failure
  • Contract config registry -- adding a contract = adding a config entry

Phase 0 fixes (pipeline/index.ts):

  • Inter-batch delay (200ms) preventing HyperSync 429 rate-limiting
  • Global progress tracking surviving throws
  • Exponential backoff with jitter on BQ insert retry
  • Structured exit codes

Scheduling (.github/workflows/pipeline-daily.yml):

  • GitHub Actions cron at 01:00 UTC daily
  • Concurrency group prevents overlap
  • dbt refresh step ready (commented, enable when ready)

Validation

  • Daily mode: both contracts process, exit 0, freshness alerts fire correctly
  • Backfill: 54 events fetched/staged/merged successfully
  • Idempotency: re-running same range produces 0 new rows (MERGE dedup works)
  • PipelineRuns table populated with correct run metadata
  • Slack webhook tested and delivering alerts

Setup

Required GitHub Secrets for the cron workflow:

  • ENVIO_API_TOKEN -- HyperSync API token
  • GCP_SA_KEY -- GCP service account JSON key
  • SLACK_WEBHOOK_URL -- Slack incoming webhook URL

Relates to #40

thalescb added 2 commits July 30, 2026 22:03
- superfluid-api.js: standalone API client + reward calculations + data
  transformations. Reviewable as pure JS, no DOM dependencies.
- index.html: presentation layer importing the module. Dark mode,
  responsive, Chart.js visualizations.
- Season toggle: switch between S6 (live) and S5 (test data, 2611 users)
  to verify correctness.
- Leaderboard with pagination, reward calculator, pool cards, CTAs.
- Event caching across season toggles for instant switching.

Replaces previous single-file approach with proper code/presentation
separation per team review standards.
- Replace streaming inserts with staging table + MERGE (idempotent, atomic, free)
- Block-aligned chunking prevents split-block data loss on crash
- IngestionStatus + PipelineRuns BQ observability tables
- Slack alerting on failure and data staleness (>36h threshold)
- Structured JSON logging with run_id tracing
- Retry with exponential backoff on both HyperSync and BQ errors
- Two CLI modes: daily (append to tip) and backfill (with --from/--to)
- Structured exit codes: 0=success, 1=partial, 2=failure
- Phase 0: stabilize v1 with inter-batch delay, progress tracking, backoff
- GitHub Actions workflow for daily cron at 01:00 UTC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant