feat(pipeline): hardened v5 pipeline with staging+MERGE - #51
Open
thalescb wants to merge 2 commits into
Open
Conversation
- superfluid-api.js: standalone API client + reward calculations + data transformations. Reviewable as pure JS, no DOM dependencies. - index.html: presentation layer importing the module. Dark mode, responsive, Chart.js visualizations. - Season toggle: switch between S6 (live) and S5 (test data, 2611 users) to verify correctness. - Leaderboard with pagination, reward calculator, pool cards, CTAs. - Event caching across season toggles for instant switching. Replaces previous single-file approach with proper code/presentation separation per team review standards.
- Replace streaming inserts with staging table + MERGE (idempotent, atomic, free) - Block-aligned chunking prevents split-block data loss on crash - IngestionStatus + PipelineRuns BQ observability tables - Slack alerting on failure and data staleness (>36h threshold) - Structured JSON logging with run_id tracing - Retry with exponential backoff on both HyperSync and BQ errors - Two CLI modes: daily (append to tip) and backfill (with --from/--to) - Structured exit codes: 0=success, 1=partial, 2=failure - Phase 0: stabilize v1 with inter-batch delay, progress tracking, backoff - GitHub Actions workflow for daily cron at 01:00 UTC
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Production-grade onchain analytics ingestion pipeline replacing the prototype v1.
Changes
Pipeline v5 (new,
pipeline-v5/):daily(append to chain tip) andbackfill(with--from/--to)Phase 0 fixes (
pipeline/index.ts):Scheduling (
.github/workflows/pipeline-daily.yml):Validation
Setup
Required GitHub Secrets for the cron workflow:
ENVIO_API_TOKEN-- HyperSync API tokenGCP_SA_KEY-- GCP service account JSON keySLACK_WEBHOOK_URL-- Slack incoming webhook URLRelates to #40