feat(pgpm-export): generic deterministic table-data export (shared data-dump helper)#1433
Merged
Conversation
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generic deterministic table-data export in
@pgpmjs/export— the shared data-dump capability thatexportScopedSourcePlaneData(constructive-db#2388) re-implemented downstream becauseMETA_EXPORT_SCHEMASis hardcoded (constructive-planning#1242 P3, tracked in constructive-planning#1244).All emitted SQL is built through
@constructive-io/query-builder(pg-ast + pgsql-deparser) — no string concatenation, no manual identifier quoting.New
pgpm/export/src/export-data.ts:getDataExportColumns(db, schema, table)— one canonical column introspection over pg_catalog (pg_attribute/pg_class/pg_namespace/pg_attrdef). Why pg_catalog and not the metaschema field catalog: this exports rows from arbitrary physical tables (routing/catalog/apps schemas included), so the physical table is the source of truth for what columns/defaults actually exist. The fragile timestamp-default regex is replaced by real volatility detection: function OIDs are extracted from the compiled default expression (pg_attrdef.adbin) and joined topg_proc.provolatile(plus aSQLVALUEFUNCTIONfallback forCURRENT_TIMESTAMP-style value functions).pg_dependcan't be used — built-in functions likenow()are pinned and get no dependency rows.isVolatileTimestampColumn— excludes timestamp columns whose default depends on a non-immutable function; constant timestamp defaults survive.exportTableData/exportTablesData— deterministic single-INSERT-per-table export (rows ordered byid::text, else by all columns),excludeColumns, equalityfilter, optionalON CONFLICT DO NOTHING. SELECT/INSERT built viaQueryBuilder(selectExpr(cast(col(...), 'text')),.insert(rows)withcast(lit(v), colType)values,.onConflict({action:'nothing'})).buildDataDeployScript/buildDataRevertScript/buildDataVerifyScript— replica-wrapped deploy; real revert (per-tableDELETE ... WHERE id IN (...)in reverse dependency order, viaQueryBuilder.delete().where({id:{in: ids.map(lit)}})), fixing the empty revert stub; divide-by-zero verify (1 / (count(*) = N)::int).Supporting addition in
@constructive-io/query-builder: acast(operand, typeName)expression helper (TypeCastnode) with aparseTypeNamethat handlesformat_typeoutput — typmods (numeric(10,2)), array suffix (text[]), and SQL-standard multi-word names mapped to their pg_catalog aliases (timestamp with time zone→pg_catalog.timestamptz).Validation
pgpm/export: full package suite 153/153 (8 suites), incl. 7 new DB-backedexport-datatests (volatility detection incl. interval/constant/clock_timestamp defaults, determinism, filters, conflict handling, id-less tables, and a live deploy→verify→revert→verify-fails→redeploy round-trip). Build green.postgres/query-builder: 112/112 tests (incl. newcast()coverage). Build green.pnpm lint(ESLint 9) currently fails config discovery repo-wide (noeslint.config.js); files were linted with the legacy.eslintrcconfig — no errors introduced.#2388's branch is untouched; the cutover lane adopts this after publish.
Link to Devin session: https://app.devin.ai/sessions/ad40d16f38a349b48eb7ce9375e27e6e
Requested by: @pyramation