Skip to content

feat(pgpm-export): generic deterministic table-data export (shared data-dump helper)#1433

Merged
pyramation merged 3 commits into
mainfrom
feat/export-data-dump
Jul 23, 2026
Merged

feat(pgpm-export): generic deterministic table-data export (shared data-dump helper)#1433
pyramation merged 3 commits into
mainfrom
feat/export-data-dump

Conversation

@pyramation

@pyramation pyramation commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Generic deterministic table-data export in @pgpmjs/export — the shared data-dump capability that exportScopedSourcePlaneData (constructive-db#2388) re-implemented downstream because META_EXPORT_SCHEMAS is hardcoded (constructive-planning#1242 P3, tracked in constructive-planning#1244).

All emitted SQL is built through @constructive-io/query-builder (pg-ast + pgsql-deparser) — no string concatenation, no manual identifier quoting.

New pgpm/export/src/export-data.ts:

  • getDataExportColumns(db, schema, table) — one canonical column introspection over pg_catalog (pg_attribute/pg_class/pg_namespace/pg_attrdef). Why pg_catalog and not the metaschema field catalog: this exports rows from arbitrary physical tables (routing/catalog/apps schemas included), so the physical table is the source of truth for what columns/defaults actually exist. The fragile timestamp-default regex is replaced by real volatility detection: function OIDs are extracted from the compiled default expression (pg_attrdef.adbin) and joined to pg_proc.provolatile (plus a SQLVALUEFUNCTION fallback for CURRENT_TIMESTAMP-style value functions). pg_depend can't be used — built-in functions like now() are pinned and get no dependency rows.
  • isVolatileTimestampColumn — excludes timestamp columns whose default depends on a non-immutable function; constant timestamp defaults survive.
  • exportTableData / exportTablesData — deterministic single-INSERT-per-table export (rows ordered by id::text, else by all columns), excludeColumns, equality filter, optional ON CONFLICT DO NOTHING. SELECT/INSERT built via QueryBuilder (selectExpr(cast(col(...), 'text')), .insert(rows) with cast(lit(v), colType) values, .onConflict({action:'nothing'})).
  • buildDataDeployScript / buildDataRevertScript / buildDataVerifyScript — replica-wrapped deploy; real revert (per-table DELETE ... WHERE id IN (...) in reverse dependency order, via QueryBuilder.delete().where({id:{in: ids.map(lit)}})), fixing the empty revert stub; divide-by-zero verify (1 / (count(*) = N)::int).

Supporting addition in @constructive-io/query-builder: a cast(operand, typeName) expression helper (TypeCast node) with a parseTypeName that handles format_type output — typmods (numeric(10,2)), array suffix (text[]), and SQL-standard multi-word names mapped to their pg_catalog aliases (timestamp with time zonepg_catalog.timestamptz).

Validation

  • pgpm/export: full package suite 153/153 (8 suites), incl. 7 new DB-backed export-data tests (volatility detection incl. interval/constant/clock_timestamp defaults, determinism, filters, conflict handling, id-less tables, and a live deploy→verify→revert→verify-fails→redeploy round-trip). Build green.
  • postgres/query-builder: 112/112 tests (incl. new cast() coverage). Build green.
  • Note: the repo's pnpm lint (ESLint 9) currently fails config discovery repo-wide (no eslint.config.js); files were linted with the legacy .eslintrc config — no errors introduced.

#2388's branch is untouched; the cutover lane adopts this after publish.

Link to Devin session: https://app.devin.ai/sessions/ad40d16f38a349b48eb7ce9375e27e6e
Requested by: @pyramation

@pyramation pyramation self-assigned this Jul 23, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@pyramation
pyramation merged commit f400661 into main Jul 23, 2026
14 checks passed
@pyramation
pyramation deleted the feat/export-data-dump branch July 23, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant