Document text-column widening and DuckDB output data types (CFTL-713 / SUPPORT-16781) - #1084
Conversation
Follow-up to SUPPORT-16781 / CFTL-713, where a customer read the docs as promising that the types cast in a DuckDB script are reproduced in Storage, and asked for two gaps to be filled. Native Data Types: - Note the one exception to "a typed column's type cannot be changed": the length of a text column can be increased in place, and that widening is one-way, so VARCHAR(n) in a transformation is not a durable constraint. - Point at the Storage API endpoint that makes the same change explicitly. - Say that base types being a small set means unmapped source types land as STRING, scoped to components that describe output with base types. DuckDB transformations: - Add an Output Data Types section with the DuckDB -> base type mapping, verified against component-duckdb-transformation src/component.py. - Cover what arrives as text and why (incl. FLOAT/REAL, which DuckDB reports as FLOAT and the component does not map), that VARCHAR(n) length cannot survive DuckDB, and the pre-created typed output table as the supported way to enforce a width or pin timestamp precision. - Reword the "properly typed columns" line that caused the misreading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
jordanrburger
left a comment
There was a problem hiding this comment.
Two things before this can go in.
1. Nine new --- in prose — these will render as three literal hyphens. Confirmed on production: /catalog/multi-project/ currently ships separate project --- Oracle while ora-history — The source database on the same page is fine, so SmartyPants converts -- and leaves --- alone. The ones here:
type actually holds them --- if the source engine…a longer length --- for example from VARCHAR(2)…the original VARCHAR(2) fails --- loading a shorter length…nullability and default value --- can also be made explicitly…(#output-data-types) --- they are not copied verbatim…sql/data_types/text) --- it accepts VARCHAR(n)…still an array literal --- and the load then fails…Everything else --- FLOAT/REAL, TIME, INTERVAL…(in the mapping table)end up with in Storage --- see [Output Data Types]…
Swap them for —. Worth knowing #1069 and #1070 are cleaning 300+ of these out of the corpus right now, so this would land fresh instances of the thing we're removing.
2. You're right about the widening paragraph needing a Storage nod, and I don't want to merge it without one. The reproduction is solid, but this documents a guarantee customers will build on, and the part that isn't in public code is exactly the load-bearing part — that the load path applies it, that it holds for incremental, and that it holds on BigQuery. I'll take that to Storage. Hold this one until it comes back.
Also mergeStateStatus is BEHIND, so it needs a rebase.
Separately — the two things you turned up while verifying are component bugs, not doc bugs:
FLOAT/REALfalling through toSTRINGbecause there's noFLOATbranch (andREALin the mapping code is unreachable, since DuckDB only reportsFLOAT).DECIMAL(10,2)[]getting declared asNUMERIC(10,2)and then failing the load.
Both deserve issues against component-duckdb-transformation on their own. Documenting them is the right stopgap but shouldn't be the end of it.
Docs follow-up to CFTL-713 / SUPPORT-16781 (Košík.cz). Engineering on the ticket is done; these are the two documentation gaps the customer and @matyas-jirat asked for.
Changes
/storage/tables/data-types/(Native Data Types) — the page stated flatly that a typed column's type cannot be changed after creation, with no exception. It now documents that the length of a text column can be increased in place (VARCHAR(2)→VARCHAR(50), metadata-only, rows preserved, full and incremental), that widening is one-way so a later shorter declaration fails, and therefore thatVARCHAR(n)written in a transformation is not a durable constraint — the enforced length is the current Storage definition. Also links thePUT …/columns/{column}/definitionendpoint that makes the same change explicitly, and scopes the new Base Types note to components that describe their output with base types./transformations/duckdb/— new Output Data Types section: the DuckDB → Keboola base type mapping table, what arrives as text and why, thatVARCHAR(n)length cannot survive DuckDB, and the pre-created typed output table as the supported way to enforce a width or pin timestamp precision. The line "The output table then contains properly typed columns" — which the customer read as "my casts are reproduced" — is reworded.Verification
The mapping table was checked against
keboola/component-duckdb-transformationsrc/component.py(_map_base_type,duckdb_type_to_base_type,_LENGTH_BEARING_TYPES), not against the ticket thread, and DuckDB's ownDESCRIBEoutput (duckdb 1.5.5). Two things that turned up while verifying and are now documented as behaviour:FLOAT/REALis not mapped. DuckDB reports both asFLOAT; the component has noFLOATbranch, so such a column falls through toSTRING. The page tells readers to cast toDOUBLE. Worth a component fix separately (REALis in the mapping code but unreachable, sinceREALis only a DuckDB alias).DECIMAL(10,2)[]) are declared asNUMERIC(10,2)because only the leading type name is read. Documented in a caution with the cast-to-text workaround.The timestamp-precision row (
TIMESTAMP_S/_MS/_NS→0/3/9) matches released behaviour as of component 0.1.7 (component PR #21, merged and released 2026-08-05) — verified inmain.npm run buildis clean andscripts/audit-phase2.mjsshows no new issues (147 total, same asmain; all 45 broken links pre-existing).One note for the reviewer
❓ Text-column widening would benefit from a Storage owner's nod. The behaviour is reproduced by the customer on a production Snowflake project (screenshots in the ticket) and the widen/one-way logic is visible in
php-table-backend-utils(SnowflakeTableQueryBuilder::getUpdateColumnFromDefinitionQuery,CANNOT_DECREASE_LENGTH), but that the load path applies it — and that it holds for incremental loads and on BigQuery — is only confirmed empirically, not in public code. Component Factory considersVARCHARout of their scope, so if anyone from Storage can confirm or correct that paragraph, I'll adjust the wording.🤖 Generated with Claude Code