fix(query)!: carry result cells as JSON text - #138
Conversation
A JSON number in a result row was parsed into a `serde_json::Value`, which has no arbitrary-precision number variant. Anything wider than an f64 can hold was rounded at deserialization, before a caller saw it: a DECIMAL(38,2) the service sent as 99999999999999999999.99 arrived as 1e20. The service's own bytes were exact; the loss was entirely on this side. Rows now carry `JsonCell`, a transparent newtype over `Box<RawValue>` holding the cell's JSON text and re-emitting it verbatim, so a number stays an unquoted JSON number and round-trips to the service's bytes. `raw_value` is deliberate and `arbitrary_precision` is not an option: that feature is crate-wide and changes how every serde_json::Number serializes, which corrupts non-JSON formats for downstreams. The row-cell schema is named in the spec normalizer and passed to the generator via type/import mappings, so regeneration keeps emitting the type and only the cell is pinned; the surrounding models still flow. A regen guard fails if `Vec<Vec<serde_json::Value>>` reappears. BREAKING CHANGE: `QueryResponse::rows`, `GetResultResponse::rows` and `QueryResponse::new` take `Vec<Vec<JsonCell>>`. `to_value()` returns the previous `serde_json::Value`, with its rounding, for callers not ready to move.
RELEASING.md is explicit that versions are not bumped by hand: notes go under [Unreleased] and `./scripts/release.sh prepare` opens the release PR. Bumping here also failed check-release.py, which wants a matching CHANGELOG section for a bumped version.
There was a problem hiding this comment.
Approved. Two non-blocking nits inline on src/models/json_cell.rs.
Checked the type change against every consumer of rows: src/query.rs (fetch_all_rows, cell_len), the generated models, examples/quickstart.rs and README.md. No other call site reads a cell. The generated API layer deserializes with serde_json::from_str, which RawValue supports, and the double_option wrapper on GetResultResponse::rows passes the original deserializer through, so both row paths keep the raw text. QueryResponse derives Clone, Default, Debug, PartialEq, Serialize, Deserialize, all of which JsonCell provides.
The regen guards cover both halves of the mapping: the field type and the mod.rs re-export. The generator behavior behind --import-mappings is not verifiable here, but drift fails the workflow rather than shipping rounded numbers.
CI checks for tests were still running at review time.
`as_str` returned an owned `String`, one allocation per call, where `Value::as_str` had handed back a borrowed `&str` — a new per-row cost on the migration path for anyone reading a string column. It now returns `Cow<'_, str>`, borrowing the slice between the quotes and decoding only an escaped string. `to_value` mapped a parse failure to `Value::Null`, which no caller could tell apart from a cell that really is null. The text is valid JSON by construction, so a failure is a broken invariant: panic and name it rather than substitute an absence.
There was a problem hiding this comment.
Both prior nits are addressed. as_str now borrows the unescaped body, which is exact: JSON forbids an unescaped ", \, or control character inside a string, so the text between the quotes is the value. to_value no longer substitutes a null.
Note that CI checks for this push were still queued or in progress when this review ran, so the test results are not visible here.
Both fallbacks invented a value for a state that cannot occur: a failed quote strip became an empty string, and a failed decode of an escaped string became `None`, which says "not a string" about a cell that is one. Either reads as real data. Panic and name the invariant, matching `to_value`.
There was a problem hiding this comment.
Cycle 3 review: the two expect calls in as_str sit on branches the RawValue invariant closes, and the escaped/plain split stays correct for a trailing escaped quote, a lone backslash, and an empty string. Response bodies are parsed with serde_json::from_str, which supports raw-value capture. JsonCell supplies every trait the two response models derive. No blocking issues; prior nits are resolved.
Note: the build and test checks were still queued when this review ran, so this approval does not assert that CI passed.
The bug
A DECIMAL wide enough to need more than ~17 significant digits comes back rounded, silently.
The response bytes are exact.
rowswas typedVec<Vec<serde_json::Value>>, andValuehas no arbitrary-precision number variant — anything that does not fit ani64/u64is parsed throughf64. The digits were gone at deserialization, before any caller could see them.Integers are unaffected (
Valuehas reali64/u64variants). In practice this is DECIMAL.The fix
rowsnow carriesJsonCell: a#[serde(transparent)]newtype overBox<RawValue>holding the cell's JSON text and re-emitting it verbatim. Nothing is decided at parse time, so the digits the service wrote are the digits that come back. A number stays an unquoted JSON number — a cell round-trips to the service's own bytes rather than turning into a string.Reading a cell:
as_json_str()(lossless),as_str(),as_array()(elements keep their own digits),is_null(),kind().to_value()returns the previousserde_json::Value, rounding included, for callers not ready to move.arbitrary_precisionis not an option here and theCargo.tomlcomment says so: it is a crate-wide feature that changes how everyserde_json::Numberserializes, which corrupts non-JSON formats for anything downstream.raw_valueonly adds a type and changes nothing for code that does not name it.Keeping it through regeneration
src/models/query_response.rsis generated, andregenerate.ymldoesrm -rf src/apis src/models docsbefore generating — so ignore-listing the file would delete it and never recreate it, leavingmod.rsdeclaring a missing module. Instead:scripts/normalize-openapi.py(which already rewrites the throwaway spec for generator reasons) names the anonymous row-cell schemaJsonCellregenerate.ymlpasses--type-mappings/--import-mappingsfor itVec<Vec<models::JsonCell>>and skips writing the model; one hand-writtensrc/models/json_cell.rssupplies the typeOnly the cell type is pinned — the surrounding models keep regenerating normally, so a future spec change to
QueryResponseis not silently dropped. A regen guard fails the workflow ifVec<Vec<serde_json::Value>>reappears ormod.rsstops re-exportingJsonCell, and the normalizer exits non-zero if the spec'srowsshape changes out from under it.This was verified by running openapi-generator 7.22.0 twice — unpatched vs. with the normalizer — and diffing the trees. The whole delta is 4 type substitutions, 2 lines in
mod.rs, and 2 doc lines, which is exactly what is committed here. The checked-in tree equals what the next regeneration produces.Verification
Both paths that carry rows:
a_wide_decimal_survives_the_inline_path/results/{id}—a_wide_decimal_survives_result_paginationcargo test --all-features --lib: 113 passed. Full suite green.Breaking change
QueryResponse::rows,GetResultResponse::rowsandQueryResponse::newnow takeVec<Vec<JsonCell>>. Comparisons against a literal becomeJsonCell::from(serde_json::json!(...)).serde_jsonis the only supported format for aJsonCell— it round-trips through aRawValue, which other serde formats do not recognise;to_value()is the way out to one.The version is deliberately not bumped here.
RELEASING.mdstates versions are never bumped by hand — notes go under## [Unreleased]and./scripts/release.sh prepare minoropens the release PR. That release needs to land beforehotdata-clican pick this up, since its own fix depends onJsonCell.Unrelated, noticed while testing
cargo testwithout--all-featuresfails to compileexamples/quickstart.rs: its#[cfg(not(feature = "arrow"))]stubs have drifted from their call sites. Pre-existing onmainand untouched here — CI only ever builds--all-features, so the non-arrow configuration is never compiled. Left out of this PR; worth its own fix.