fix: handle pandas nullable/extension dtypes in no-pyarrow DataFrame fallback

[27Bslash6]

Summary

Fixes #160. The columnar DataFrame fallback in AutoSerializer._serialize_dataframe (used only when pyarrow is absent and serializer="auto") crashed on modern pandas dtypes:

• Nullable dtypes (Int64/Float64) have a capitalized dtype.name, so they missed the startswith("int")/("float") numeric branch, fell to the object branch, and msgpack.packb() raised on the pd.NA sentinels from series.tolist().
• pyarrow-backed dtypes (int64[pyarrow]) matched startswith("int"), hit the numeric branch, and raised AttributeError on series.values.tobytes() (the Arrow extension array has no .tobytes()).

Fix

• Gate the fast raw-buffer path on a plain numpy numeric dtype — not pd.api.types.is_extension_array_dtype(dtype) and dtype.kind in {i,u,f} — which is reliable where the string-name check was not (and now also covers unsigned ints).
• Make the object branch NA-safe: convert scalar pandas NA sentinels (pd.NA/NaT/NaN) to None so msgpack can pack them; array-like cells (e.g. a list value) are left untouched.

Round-trip through this fallback is intentionally lossy (NA → null; nullable dtype not preserved) — exact dtype fidelity is the Arrow path's job.

Tests

TestColumnarFallbackExtensionDtypes (unit, runs on PRs):

• Int64/Float64/object columns with NA round-trip without crashing; non-NA values and null positions preserved.
• int64[pyarrow] column does not crash.

Verification

• ruff clean, basedpyright 0 errors; 83 auto-serializer unit tests pass (no regression) + 2 new.

Summary by CodeRabbit

• Bug Fixes

  • Stricter numeric-type classification for pandas serialisation; non-eligible dtypes now use an NA-safe object fallback so missing values become nulls while preserving non-missing values.
  • Prevents crashes and improves stability for nullable/extension and pyarrow-backed columns during serialise/deserialise.

• Tests

  • Added coverage for nullable integer/float dtypes, object columns and series, verifying round-trip behaviour and correct handling of missing values.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4164a70c-9c36-451d-93a3-85a9b54d891c

📥 Commits

Reviewing files that changed from the base of the PR and between 921c608 and 6d62bb2.

📒 Files selected for processing (2)

• src/cachekit/serializers/auto_serializer.py
• tests/unit/test_auto_serializer_new_types.py

---

Walkthrough

AutoSerializer's no-pyarrow DataFrame and Series fallback now uses a strict numeric dtype classifier and an NA-safe object-list builder; numeric buffers remain raw bytes, while non-eligible dtypes convert pandas missing sentinels to Python None during serialization.

Changes

Nullable/extension dtype serialisation

Layer / File(s)	Summary
Numeric classifier and NA-safe list helper src/cachekit/serializers/auto_serializer.py	Adds _is_plain_numpy_numeric(dtype) and _na_safe_object_list(series) to detect plain numpy numeric dtypes and build object lists that map pandas missing sentinels to None.
DataFrame serialisation branch src/cachekit/serializers/auto_serializer.py	_serialize_dataframe now uses _is_plain_numpy_numeric(series.dtype) for the numeric/raw-buffer branch; otherwise it serialises columns via _na_safe_object_list instead of series.tolist().
Series serialisation branch src/cachekit/serializers/auto_serializer.py	_serialize_series follows the same rule: raw-buffer encoding only for _is_plain_numpy_numeric, otherwise uses _na_safe_object_list rather than series.tolist().
Tests for nullable/extension dtype round-trip tests/unit/test_auto_serializer_new_types.py	Adds TestColumnarFallbackExtensionDtypes with DataFrame and Series tests covering pandas nullable Int64/Float64 and int64[pyarrow]-backed dtypes, asserting round-trip non-missing value preservation and NA→null conversion.

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 54.55% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarises the main change: handling pandas nullable and extension dtypes in the DataFrame fallback serialisation path.
Description check	✅ Passed	The description covers all key sections from the template including Summary, Motivation, Type of Change, Testing, and Verification, with detailed explanations of the fix.
Linked Issues check	✅ Passed	The PR fully addresses the requirements from issue #160: replaces brittle string-based dtype detection with _is_plain_numpy_numeric(), implements NA-safe serialisation, and adds comprehensive tests for nullable and pyarrow-backed dtypes.
Out of Scope Changes check	✅ Passed	All changes are scoped to fixing the nullable/extension dtype handling in the no-pyarrow DataFrame fallback. The modifications to AutoSerializer and corresponding test additions are directly aligned with issue #160.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/nullable-dtype-fallback

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/cachekit/serializers/auto_serializer.py (1)

750-754: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

_serialize_series has the same vulnerability that was fixed in _serialize_dataframe.

The Series serialisation path still uses the old dtype.name.startswith("int"/"float") check and does not convert NA sentinels to None. This means:

1. pyarrow-backed dtypes (e.g., int64[pyarrow]) will match startswith("int") and crash with AttributeError when calling .values.tobytes().
2. Nullable dtypes with NA values fall through to the tolist() branch but without NA→None conversion, so msgpack.packb() will fail on pd.NA sentinels.

Apply the same fix pattern used in _serialize_dataframe:

🐛 Proposed fix to align Series serialisation with DataFrame fix

-        if series.dtype.name.startswith("int") or series.dtype.name.startswith("float"):
+        if not pd.api.types.is_extension_array_dtype(series.dtype) and series.dtype.kind in ("i", "u", "f"):
             serialized.update({"type": "numeric", "data": series.values.tobytes(), "dtype": str(series.dtype)})  # type: ignore[union-attr]
         else:
-            # Use tolist() for object types - preserves datetime objects for custom encoder
-            serialized.update({"type": "object", "data": series.tolist()})
+            # Strings, datetimes, and nullable/extension columns. tolist() preserves
+            # datetime objects for our custom encoder; scalar pandas NA sentinels
+            # (pd.NA/NaT/NaN) are converted to None so msgpack can pack them.
+            values = []
+            for v in series.tolist():
+                try:
+                    is_na = bool(pd.isna(v))
+                except (TypeError, ValueError):
+                    is_na = False  # array-like cell (e.g. a list) is not a scalar NA
+                values.append(None if is_na else v)
+            serialized.update({"type": "object", "data": values})

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cachekit/serializers/auto_serializer.py` around lines 750 - 754, The
Series serialization uses dtype.name.startswith(...) and doesn't convert pandas
NA sentinels to None; update _serialize_series to mirror the
_serialize_dataframe fix: replace the startswith checks with pandas.api.types
helpers (e.g., is_integer_dtype/is_float_dtype or is_numeric_dtype) when
inspecting series.dtype, and in the non-numeric/object path convert NA sentinels
to Python None (e.g., use series.where(series.notna(), None) or
series.astype(object).where(pd.notna(series), None) before calling tolist());
for the numeric branch use series.to_numpy().tobytes() (keeping
dtype=str(series.dtype)) instead of series.values.tobytes() and handle
pyarrow-backed/extension dtypes via the same try/fallback pattern used in
_serialize_dataframe so AttributeError from .tobytes() doesn't crash.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unit/test_auto_serializer_new_types.py`:
- Around line 697-708: Add parallel Series tests to mirror the DataFrame tests
so regressions in _serialize_series/_deserialize_series are caught: create two
new test functions in tests/unit/test_auto_serializer_new_types.py —
test_series_nullable_dtypes_roundtrip and
test_series_pyarrow_backed_dtype_does_not_crash — that import pandas (and
pyarrow for the second via pytest.importorskip), instantiate
AutoSerializer(enable_integrity_checking=False), build a nullable Int64 Series
and a pyarrow-backed int64 Series respectively, call ser._serialize_series(...)
then ser._deserialize_series(...), and assert the returned Series preserves
values and nulls (e.g. out.iloc checks and pd.isna) and that pyarrow-backed
roundtrip returns [1,2,3]; this will surface the old dtype-detection bug in
_serialize_series.

---

Outside diff comments:
In `@src/cachekit/serializers/auto_serializer.py`:
- Around line 750-754: The Series serialization uses dtype.name.startswith(...)
and doesn't convert pandas NA sentinels to None; update _serialize_series to
mirror the _serialize_dataframe fix: replace the startswith checks with
pandas.api.types helpers (e.g., is_integer_dtype/is_float_dtype or
is_numeric_dtype) when inspecting series.dtype, and in the non-numeric/object
path convert NA sentinels to Python None (e.g., use series.where(series.notna(),
None) or series.astype(object).where(pd.notna(series), None) before calling
tolist()); for the numeric branch use series.to_numpy().tobytes() (keeping
dtype=str(series.dtype)) instead of series.values.tobytes() and handle
pyarrow-backed/extension dtypes via the same try/fallback pattern used in
_serialize_dataframe so AttributeError from .tobytes() doesn't crash.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 85c3f152-1909-4776-8537-9710829630c3

📥 Commits

Reviewing files that changed from the base of the PR and between d90958c and 921c608.

📒 Files selected for processing (2)

• src/cachekit/serializers/auto_serializer.py
• tests/unit/test_auto_serializer_new_types.py

[codecov]

Codecov Report

❌ Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/cachekit/serializers/auto_serializer.py	90.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!