fix(serialization): encode object arrays and builtin dtypes in specs

[ChrisW09]

Fixes #66.

A fitted Preprocessor whose object-dtype state holds numpy scalars (such as np.bool_ in SimpleImputer.statistics_ or OneHotEncoder.categories_) could not be fingerprinted or saved: fingerprint_, reproducibility_report() and json.dumps(to_spec()) raised TypeError, and to_spec(path) left a truncated file behind, overwriting any spec already at that path. Preprocessor(dtype=float) (or int, bool) saved a spec that from_spec always refused.

Root cause

• _encode wrote an object array as obj.tolist(), which keeps the original element objects, so numpy scalars inside it never went through the scalar conversion and reached json.dumps.
• _encode wrote every type as module:qualname. Builtin dtype specifiers resolve to the builtins module, which the loader's allow-list refuses.
• to_spec(path) opened the file with "w" and streamed json.dump into it, so an encoding error left a truncated file.

Fix

• _encode (pretab/compose/serialize.py) encodes the elements of an object array one by one through the new _map_elements helper. The __ndarray__ format is unchanged. _decode_ndarray fills an object array element by element with _decode, so a decoded element that is itself a sequence stays one element.
• bool, int, float and complex (_BUILTIN_DTYPES) are written as {"__npdtype__": ...} for the numpy dtype they denote. Any other type from a module outside the allow-list raises PretabSerializationError when the spec is built.
• The new _spec_json in pretab/preprocessor.py serializes a spec to JSON text and raises PretabSerializationError for a TypeError or ValueError from the JSON writer. Preprocessor.to_spec builds the text before it opens path, and fingerprint_ uses the same helper.

Behaviour changes

• A builtin dtype (float, int, bool, complex) is stored as the numpy dtype it denotes. After from_spec, the dtype parameter is that np.dtype (for example dtype('float64')), not the builtin type, and fingerprint_ of such a preprocessor changes. Fingerprints of the default preprocessor, one without imputers and a one-hot preprocessor are unchanged (compared with main).
• to_spec() and fingerprint_ raise PretabSerializationError instead of TypeError when the spec cannot be written as JSON. to_spec() serializes the spec once even without path, so this error is raised there instead of in a later json.dumps.
• State that references a type outside pretab, sklearn, numpy and scipy raises PretabSerializationError at save time. Before, the spec was written and from_spec refused it.
• A failed to_spec(path) leaves an existing file at path untouched.

Documentation

• Preprocessor.to_spec docstring: new Raises section (a value or type a spec cannot store or load; raised before path is opened).
• Preprocessor.fingerprint_ docstring: new Raises section.
• Preprocessor docstring, dtype parameter: a builtin specifier such as float is stored in a spec as the numpy dtype it denotes.
• docs/core_concepts/reproducibility.md ("Portable serialization"): PretabSerializationError is raised when the spec is built, and a failed to_spec(path) leaves an existing spec at that path untouched.
• Docstrings of the new _map_elements and _spec_json.

Tests

• test_object_arrays_of_numpy_scalars_serialize: the issue's np.bool_ / None object column; json.dumps(to_spec()) and fingerprint_ work, and a file round trip reproduces transform. Fails on main, passes here.
• test_builtin_and_numpy_dtypes_round_trip[float|int]: a builtin dtype survives a JSON round trip with identical output and fingerprint. Fails on main, passes here.
• test_builtin_and_numpy_dtypes_round_trip[float32]: the same for np.float32. Guard test: passes on both by design.
• test_failed_save_leaves_an_existing_spec_untouched: a spec that the JSON writer rejects raises PretabSerializationError, and the file saved earlier at the same path is unchanged. Fails on main, passes here.
• test_types_a_spec_cannot_load_are_rejected_at_save_time: _encode(list) raises PretabSerializationError. Fails on main, passes here.
• test_object_arrays_written_before_element_encoding_still_load: an object array in the 1.0.0 format decodes to the same 2-D object array. Guard test: passes on both by design.

Reproduction

Output of the script from #66 on this branch:

fingerprint_: f7270b1d7eab1ba76aeaa148393697a6991c9259f2c28862bf3417430f09a00b
json.dumps(to_spec()): 12906 characters
to_spec(path): ok
bytes left on disk: 39326
from_spec(path): same transform: True
spec dtype: {'__npdtype__': '<f8'}
from_spec: dtype dtype('float64') | same transform: True

Each call runs in its own try/except, as in the issue, and prints its result; both from_spec calls also compare transform with the original preprocessor.

Checks

• just test: 1445 passed, 59 skipped, 7 xfailed; coverage 93.74% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1432 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.

• The PR for Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80 (fix(serialization): write specs as strict JSON without NaN tokens #134) changes the same code. Both add the same _map_elements helper, which git merges once, and they conflict in the ndarray branch of _encode, in _decode_ndarray, in to_spec and fingerprint_ in pretab/preprocessor.py, and at the end of tests/integration/test_serialization.py, where both append tests. Resolution:

  • _encode: keep this PR's object branch, data = _map_elements(data, obj.ndim, _encode) (_encode then tags non-finite floats itself), followed by Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80's elif obj.dtype.kind == "f" and not np.isfinite(obj).all(): data = _map_elements(data, obj.ndim, _encode_float).
  • _decode_ndarray: keep this PR's element-by-element object branch (its _decode knows the __float__ tag), then decode float arrays with data = _map_elements(data, len(payload["shape"]), _decode) when dtype.kind == "f".
  • Drop Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80's _encode_object_element and _decode_float_element.
  • pretab/preprocessor.py: keep this PR's to_spec and fingerprint_ and pass allow_nan=False in _spec_json (json.dumps(spec, allow_nan=False, **kwargs)) instead of Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80's two direct dumps. Optionally say "strict JSON" in its docstring.
  • Tests: keep both blocks.

  Resolved this way, the full suite passes (1449 passed, 59 skipped, 7 xfailed), and the scripts from to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66 and Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80 both show the fixed behaviour.

• The PR for Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 (fix(transformers): record and check DataFrame feature names #135) changes _representation_summary and the ..exceptions import in pretab/compose/serialize.py. Git merges it cleanly with this branch.

This change was previously part of #75, which is closed in favour of one pull request per issue.