Summary
When testing with pandas >= 3.1.0 (currently in pre-release on PyPI as pandas==3.1.0rc0), the db-dtypes compliance test suite (tests/compliance) fails due to upstream changes in pandas internal utilities and ExtensionArray interface contracts.
To prevent presubmit PR CI from failing unexpectedly on developer branches, a temporary ceiling constraint (pandas < 3.1.0dev0) was added in packages/db-dtypes/testing/constraints-3.14.txt. This issue tracks the feature work required to update db-dtypes for full pandas 3.1 compatibility.
Problem Details
1. Private Import Removal (np_version_gt2)
The compliance test files (test_date_compliance.py, test_json_compliance.py, test_time_compliance.py) currently import a private helper:
from pandas.compat.numpy import np_version_gt2
In pandas 3.1, this private attribute has been removed from pandas.compat.numpy, resulting in test collection errors:
ImportError: cannot import name 'np_version_gt2' from 'pandas.compat.numpy'
2. ExtensionArray Interface Contract Changes
Pandas 3.1 introduces stricter interface requirements in pandas.tests.extension.base:
fillna mapping: ExtensionArray.fillna() no longer accepts dictionary or mapping values.
- Comparison Operator Outputs: Changes to boolean comparison return types and PyArrow-backed boolean handling.
- Datetime / ZoneInfo fixture dependencies: Test fixtures generating localized timestamps require explicit
tzdata resolution on newer Python runtimes.
Desired Solution
- Decouple Version Checks:
Replace from pandas.compat.numpy import np_version_gt2 in tests/compliance/ with standard version checks using packaging.version.parse(np.__version__) >= Version("2.0.0").
- Update Array Implementations for Pandas 3.1:
Audit and update DateArray, TimeArray, and JSONArray in db_dtypes/dtypes.py to satisfy both pandas 3.0 and pandas 3.1 ExtensionArray contracts simultaneously.
- Verify Pre-release & Compliance:
Run both unit and compliance test suites against pandas>=3.1.0rc0 using nox -s unit_prerelease to ensure 100% compliance.
- Remove Temporary Constraint:
Remove pandas < 3.1.0dev0 from packages/db-dtypes/testing/constraints-3.14.txt.
Summary
When testing with
pandas >= 3.1.0(currently in pre-release on PyPI aspandas==3.1.0rc0), thedb-dtypescompliance test suite (tests/compliance) fails due to upstream changes in pandas internal utilities andExtensionArrayinterface contracts.To prevent presubmit PR CI from failing unexpectedly on developer branches, a temporary ceiling constraint (
pandas < 3.1.0dev0) was added inpackages/db-dtypes/testing/constraints-3.14.txt. This issue tracks the feature work required to updatedb-dtypesfor full pandas 3.1 compatibility.Problem Details
1. Private Import Removal (
np_version_gt2)The compliance test files (
test_date_compliance.py,test_json_compliance.py,test_time_compliance.py) currently import a private helper:In pandas 3.1, this private attribute has been removed from
pandas.compat.numpy, resulting in test collection errors:2. ExtensionArray Interface Contract Changes
Pandas 3.1 introduces stricter interface requirements in
pandas.tests.extension.base:fillnamapping:ExtensionArray.fillna()no longer accepts dictionary or mapping values.tzdataresolution on newer Python runtimes.Desired Solution
Replace
from pandas.compat.numpy import np_version_gt2intests/compliance/with standard version checks usingpackaging.version.parse(np.__version__) >= Version("2.0.0").Audit and update
DateArray,TimeArray, andJSONArrayindb_dtypes/dtypes.pyto satisfy both pandas 3.0 and pandas 3.1 ExtensionArray contracts simultaneously.Run both
unitandcompliancetest suites againstpandas>=3.1.0rc0usingnox -s unit_prereleaseto ensure 100% compliance.Remove
pandas < 3.1.0dev0frompackages/db-dtypes/testing/constraints-3.14.txt.