Skip to content

[db-dtypes] Support pandas 3.1 ExtensionArray compliance interface changes #18514

Description

@chalmerlowe

Summary

When testing with pandas >= 3.1.0 (currently in pre-release on PyPI as pandas==3.1.0rc0), the db-dtypes compliance test suite (tests/compliance) fails due to upstream changes in pandas internal utilities and ExtensionArray interface contracts.

To prevent presubmit PR CI from failing unexpectedly on developer branches, a temporary ceiling constraint (pandas < 3.1.0dev0) was added in packages/db-dtypes/testing/constraints-3.14.txt. This issue tracks the feature work required to update db-dtypes for full pandas 3.1 compatibility.


Problem Details

1. Private Import Removal (np_version_gt2)

The compliance test files (test_date_compliance.py, test_json_compliance.py, test_time_compliance.py) currently import a private helper:

from pandas.compat.numpy import np_version_gt2

In pandas 3.1, this private attribute has been removed from pandas.compat.numpy, resulting in test collection errors:

ImportError: cannot import name 'np_version_gt2' from 'pandas.compat.numpy'

2. ExtensionArray Interface Contract Changes

Pandas 3.1 introduces stricter interface requirements in pandas.tests.extension.base:

  • fillna mapping: ExtensionArray.fillna() no longer accepts dictionary or mapping values.
  • Comparison Operator Outputs: Changes to boolean comparison return types and PyArrow-backed boolean handling.
  • Datetime / ZoneInfo fixture dependencies: Test fixtures generating localized timestamps require explicit tzdata resolution on newer Python runtimes.

Desired Solution

  1. Decouple Version Checks:
    Replace from pandas.compat.numpy import np_version_gt2 in tests/compliance/ with standard version checks using packaging.version.parse(np.__version__) >= Version("2.0.0").
  2. Update Array Implementations for Pandas 3.1:
    Audit and update DateArray, TimeArray, and JSONArray in db_dtypes/dtypes.py to satisfy both pandas 3.0 and pandas 3.1 ExtensionArray contracts simultaneously.
  3. Verify Pre-release & Compliance:
    Run both unit and compliance test suites against pandas>=3.1.0rc0 using nox -s unit_prerelease to ensure 100% compliance.
  4. Remove Temporary Constraint:
    Remove pandas < 3.1.0dev0 from packages/db-dtypes/testing/constraints-3.14.txt.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions