BUG: Fix assert_frame_equal with check_dtype=False for pd.NA dtype differences (GH#61473)

[gamzeozgul]

• closes BUG: assert_frame_equal(check_dtype=False) fails when comparing two DFs containing pd.NA that only differ in dtype (object vs Int32) #61473
• tests added / passed
• Ensure all linting tests pass
• whatsnew entry

Problem

When comparing two DataFrames containing pd.NA values with check_dtype=False, assert_frame_equal fails when the DataFrames only differ in dtype (object vs Int32). This happens because pd.NA and np.nan are treated as different values even though they represent the same missing value.

Solution

Modified assert_frame_equal in pandas/_testing/asserters.py to normalize pd.NA and np.nan values when check_dtype=False is specified. This ensures that DataFrames with equivalent missing values but different dtypes can be compared successfully.

Changes Made

• Added normalization logic in assert_frame_equal function to handle pd.NA and np.nan equivalence when check_dtype=False
• Added comprehensive unit test in pandas/tests/util/test_assert_frame_equal.py to verify the fix

Testing

• Added unit test test_assert_frame_equal_pd_na_dtype_difference that reproduces the original issue and verifies the fix
• Test passes successfully with the implemented solution

[gamzeozgul]

Hello,

It seems that the "Downstream Compat" CI job is failing with an ImportError related to statsmodels and scipy. This appears to be an issue with the CI environment's dependencies (e.g., version mismatch between statsmodels and scipy), and not directly related to the changes introduced in this PR.

The error message is:

_______________________________ test_statsmodels _______________________________ E pytest.PytestDeprecationWarning: E Module 'statsmodels.formula.api' was found, but when imported by pytest it raised: E ImportError("cannot import name '_lazywhere' from 'scipy._lib._util' (/home/runner/micromamba/envs/test/lib/python3.11/site-packages/scipy/_lib/_util.py)") 

I believe this is an infrastructure/dependency issue on the CI side, rather than a bug in my proposed changes for GH#61473.

Could you please confirm this and let me know if there's anything I can do to help resolve this CI issue, or if this is something the core pandas team needs to address?

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.