Confirming equality of two pandas dataframes?

Question

How to assert that the following two dataframes df1 and df2 are equal?

import pandas as pd df1 = pd.DataFrame([1, 2, 3]) df2 = pd.DataFrame([1.0, 2, 3])

The output of df1.equals(df2) is False. As of now, I know two ways:

print (df1 == df2).all()[0]

or

df1 = df1.astype(float) print df1.equals(df2)

It seems a little bit messy. Is there a better way to do this comparison?

@Divakar np.allclose(df1, df2) works for this case. But what if you have some strings in your dataframes as well? — Mehdi Jafarnia Jahromi
– Mehdi Jafarnia Jahromi, Commented Jul 5, 2016 at 21:10
@Divakar, could you please add it as an answer - it could help others in future? — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Jul 5, 2016 at 21:10
@MaxU Hmm I am not sure, was mostly a wild guess. Also, as OP pointed out for strings it might be producing unexpected output? — Divakar
– Divakar, Commented Jul 5, 2016 at 21:14
try this: np.allclose(df1.select_dtypes(exclude=[object]), df2.select_dtypes(exclude=[object])) & df1.select_dtypes(include=[object]).equals(df2.select_dtypes(include=[object])) - it's based on @Divakar's solution — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Jul 5, 2016 at 21:14

Alexander · Accepted Answer · 2018-08-30 20:40:19Z

42

You can use assert_frame_equal and not check the dtype of the columns.

# Pre v. 0.20.3 # from pandas.util.testing import assert_frame_equal from pandas.testing import assert_frame_equal assert_frame_equal(df1, df2, check_dtype=False)

edited Aug 30, 2018 at 20:40

answered Jul 5, 2016 at 22:40

Alexander

111k32 gold badges212 silver badges208 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Matthew Turner Over a year ago

With pandas 0.20.3 assert_frame_equal is in the pandas.testing package: pandas.pydata.org/pandas-docs/stable/generated/…

Nabin Over a year ago

And important to notice, if there is no any output after the execution of assert_frame_equal function, then the two dataframes are equal.

Community · Accepted Answer · 2017-05-23 10:28:52Z

Using elegant @Divakar's idea - numpy's allclose() will do the main trick for numbers:

In [128]: df1 Out[128]: 0 s n 0 1 aaa 1 1 2 aaa 2 2 3 aaa 3 In [129]: df2 Out[129]: 0 s n 0 1.0 aaa 1.0 1 2.0 aaa 2.0 2 3.0 aaa 3.0 In [130]: (np.allclose(df1.select_dtypes(exclude=[object]), df2.select_dtypes(exclude=[object])) .....: & .....: df1.select_dtypes(include=[object]).equals(df2.select_dtypes(include=[object])) .....: ) Out[130]: True

select_dtypes() will help you to separate strings and all other numeric dtypes

András Aszódi · Accepted Answer · 2025-02-28 15:39:27Z

I am a bit late to the party, but with more modern versions of Pandas you do not need to resort to NumPy's np.allclose() for checking approximate numerical equality. For instance, in recent versions of Pandas (2.2.x at the time of this writing) the assert_frame_equal method supports the check_exact= Boolean option. If it is set to False, then you can control the relative and absolute tolerance of floating-point value comparisons with the rtol= and atol= optional parameters, respectively.

Collectives™ on Stack Overflow

Confirming equality of two pandas dataframes?

3 Answers 3

2 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Linked

Related