PERF: ArrowExtensionArray.fillna when array does not contains any nulls

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.1.0.rst file if fixing a bug or adding a new feature.

import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(1_000_000, 10), dtype="float64[pyarrow]") %timeit df.fillna(0.0) 

25 ms ± 5.42 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) -> main 76.4 µs ± 4.43 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each) -> PR 

[phofl]

Could you test this on #51411 as well? This should address the problem for CoW

[lukemanley]

Could you test this on #51411 as well? This should address the problem for CoW

I tested locally on top of your branch in #51411 and it seems to work. What "problem" are you referring to?

[phofl]

Performance, we are avoiding the copy on the block level there

[lukemanley]

@phofl - hmmm... clearly this doesn't play well with #51411

Not sure what I did yesterday to convince myself it did... sorry

[lukemanley]

actually, what is the expected behavior for COW wrt pyarrow? Since arrow arrays are immutable, ArrowExtensionArray.copy doesn't actually copy the underlying pyarrow array. I think tests like this (which are failing) are expected to fail:

import pandas as pd import numpy as np arr = pd.array([1, 2, 3], dtype="int64[pyarrow]") arr2 = arr.copy() assert not np.shares_memory(arr, arr2) # -> raises 

[phofl]

CoW handles the copies globally on the block level, hence we can remove the check above.

You can adjust the tests, eas made to many defensive copies without CoW, you got rid of one of them, which is a good thing

[lukemanley]

Ok will do. Is the comment you added still relevant? (# TODO(CoW): Not necessary anymore when CoW is the default)

I would think this fastpath can still apply after CoW is the default (e.g. for whatever reason CoW is not being used)

[phofl]

We are not planning on having a non CoW mode when CoW is the default

[mroeschke]

Merge when ready @phofl

[mroeschke]

Could you rebase @lukemanley

[mroeschke]

Thanks @lukemanley