API: consistently preserve null values for `**0` in nullable dtypes

[MarcoGorelli]

In [35]: pd.Series([1, pd.NA], dtype='Int64') ** 0
Out[35]:
0    1
1    1
dtype: Int64

In [36]: pd.Series([1, pd.NA], dtype='Int64[pyarrow]') ** 0
Out[36]:
0       1
1    <NA>
dtype: int64[pyarrow]

The pandas nullable types return 1 for NA ** 0, whereas the pyarrow ones preserve NA

Following the general rule outlined in #62506 (comment) by @jorisvandenbossche

I think the general rule is that NA values always propagate in element-wise functions (certainly for a generic ufunc where we might not know what it is doing exactly, for the reason @twast outlined), except for a function that specifically checks for NA (isna/notna).

, and also for compatibility with other tools, I'd suggest that NA should be consistently preserved here

---

Might this be considered for a breaking change in 3.0? #57064

[jorisvandenbossche]

The original issue that discussed this topic and which led to introducing this behaviour: #29997 (cc @TomAugspurger)

[TomAugspurger]

, and also for compatibility with other tools, I'd suggest that NA should be consistently preserved here

What are the other tools? In #29997, it looks like consistency with other languages and how NA behaves in binops was the motivation.

[MarcoGorelli]

here's some examples

duckdb / sql:

In [4]: duckdb.sql("""select a, a**0 from values (1), (null) df(a)""")
Out[4]:
┌───────┬──────────┐
│   a   │ (a ** 0) │
│ int32 │  double  │
├───────┼──────────┤
│     1 │      1.0 │
│  NULL │     NULL │
└───────┴──────────┘

pyarrow:

In [8]: pc.power(pa.array([1, None]), 0)
Out[8]:
<pyarrow.lib.Int64Array object at 0x7e1aa80eee00>
[
  1,
  null
]

pyspark:

In [13]: df.withColumn('a**0', F.col('a')**0).show()
+----+----+
|   a|a**0|
+----+----+
|   1| 1.0|
|NULL|NULL|
+----+----+

polars

In [20]: pl.DataFrame({'a': [1, None]}).with_columns(pl.col('a').pow(0).alias('a**0'))
Out[20]:
shape: (2, 2)
┌──────┬──────┐
│ a    ┆ a**0 │
│ ---  ┆ ---  │
│ i64  ┆ i64  │
╞══════╪══════╡
│ 1    ┆ 1    │
│ null ┆ null │
└──────┴──────┘

[jorisvandenbossche]

I think at the time, we modeled it partly after how this behaved in other packages that use NaN and interpolated that to NA (i.e. numpy for lack of NA, and also in R NA and NaN are a bit a strange hybrid (they distinguish, but in NaN is often treated as NA as well), and for NaN there seems to be consistent behaviour to return the "known" result for power if NaN is one of the operands:

PyArrow:

>>> import pyarrow.compute as pc
>>> pc.power(pa.array([1, None, np.nan]), 0)
<pyarrow.lib.DoubleArray object at 0x7f10eefdb700>
[
  1,
  null,
  1
]

Polars:

>>> pl.DataFrame({'a': [1.0, None, np.nan]}).with_columns(pl.col('a').pow(0).alias('a**0'))
shape: (3, 2)
┌──────┬──────┐
│ a    ┆ a**0 │
│ ---  ┆ ---  │
│ f64  ┆ f64  │
╞══════╪══════╡
│ 1.0  ┆ 1.0  │
│ null ┆ null │
│ NaN  ┆ 1.0  │
└──────┴──────┘

DuckDB

>>> duckdb.sql("""select a, a**0 from values (1.0::FLOAT), (null), ('NaN') df(a)""")
┌───────┬──────────┐
│   a   │ (a ** 0) │
│ float │  double  │
├───────┼──────────┤
│   1.0 │      1.0 │
│  NULL │     NULL │
│   nan │      1.0 │
└───────┴──────────┘

For numpy we could of course only compare with NaN, and then it seems that (only?) R treats NaN and NA similar in this regard:

> c(1.0, NA, NaN) ^ 0
[1] 1 1 1

[jorisvandenbossche]

For rationale, pasting Tom's comment from the PR here:

General question - is this behavior defined or standardized somewhere?

It's part of some IEEE spec, but I haven't tracked down the source. Both Python and NumPy use this behavior for NaN.

For NA, I think it aligns with the principles behind Kleene logic. True | NA is True, since True | anything is True. Similarly, NA ** 0 is 1, since anything ** 0 is 1.

And from the issue:

I think our invariant is that NA represents unknown. So if you have an operation that is know with only one operand, like “True | NA”, then the output should be known and not NA.

---

Personally, I think this is a case where there are valid arguments both ways, and choosing one will always make some people unhappy (a bit like the sum of empty being 0 vs NA case, but then on a smaller scale ;)).
Always propagating nulls in arithmetic operations gives a simpler behaviour (and implementation), while returning a non-NA value if the result can be known is more in line with the boolean operators (Kleene logic).

But given that most of the comparable tools nowadays follow the "always propagate" case for arithmetic, that feels like a good reason to go for that as well (if there are no strong reasons otherwise to pick the other behaviour).

[TomAugspurger]

if there are no strong reasons otherwise to pick the other behaviour

Probably the only reason would be if people are relying on the current behavior, which is hard to know.