You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adding or multiplying a pandas nullable dtype Series with a pyarrow backed dtype Series raises a TypeError, but reversing the order works as expected
Tested and confirmed that the same problem occurs for any combination of numeric pandas nullable dtypes and pyarrow dtypes
('Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64', 'Float32', 'Float64') and
('int8[pyarrow]', 'int16[pyarrow]', 'int32[pyarrow]', 'int64[pyarrow]', 'uint8[pyarrow]', uint16[pyarrow]', 'uint32[pyarrow]', 'uint64[pyarrow]', 'float32[pyarrow]', 'float64[pyarrow]']) respectively
In [30]: a = pd.Series(range(5), dtype="Float64")
...: b = pd.Series(range(5), dtype="float64[pyarrow]")
In [31]: a * b
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[31], line 1
----> 1 a * b
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:76, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
72 return NotImplemented
74 other = item_from_zerodim(other)
---> 76 return method(self, other)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:202, in OpsMixin.__mul__(self, other)
200 @unpack_zerodim_and_defer("__mul__")
201 def __mul__(self, other):
--> 202 return self._arith_method(other, operator.mul)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:6126, in Series._arith_method(self, other, op)
6124 def _arith_method(self, other, op):
6125 self, other = self._align_for_op(other)
-> 6126 return base.IndexOpsMixin._arith_method(self, other, op)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\base.py:1382, in IndexOpsMixin._arith_method(self, other, op)
1379 rvalues = np.arange(rvalues.start, rvalues.stop, rvalues.step)
1381 with np.errstate(all="ignore"):
-> 1382 result = ops.arithmetic_op(lvalues, rvalues, op)
1384 return self._construct_result(result, name=res_name)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:273, in arithmetic_op(left, right, op)
260 # NB: We assume that extract_array and ensure_wrapped_if_datetimelike
261 # have already been called on `left` and `right`,
262 # and `maybe_prepare_scalar_for_op` has already been called on `right`
263 # We need to special-case datetime64/timedelta64 dtypes (e.g. because numpy
264 # casts integer dtypes to timedelta64 when operating with timedelta64 - GH#22390)
266 if (
267 should_extension_dispatch(left, right)
268 or isinstance(right, (Timedelta, BaseOffset, Timestamp))
(...)
271 # Timedelta/Timestamp and other custom scalars are included in the check
272 # because numexpr will fail on it, see GH#31457
--> 273 res_values = op(left, right)
274 else:
275 # TODO we should handle EAs consistently and move this check before the if/else
276 # (https://github.com/pandas-dev/pandas/issues/41165)
277 # error: Argument 2 to "_bool_arith_check" has incompatible type
278 # "Union[ExtensionArray, ndarray[Any, Any]]"; expected "ndarray[Any, Any]"
279 _bool_arith_check(op, left, right) # type: ignore[arg-type]
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:76, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
72 return NotImplemented
74 other = item_from_zerodim(other)
---> 76 return method(self, other)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:202, in OpsMixin.__mul__(self, other)
200 @unpack_zerodim_and_defer("__mul__")
201 def __mul__(self, other):
--> 202 return self._arith_method(other, operator.mul)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\masked.py:806, in BaseMaskedArray._arith_method(self, other, op)
803 # x ** 0 is 1.
804 mask = np.where((self._data == 0) & ~self._mask, False, mask)
--> 806 return self._maybe_mask_result(result, mask)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\masked.py:870, in BaseMaskedArray._maybe_mask_result(self, result, mask)
867 if result.dtype.kind == "f":
868 from pandas.core.arrays import FloatingArray
--> 870 return FloatingArray(result, mask, copy=False)
872 elif result.dtype.kind == "b":
873 from pandas.core.arrays import BooleanArray
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\numeric.py:245, in NumericArray.__init__(self, values, mask, copy)
239 if not (isinstance(values, np.ndarray) and checker(values.dtype)):
240 descr = (
241 "floating"
242 if self._dtype_cls.kind == "f" # type: ignore[comparison-overlap]
243 else "integer"
244 )
--> 245 raise TypeError(
246 f"values should be {descr} numpy array. Use "
247 "the 'pd.array' function instead"
248 )
249 if values.dtype == np.float16:
250 # If we don't raise here, then accessing self.dtype would raise
251 raise TypeError("FloatingArray does not support np.float16 dtype.")
TypeError: values should be integer numpy array. Use the 'pd.array' function instead
In [32]: b * a
Out[32]:
0 0.0
1 1.0
2 4.0
3 9.0
4 16.0
dtype: double[pyarrow]
Expected Behavior
Addition and multiplication should work with either order of operands
Installed Versions
INSTALLED VERSIONS
commit : bdc79c1
python : 3.12.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252
Thanks for the report! I agree the behavior of addition and multiplication should be consistent. Addition and multiplication are commutative so a + b should be equal to b + a (even if it means both should fail to accomplish this commutativity)
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
Adding or multiplying a pandas nullable dtype Series with a pyarrow backed dtype Series raises a TypeError, but reversing the order works as expected
Tested and confirmed that the same problem occurs for any combination of numeric pandas nullable dtypes and pyarrow dtypes
('Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64', 'Float32', 'Float64') and
('int8[pyarrow]', 'int16[pyarrow]', 'int32[pyarrow]', 'int64[pyarrow]', 'uint8[pyarrow]', uint16[pyarrow]', 'uint32[pyarrow]', 'uint64[pyarrow]', 'float32[pyarrow]', 'float64[pyarrow]']) respectively
Expected Behavior
Addition and multiplication should work with either order of operands
Installed Versions
INSTALLED VERSIONS
commit : bdc79c1
python : 3.12.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252
pandas : 2.2.1
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 69.2.0
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.1.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.22.2
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.8
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
gcsfs : None
matplotlib : 3.8.3
numba : 0.59.1
numexpr : 2.9.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 15.0.2
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2024.3.1
scipy : 1.12.0
sqlalchemy : None
tables : None
tabulate : None
xarray : 2024.2.0
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None
The text was updated successfully, but these errors were encountered: