ENH: InvalidIndexError using at for multiple assignment

[sjfleming]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/75688441/pandas-invalidindexerror-during-multiple-assignment-using-at

Question about pandas

This might not be a bug, so I'm not opening the issue as a bug.

I used to rely on at for multiple assignment in the following way:

import pandas as pd
import numpy as np

df = pd.DataFrame(data={'a': [2, 4, 1, 5, 56, 6, 1, 2, 1, 0], 
                        'b': [3, 4, 5, 2, 6, 7, 5, 3, 5, 6]}, 
                  index=range(10))
df.at[df['b'] == 6, 'b'] = np.nan  # np.nan is not special. any float here will produce the same thing.

df

which produces (my desired output) in pandas versions from 1.2.3 to 1.3.5 (I did not test before 1.2.3):

    a    b
0   2  3.0
1   4  4.0
2   1  5.0
3   5  2.0
4  56  NaN
5   6  7.0
6   1  5.0
7   2  3.0
8   1  5.0
9   0  NaN

As of version 1.4.0 (and above), the same code produces the following error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sfleming/opt/anaconda3/envs/test_installs/lib/python3.8/site-packages/pandas/core/indexing.py", line 2273, in __setitem__
    return super().__setitem__(key, value)
  File "/Users/sfleming/opt/anaconda3/envs/test_installs/lib/python3.8/site-packages/pandas/core/indexing.py", line 2228, in __setitem__
    self.obj._set_value(*key, value=value, takeable=self._takeable)
  File "/Users/sfleming/opt/anaconda3/envs/test_installs/lib/python3.8/site-packages/pandas/core/frame.py", line 3871, in _set_value
    loc = self.index.get_loc(index)
  File "/Users/sfleming/opt/anaconda3/envs/test_installs/lib/python3.8/site-packages/pandas/core/indexes/range.py", line 388, in get_loc
    self._check_indexing_error(key)
  File "/Users/sfleming/opt/anaconda3/envs/test_installs/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 5637, in _check_indexing_error
    raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: 0    False
1    False
2    False
3    False
4     True
5    False
6    False
7    False
8    False
9     True
Name: b, dtype: bool

Is this a bug? Should this work?

If this is the desired behavior, a more descriptive error message would help others debug much faster.

[MarcoGorelli]

when I try this I get

InvalidIndexError: You can only assign a scalar value not a <class 'float'>

so it looks like a clear error message has already been added

[MarcoGorelli]

closing as a duplicate of #48729 then

[sjfleming]

Hmm, I'm not sure I agree with you. (Also not sure why I'm not actually seeing that error message.) "You can only assign a scalar value not a <class 'float'>" does not tell me that the problem is that .at can only modify one dataframe element at a time. A 'float' is a scalar value, so I find it confusing.

I gather that the new way to do this kind of multiple-element assignment is using .loc

[MarcoGorelli]

not sure why I'm not actually seeing that error message

you might be on an old version of pandas, please try upgrading

---

but you're right, the error message probably can be improved for when you're trying to set a scalar but with a mask - reopening then

[LFBJC]

@MarcoGorelli I'd suggest something like this: "Tried to set a scalar value with a mask, please use a scalar index or use df.loc instead". What do you think?

[MarcoGorelli]

i haven't looked at this closely, but it it's possible to add that error message in such a way that it wouldn't be incorrect in some cases, then that looks good

interested in opening a pull request?