ENH: Add dtype argument to str.decode

[rhshadrach]

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Ref: #60795 (comment)

PyArrow-backed strings cannot handle surrogates. When users have infer_string=True and PyArrow installed, they can end up with a failure they can't workaround when calling str.decode. Adding the dtype argument allows for a workaround.

[jorisvandenbossche]

FWIW, is_string_dtype currently (confusingly, sometimes, if you really only want StringDtype) also returns True for object dtype, so this final and not is_object_dtype is not strictly needed (although I find it useful for reading the code ;))

[rhshadrach]

Thanks!

[jorisvandenbossche]

Looks good!

Maybe add a test with actual surrogates to make sure the dtype=object workaround works in that case?

[rhshadrach]

Maybe add a test with actual surrogates to make sure the dtype=object workaround works in that case?

There is a surrogate in test_decode_object_dtype.

[WillAyd]

Have to think about this some more but my initial reaction is a -1. If someone needs non UTF-8 support I think we should push them to using the object dtype, rather than extending the API like this

[jorisvandenbossche]

Maybe add a test with actual surrogates to make sure the dtype=object workaround works in that case?

There is a surrogate in test_decode_object_dtype.

Sorry, missed that!

If someone needs non UTF-8 support I think we should push them to using the object dtype, rather than extending the API like this

That's exactly what this PR is intending to enable? With this keyword, people can choose to use object dtype explicitly (without it, we always use str dtype, which then fails)

[WillAyd]

With this keyword, people can choose to use object dtype explicitly (without it, we always use str dtype, which then fails)

Ah sorry I am getting tripped up over terminology and the existing API. I'm guessing this is a Python2 relic that we even offer str.decode, since there is no string method in Python3 for decode.

So makes sense, I just want to really limit the use of keywords to control data types because as they can be hard to reason about. I don't think it can be avoided specifically for this method though

[mroeschke]

Thanks @rhshadrach

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.3.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 63249f2aa95ef0b0300ea2f1cc68200cc8b13484

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #60940: ENH: Add dtype argument to str.decode'

4. Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-60940-on-2.3.x

5. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #60940 on branch 2.3.x (ENH: Add dtype argument to str.decode)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[rhshadrach]

I'm guessing this is a Python2 relic that we even offer str.decode, since there is no string method in Python3 for decode.

It's bytes.decode, I think we just don't want to add a bytes namespace so we stashed it under str.

[WillAyd]

Back in Python2 the str type had a decode method, as it was closer to an array of bytes than the unicode object type that it currently is in Python3. The accessor being under that namespace today is a bit of cruft owing back to that legacy.

Certainly not something to fix here, but its an interesting API nonetheless

[WillAyd]

Actually giving this a little more thought, instead of adding a dtype argument what if we coerce to object when the encoding is anything but utf-8 and just use the standard string inference logic when it is utf-8?

[rhshadrach]

I'm negative on adding values-specific behavior.