API: ignore empty range/object dtype in Index setop operations (string dtype compat)

[jorisvandenbossche]

Exploring one option to address #60338

This makes that inserting/union etc with an empty RangeIndex or empty Index with object dtype (the types of index that get created for empty objects by default), ignores the dtype of that empty index.
This improves the workflow of creating an empty dataframe and adding columns or resetting the index (you can see the removed xfails). Also most of the test changes are changes that are essentially reverting previous edits (typically adding an explicit dtype=object) to get the tests passing for infer_string, and I think almost all those test changes show a better behaviour now.

I still want to add some explicit tests for the original use case (although everything is implicitly already covered in other tests, I think)

[WillAyd]

lgtm - I think this is a pretty reasonable workaround for the time being

[rhshadrach]

Agreed. While it's undesirable to me to ignore empty object dtype when using e.g. concat, using StringDtype should make having object dtype in the first place much more of an edge case. And the impact of this on e.g. adding columns to an empty DataFrame is worth the tradeoff.

[rhshadrach]

lgtm, needs a whatsnew, yea? I'd suggest a section at the top, I expect this to be somewhat commonly hit.

[jorisvandenbossche]

Added a whatsnew note (for 2.3, didn't directly find a suitable existing section, but we will have to clean up that file anyway)

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.3.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 ee06e714fcb35e0b6d321b3edd454eb0e363e5e4

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #60797: API: ignore empty range/object dtype in Index setop operations (string dtype compat)'

4. Push to a named branch:

git push YOURFORK 2.3.x:auto-backport-of-pr-60797-on-2.3.x

5. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #60797 on branch 2.3.x (API: ignore empty range/object dtype in Index setop operations (string dtype compat))"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

[jorisvandenbossche]

Manual backport -> #60948