-
-
Notifications
You must be signed in to change notification settings - Fork 18.5k
BUG: Fix pd.json_normalize to not skip the first element of a generator input #38698
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
898ccdf
66f5ba6
8537705
e885c86
89546ab
bf25860
9f1f2f9
775f80e
f9e5332
cc9cfae
2ca6be7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,7 @@ | ||
# --------------------------------------------------------------------- | ||
# JSON normalization routines | ||
|
||
from collections import defaultdict | ||
from collections import abc, defaultdict | ||
import copy | ||
from typing import Any, DefaultDict, Dict, Iterable, List, Optional, Union | ||
|
||
|
@@ -262,6 +262,11 @@ def _pull_records(js: Dict[str, Any], spec: Union[List, str]) -> List: | |
if isinstance(data, list) and not data: | ||
return DataFrame() | ||
|
||
if isinstance(data, abc.Iterator): | ||
# GH35923 Fix pd.json_normalize to not skip the first element of a | ||
# generator input | ||
data = list(data) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This could have some big performance implications when dealing with large generators - is it not alternately possible to just store the first element for inspection and reuse as necessary while maintaining the state of the generator? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we barely support generators (its not even documented), so -1 if this adds any complexity. |
||
|
||
# A bit of a hackjob | ||
if isinstance(data, dict): | ||
data = [data] | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you make these if/elif (all 3 conditions)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done