Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parents/ancestry methods #231

Merged
merged 9 commits into from
Apr 7, 2025

Conversation

jm-rivera
Copy link
Contributor

@jm-rivera jm-rivera commented Mar 31, 2025

This PR is part of a group of PRs which will bring some key features from the Data Commons website API to the client library (e.g #229, #230)

TL;DR
Fetch entity parents: A new Node method (fetch_entity_parents) which uses containedInPlace to get the immediate parents for a given entity or list of entities.
Fetch entity ancestry: A new Node method fetch_entity_ancestry performs a parallel breadth-first traversal to construct the full ancestry chain for an entity — all the way up to Earth — and returns it in either flat or nested form.


With this PR, I aimed to replicate the /api/place/parent of the website API. Under the hood, this endpoint relies on legacy calls to the REST V1 bulk info endpoint.

Being able to get the full ancestry for an entity (or list of entities) is a very helpful feature. However, doing this through the Node endpoint requires recursively resolving each parent until you reach the root of the place graph (in this case always "Earth"). This PR introduces logic to make that process as efficient and easy to use as possible for users (and being careful not to make any repeated or unnecessary calls to the API).

  • Breadth-first traversal: I used a breadth-first approach (build_ancestry_map) to ensure we resolve ancestry level by level, starting from the root entity. This avoids recursion and makes parallel execution easier to manage and reason about.
  • Parallel fetching with threads: Since every level of the graph may require one or more network calls to resolve containedInPlace, I used ThreadPoolExecutor to fetch parent nodes in parallel. This drastically reduces total resolution time for large (e.g many entities) or deep ancestry chains (e.g. "Kampala" -> "Uganda" -> "Africa" -> "Earth").
  • Memoization with LRU cache: To avoid repeated API calls for shared parents (e.g. if two cities belong to the same country, two countries to the same region, etc.), I used an @lru_cache wrapper on the fetcher. This ensures we only fetch a node’s parents once per ancestry resolution session.

The data can be returned as two different types:

  • A flat list of parent dicts (as_tree=False) mirrors the structure returned by the website’s /api/place/parent — for parity with the website version or cases when only the list of upstream places is needed.
  • A nested ancestry tree (as_tree=True) returns a full parent-child hierarchy rooted at the input entity. Note that not all nodes in the place graph appear to be fully connected — this method doesn’t attempt to infer or resolve missing links in the hierarchy.

Example usage:

For entity parents

from datacommons_client import DataCommonsClient

dc = DataCommonsClient(dc_instance="datacommons.one.org")

parents = dc.node.fetch_entity_parents(entity_dcids=[
    "africa",
    "wikidataId/Q2608785",
])

{'africa': [{'dcid': 'Earth', 'name': 'World', 'type': 'Place'}],
 'wikidataId/Q2608785': [{'dcid': 'country/GTM',
                          'name': 'Guatemala',
                          'type': 'Country'}]}

By itself, the above doesn't match the website endpoint, or provide a complete ancestry.

Full ancestry

from datacommons_client import DataCommonsClient

dc = DataCommonsClient(dc_instance="datacommons.one.org")

parents = dc.node.fetch_entity_ancestry(
    entity_dcids=[
        "africa",
        "wikidataId/Q2608785",
    ],
    as_tree=False,
)


{'africa': [{'dcid': 'Earth', 'name': 'World', 'type': 'Place'}],
 'wikidataId/Q2608785': [{'dcid': 'country/GTM',
                          'name': 'Guatemala',
                          'type': 'Country'},
                         {'dcid': 'CentralAmerica',
                          'name': 'Central America (including Mexico)',
                          'type': 'UNGeoRegion'},
                         {'dcid': 'LatinAmericaAndCaribbean',
                          'name': 'Latin America and the Caribbean',
                          'type': 'UNGeoRegion'},
                         {'dcid': 'northamerica',
                          'name': 'North America',
                          'type': 'Continent'},
                         {'dcid': 'undata-geo/G00134000',
                          'name': 'Americas',
                          'type': 'GeoRegion'},
                         {'dcid': 'Earth', 'name': 'World', 'type': 'Place'}]}

Which can also be returned as a nested structure:

parents = dc.node.fetch_entity_ancestry(
    entity_dcids=[
        "africa",
        "wikidataId/Q2608785",
    ],
    as_tree=True,
)

{'africa': {'dcid': 'africa',
            'name': None,
            'parents': [{'dcid': 'Earth',
                         'name': 'World',
                         'parents': [],
                         'type': 'Place'}],
            'type': None},
 'wikidataId/Q2608785': {'dcid': 'wikidataId/Q2608785',
                         'name': None,
                         'parents': [{'dcid': 'country/GTM',
                                      'name': 'Guatemala',
                                      'parents': [{'dcid': 'CentralAmerica',
                                                   'name': 'Central America '
                                                           '(including Mexico)',
                                                   'parents': [{'dcid': 'Earth',
                                                                'name': 'World',
                                                                'parents': [],
                                                                'type': 'Place'},
                                                               {'dcid': 'LatinAmericaAndCaribbean',
                                                                'name': 'Latin '
                                                                        'America '
                                                                        'and '
                                                                        'the '
                                                                        'Caribbean',
                                                                'parents': [],
                                                                'type': 'UNGeoRegion'},
                                                               {'dcid': 'undata-geo/G00134000',
                                                                'name': 'Americas',
                                                                'parents': [],
                                                                'type': 'GeoRegion'}],
                                                   'type': 'UNGeoRegion'},
                                                  {'dcid': 'LatinAmericaAndCaribbean',
                                                   'name': 'Latin America and the Caribbean',
                                                   'parents': [{'dcid': 'Earth',
                                                                'name': 'World',
                                                                'parents': [],
                                                                'type': 'Place'},
                                                               {'dcid': 'undata-geo/G00134000',
                                                                'name': 'Americas',
                                                                'parents': [],
                                                                'type': 'GeoRegion'}],
                                                   'type': 'UNGeoRegion'},
                                                  {'dcid': 'northamerica',
                                                   'name': 'North America',
                                                   'parents': [{'dcid': 'Earth',
                                                                'name': 'World',
                                                                'parents': [],
                                                                'type': 'Place'}],
                                                   'type': 'Continent'},
                                                  {'dcid': 'undata-geo/G00134000',
                                                   'name': 'Americas',
                                                   'parents': [{'dcid': 'Earth',
                                                                'name': 'World',
                                                                'parents': [],
                                                                'type': 'Place'}],
                                                   'type': 'GeoRegion'}],
                                      'type': 'Country'}],
                         'type': None}}

@jm-rivera jm-rivera force-pushed the add-fetch-ancestry branch from 45316c5 to 8fccc33 Compare April 1, 2025 14:32
@jm-rivera jm-rivera requested a review from dwnoble April 4, 2025 06:23
@jm-rivera
Copy link
Contributor Author

@dwnoble ready for a new review. Thank you!!

Copy link
Contributor

@dwnoble dwnoble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jm-rivera!

@jm-rivera jm-rivera merged commit be402cc into datacommonsorg:master Apr 7, 2025
2 checks passed
@jm-rivera jm-rivera deleted the add-fetch-ancestry branch April 7, 2025 19:15
@jm-rivera
Copy link
Contributor Author

FYI @kmoscoe in case it's helpful for the docs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants