Skip to content
This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Add support for batch object info into ExecutionEnvironments #68

Open
alexbarcelo opened this issue Apr 28, 2020 · 0 comments
Open

Add support for batch object info into ExecutionEnvironments #68

alexbarcelo opened this issue Apr 28, 2020 · 0 comments
Assignees
Labels

Comments

@alexbarcelo
Copy link
Member

Add a new RPC in ExecutionEnvironments for retrieving batch information about the objects.

The current use case for this call is to enable a performant way to retrieve extra information, to be used by the split, and use a single RPC call per ExecutionEnvironment (instead of doing a RPC per object).

This is useful because the split will need information about a bunch of objects (e.g. the split needs to know if the objects are in-memory or have been evicted into disk) and the split can use the object hint to aggregate all the objects into a single batch call to the ExecutionEnvironment.

The proof-of-concept implementation will allow to ask for "is_in_memory", but further metadata may be provided in the future (e.g. replica stuff, versions, memory tier, placement, whtaever).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant