Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage for Aggregating Functions #1390

Open
norberttech opened this issue Jan 18, 2025 · 0 comments
Open

Storage for Aggregating Functions #1390

norberttech opened this issue Jan 18, 2025 · 0 comments

Comments

@norberttech
Copy link
Member

norberttech commented Jan 18, 2025

Currently, all Aggregating Functions are storing aggregated results in memory.
This is not fully aligned with Flow core philosophy which says that Flow is supposed to be memory efficient in the first place.

To solve that problem we should introduce a dedicated storage for mixed results of aggregating functions into which they can easily read/write to.

It should be configurable, meaning that end user should be able to chose if he wants to aggregate values in memory (super fast but not scalable) or to use one of the available storage strategies.

We can approach it in the similar way to GroupBy Sort:

  • when memory consumption < X - use memory
  • when memory consumption > X - use cache

We might be better using dedicated storage rather than Cache which is optimized for Rows. In this case, Redis-based storage might be a very good solution.

@norberttech norberttech moved this to Todo in Roadmap Jan 18, 2025
@norberttech norberttech added this to the 0.12.0 milestone Jan 18, 2025
@norberttech norberttech removed this from the 0.12.0 milestone Jan 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

No branches or pull requests

1 participant