You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, all Aggregating Functions are storing aggregated results in memory.
This is not fully aligned with Flow core philosophy which says that Flow is supposed to be memory efficient in the first place.
To solve that problem we should introduce a dedicated storage for mixed results of aggregating functions into which they can easily read/write to.
It should be configurable, meaning that end user should be able to chose if he wants to aggregate values in memory (super fast but not scalable) or to use one of the available storage strategies.
We can approach it in the similar way to GroupBy Sort:
when memory consumption < X - use memory
when memory consumption > X - use cache
We might be better using dedicated storage rather than Cache which is optimized for Rows. In this case, Redis-based storage might be a very good solution.
The text was updated successfully, but these errors were encountered:
Currently, all Aggregating Functions are storing aggregated results in memory.
This is not fully aligned with Flow core philosophy which says that Flow is supposed to be memory efficient in the first place.
To solve that problem we should introduce a dedicated storage for mixed results of aggregating functions into which they can easily read/write to.
It should be configurable, meaning that end user should be able to chose if he wants to aggregate values in memory (super fast but not scalable) or to use one of the available storage strategies.
We can approach it in the similar way to
GroupBySort:We might be better using dedicated storage rather than Cache which is optimized for Rows. In this case, Redis-based storage might be a very good solution.
The text was updated successfully, but these errors were encountered: