Rule eval result caching to improve Ruler performance

[alvinlin123]

Is your feature request related to a problem? Please describe.
It's not related to a problem, but I think I would like rule groups to be evaluated faster.

Describe the solution you'd like
For a rule group with 2 rules, A, and B; B depends on A. If we cache the result of A, then B can used the cached result rather than waiting for A's result to be written to ingester then read from ingester.

This will be particularly useful when A produces a lot of time series where writing and querying ingestr will take time.

Describe alternatives you've considered
No. Didn't think of other alternatives. Of course concurrent_evals_enabled would boost performance, be rule eval result caching would boost the performance further.

Additional context
No.

[friedrichg]

Have you tried query-frontend usage from ruler? There are good reports from the community

I like your suggestion and at the same time I am not sure. This can generate inconsistencies if the series from A end up getting rejected by the ingesters.

Also there is concurrent_evals, which I have tried concurrent_evals, but it only works In a single ruler pod. A better concurrent_evals, one that scales one group to many pods, is much needed and a bigger problem.

What should we prioritize? low latency or horizontal scalability?