Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Query-Frontend] Add dynamic query vertical sharding #6678

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

afhassan
Copy link
Contributor

@afhassan afhassan commented Mar 28, 2025

What this PR does:

  • Refactor dynamic query splitting to reduce complexity
    • The main source of complexity in the previous implementation was the variability of the first split duration compared to other subsequent splits when calculating the total duration of data that will be fetched from storage by the query.
    • The solution is to ignore the first interval when calculating the duration fetched. This means that the duration fetched becomes an estimate, but the impact of ignoring the first interval is small compared to the reduced complexity.
  • Add dynamic query vertical sharding
    • The goal of dynamic query splitting is to protect store-gateways by limiting the number of read requests sent per query, while providing enough parallelism for the query to execute efficiently
    • In some scenarios, a smaller vertical shard size allows more horizontal splitting while keeping store-gateways protected. This allows more efficient query execution.
    • The new change calculates the most optimal combination of vertical sharding and horizontal spliting to result in the largest number of total shards (without exceeding any of the configured limits)

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@afhassan afhassan changed the title [Query-Frontend] Refactor dynamic query splitting to reduce complexity [Query-Frontend] Add dynamic query vertical sharding Apr 3, 2025
@afhassan afhassan marked this pull request as ready for review April 10, 2025 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant