Gaps in Cortex Metrics in Grafana Dashboards for Time Intervals longer than 6 Hours

[alexinfoblox]

We are running Cortex in a dedicated EKS cluster.
More than 70 other clusters send their metrics to this Cortex instance.
Each cluster’s Grafana is configured to query Cortex for data visualization.

For the past couple of months, we have been observing gaps in Grafana panels for time ranges longer than 6 hours (this only has been observed for our biggest tenant - around 27.7 Mil series).

There are no missing metrics — all data is successfully received by Cortex.
It appears the issue is related to metrics caching.
We’ve noticed that restarting the Memcached frontend resolves the problem temporarily — after the restart, the gaps disappear.

Memcached-fronted config:

    query_range:
      cache_results: true
      results_cache:
        cache:
          memcached_client:
            host: cortex-infra-memcached-frontend.cortex-infra.svc.cluster.local
            timeout: 3s
            max_idle_conns: 200

[alexinfoblox]

Hi @yeya24, could you please review

[yeya24]

@alexinfoblox Have you tried bypassing query cache. For example, query querier instead of query frontend or add Cache-Control: no-store header to your query. Would it fix issue?

If this fixes the issue then it is query cache problem. In which Cortex version you see this issue and which version works for you?

[alexinfoblox]

@yeya24 , Since we only see this issue in our prod cluster, we can’t test it by bypassing the query cache.
We tried the second option you suggested — adding Cache-Control: no-store to the header — but it had no effect.
The gaps still appear.

[yeya24]

@alexinfoblox In your prod environment you can still create a Grafana data source with that header configured to not impact your normal queries.

But yeah if adding this header didn't help then I don't think it is a query cache related issue. How would restarting cache help in this case