Use Cortex compactor for Thanos receive/store

[yuchen-db]

Is your feature request related to a problem? Please describe.
I have a Thanos stack and face compactor scalability issue for large tenants. I'm considering to replace Thanos compactor with Cortex compactor to leverage its better scalability, as multiple Cortex compactors can be configuerd to work on one tenant. We are exploring the feasibility to mix Cortex compactor with Thanos receive/store.

Describe the solution you'd like
Cortex and Thanos compactor can be used interchangeably.

Describe alternatives you've considered
We have more than 50 Thanos compactor pods in a large cluster, and some of them have more than 200k TODO blocks. We have been manually splitting large tenant to workaround the scalability issue with Thanos compactor, but that's very operational heavy.

Additional context
Each of our Thanos ingester has 3 replications. Our compactor vertically deduplicates replicated blocks, and horizontally merge blocks to longer time range. We disable downsampling. We are running at a scale where single tenant often has too many time series that can't be handled by a single compactor.

[yeya24]

Hey @yuchen-db, thanks for creating the issue. Cortex data layout has slight difference to Thanos as Cortex has multi tenancy by default. Thanos s3 data layout would look like the format below and there is no path for tenant ID.

Thanos data path: s3://<bucket_name>/block_id

Cortex data path: s3://<bucket_name>/<tenant_id>/block_id

Cortex compactor needs to discover all user IDs first and then shard by user ID and create compaction jobs per tenant. User discovery is through UserScanner https://github.com/cortexproject/cortex/blob/master/pkg/storage/tsdb/users/scanner.go#L29. To support Thanos usecase, a new user scanner type with no tenancy can be introduced.

type noTenancyScanner struct{}

func (s *noTenancyScanner) ScanUsers(ctx context.Context) (active, deleting, deleted []string, err error) {
	return []string{""}, nil, nil, nil
}

Other than this, the rest of things should be compatible with Thanos compactor. Please test it out and let us know if there is any other issue observed.

[danielblando]

Cool issue. Interesting to see if this will work well. I never tried.

Another point that you might have an issue is the prefix on s3 as @yeya24 mentioned.

We probably will need to change the bucket creation on

cortex/pkg/compactor/compactor.go

Line 998 in 2fed9b7

bucket := bucket.NewUserBucketClient(userID, c.bucketClient, c.limits)

to avoid thanos to call s3://<bucket_name>/<tenant_id>/

[yeya24]

If we use the noTenancyScanner above to return empty tenant ID I think it is same as s3://<bucket_name>/<block_id> but yeah someone needs to really test it out.

I tried it locally and my local compactor is able to compact against my filesystem bucket. But I never tested in a real thanos deployment