Segments clean-up when messages expire due to TTL

[vaimer]

Describe the bug

RabbitMQ version 4.1.3

raft.wal_max_size_bytes = 64000000
raft.segment_max_entries = 4096

Set-up:

fair.retryTest.1 durable quorum queue with TLL = 16000, after expiration, messages are moved to the deadletter-fair-test durable quorum queue by routing key fair-test.

Issue:

Segments for fair.retryTest.1 aren't cleaned up after moving messages, which causes disk memory leaks.

In some cases, segments might be cleaned up only by queue removal.

Purging messages from deadletter-fair-test doesn't clean up the whole consumed memory.

The same behaviour is not reproducible on the 3.11.3 version.

Reproduction steps

1. Create a queue with TTL, set up a route to another queue
2. Publish a message to the queue with TTL
3. Segments for the queue with TTL aren't cleaned up after the message moves to another queue. (can persist for weeks)

Expected behavior

Segments are cleaned up after moving messages to another queue. (same behaviour as acknowledge)

Additional context

First test case

Publishing messages, approximate size of message 300KB

leader for fair.retryTest.1 memory consumption before and after the publishing process

Almost 150 MB is still in use by the queue and its segments

Second test case

Publishing message, approximate size of message is more than 15MB

The same situation as in the first case, with higher memory consumption.

[michaelklishin]

@vaimer TTL is a massive can of worms and a set of completely different code paths. Some major difference with how expiration is not "an equivalent of acknowledgement" has been documented for years: queues are discarded due to expiration only when they reach the head of the queue, which implies an active consumer.

There is also documentation on quorum queue tuning for large messages.

4.1.0 made segment cleanup significantly more aggressive for the typical case where messages are consumed. I can see how slow segment cleanup in that case could be considered a bug but message expiration per TTL is a completely different beast and I somewhat doubt that our team will get to it before 4.4 or so. It's simply not a common enough operational problem (the common one was addressed in 4.1.0).

That said, RabbitMQ is open source software and you are welcome to investigate what can be done, convince our team that a specific approach is worth its downsides, and submit a PR.

[vaimer]

Hi @michaelklishin thank you for the answer!

We noticed the issue once we migrated from 3.11.3 to 4.1.3. And it was unexpected behaviour for us, which wasn't mentioned anywhere.

Could you please give some advice/answers to the next questions? It will help us investigate further and find an optimal solution

What I found in 3.11.3 was a configuration - queue_explicit_gc_run_operation_threshold. And I suspect, because of it, segments were cleaned up well in the previous version.
I haven't found it in the docs for the new version. Do we have something similar in the new version to force GC?

4.1.0 made segment cleanup significantly more aggressive for the typical case where messages are consumed.

As you said, acknowledgement of the message will work better. We can replace the message TTL queue config with a consumer which will move messages according to their time in the queue, acknowledge them and move to another queue. What's bothering is that there is delivery limit for messages. Did I get it right that we can still set up such queues with unlimited delivery limits? The problem is that it may consume disk space, but since it will be acknowledged in the end, it should be cleaned up.

There is also documentation on quorum queue tuning for large messages.

Yep, I am currently testing different configurations. Just for clarification, did I get it correctly that a lower segment_max_entries number causes the creation of more segment files. And large number of segment files will also trigger clean-up more frequently.
But in general, it should consume more than wal_max_size_bytes.

Thank you very much for help!

[vaimer]

Just a bit more context.
We use queues with TTL for the retry mechanism. Once a message is discarded by an ordinary consumer, it comes to the retry queue with a TTL. After TTL expiration, it tries for more time to be acknowledged in another queue.
And there is a chain of queues with different TTLs.

So there is a trend of memory leaks.

[kjnilsson]

@vaimer

could you run rabbitmqctl eval "persistent_term:put(quorum_queue_checkpoint_config, {500, 256, 666667})." on each of your rabbitmq nodes and rerun your tests?

[vaimer]

Sorry for the confusion, I missed it during copying from my internal notes.

It is a memory once all messages end up in dead letter queue. You can see this step in the provided graphic. Memory jumped to this value, right after processing is done.

I will check out your recommendation, thank you!

[kjnilsson]

if you keep publishing some more message will it eventually reclaim?

[vaimer]

Hi, thank you for your patience, and I apologise for the delayed response.

Yes, the memory will eventually be reclaimed. However, the underlying issue still persists: while messages are cycling between TTL queues, memory consumption increases.

Standard quorum queues also consume memory. We tested multiple configurations for the segment size, but overall, this did not provide significant improvement. What did have a positive impact was setting a delivery limit of 1000 for the qourum queues, which noticeably reduced memory consumption, although the exact reason for this behaviour is not entirely clear.

As a result, we upgraded the cluster to version 4.1.3, and over the past two months, it has remained stable.

[michaelklishin]

FYI, 4.1 goes out of community support this week after 4.1.8 ships.

Relevant changes in the 4.1 and 4.2 series are

• For 4.1.x, by @aaron-seo: introduce a command that would force QQs to take a checkpoint and truncate its segments (backport #13548) #13938 (a CLI command that forces segment cleanup)
• Quorum queues: exclude delivery_limit from policy repair comparison for more frequent segment file cleanup #14458 (the key change that makes segment cleanup significantly more frequent), backported for v4.1.x as Quorum queues: exclude delivery_limit from policy repair comparison for more frequent segment file cleanup (backport #14458) (backport #14504) #14505 and included into more recent patch releases such as the upcoming 4.1.8

[michaelklishin]

Relevant changes in the 4.1 and 4.2 series are

• For 4.1.x, by @aaron-seo: introduce a command that would force QQs to take a checkpoint and truncate its segments (backport #13548) #13938 (a CLI command that forces segment cleanup)
• Quorum queues: exclude delivery_limit from policy repair comparison for more frequent segment file cleanup #14458 (the key change that makes segment cleanup significantly more frequent), backported for v4.1.x as Quorum queues: exclude delivery_limit from policy repair comparison for more frequent segment file cleanup (backport #14458) (backport #14504) #14505 and included into more recent patch releases such as the upcoming 4.1.8