Allow per-message Nats-TTL to be bigger than JetStream Stream's MaxAge property

[nikopavlica]

Proposed change

Either of
A) By allowing Nats-TTL to be bigger than stream's MaxAge.

or
B) Adding a new config to the stream DefaultTTL
This could be used as a default TTL for all new messages that don't have the TTL set - in case existing MaxAge is used as a hard limit on maximum message age.

Use case

When publishing a message for storage into jetstream, we can now either:

• set a Nats-TTL: <duration>, where <duration> that is shorter than stream's configured MaxAge
• set a Nats-TTL: never, to have this message never expire at all, but then handle purging of such message on our own. Which is the exact thing that TTL tries to solve.

It would be useful to have some messages with longer TTL, just as it is useful to have some with Nats-TTL: never.

Main benefit, that I see, are:

• instead of always sending Nats-TTL: <default-shorter-duration> on every message we don't need such long TTL, we could leave the defaults up to stream configuration, and only set on exceptional messages. This is especially helpful in existing streams with active publishers. If we'd like to have some messages with bigger TTL, we'd now first need to update all publishers to send shorter TTL, before updating the stream's config to allow for bigger TTL.
• Reducing confusion around Nats-TTL; ADR-43 doesn't mention anything about limiting the TTL to the stream's max age. And given that it can be set to never, it is really non-intuitive that it can't be longer than MaxAge.

Contribution

I'd be interested to contribute - if time permits :)

[ripienaar]

When we designed this we felt stream operations should have a hard cap of age in streams - as that is often tied to storage usage and such. Hence the MaxAge behaviour, agree though we should add that to the ADR.

I think I still think that is the correct behaviour, the never key is kind of a special case where one might want to store a data schema for a stream in the stream itself, not really for general use. If anything we should add a way to disable / enable the never behaviour for a stream as I think MaxAge should mean just that.

Default TTL should be considered as MaxAge basically

[nikopavlica]

I can understand why it is as such, and that the never case is really a backdoor for another feature.

That's why I think it would also be good to check the B) option - the DefaultTTL in stream's config. Having two seperate would definitely better communicate the intent of each parameter, and why the Nats-TTL cannot be bigger than MaxAge.

It would also solve the prolongation of max-age on existing streams, without necessity of updating all the publishers with previous value..

[ripienaar]

Personally I think adding more options, more if-this-then-that kind of features here is counter productive. MaxAge is default TTL seems sufficient.

Maybe you can explain more why you would want to see MaxAge as different from default TTL?

[nikopavlica]

Thats definitely one way of looking at it. Another is that options give users a freedom to use the software in a way it makes sense for them.

Having different DefautTTL from MaxAge is just so logical from configuration perspective. One is for hard limit, and another is a sane default for majority of messages. Even the name of the config implies different things.

I also think that in a perfect world, finite MaxAge and Nats-TTL: never should result in server rejecting the message. As should where Nats-TTL is bigger than MaxAge.

[ripienaar]

I also think that in a perfect world, finite MaxAge and Nats-TTL: never should result in server rejecting the message. As should where Nats-TTL is bigger than MaxAge.

What would then be the purpose of never in that case?

I think never has a very real and specific case. Imagine you have encoded key value stores - it’s all base64 encoded for complex key names and uses our standard kv codecs - the bucket can be I respected to discover this even at levels like different behaviours for different subject spaces in the bucket. Those hints could not be subject to max age.

Adding a another layer of default max age that isn’t max age when not given - sounds like a application level concern to me

[nikopavlica]

Having a DefaultTTL is indeed a quality-of-life improvement. We can live without it and handle everything in apps, but I thought it might be useful to cover the cases i presented, and (I guess) not too disrupting change on the whole config system.

What would then be the purpose of never in that case?

Purpose of never would then only be useful when MaxAge is not set (ie. its infinity) and there is a DefaultTTL set to some finite number. And it could be used to indicate that this message should never be purged.
But I know that enforcing this now is impossible without making a breaking change for KV configs.

Also, KV has a prerequisite of DiscardPolicy: new, which implies that cleanup based on oldest items never happens when limits are reached. So if you fill the stream with Nats-TTL: never messages, it just won't accept any further messages.

And writing this got me wondering how Nats-TTL: never messages behave when DiscardPolicy: old is set on a stream 🤔 Could you shed some light on this?

• Does it still remove them, since they're older?
• Or does it skip them completely, still allowing them to fill the stream? If this, what happens when stream gets full?

P.S. Thanks for taking the time to discuss this. I really appreciate it.