Tag listing doesn't allow for tag limit

[jcarter3]

The Tags method doesn't allow for passing a limit, as defined in the spec. There are some registry implementations of this method that would benefit from knowing the limit (setting a limit in SQL, knowing how much to page in a NoSQL, etc).

Was omitting this a conscious decision to just assume a 1000 max to simplify things? Or was it missed and can be added?

[rogpeppe]

The reason for omitting the limit from that method is that the method signature isn't page-oriented: the interface implementation is responsible for using whatever paging might be appropriate under the hood and delivering all the results in the iterator. From this perspective the HTTP API is just one implementation of that top level protocol and thus the fact that it's using paging under the hood is just an implementation detail, hence this is configurable along with the other HTTP-specific options, in https://pkg.go.dev/cuelabs.dev/go/oci/ociregistry/ociclient#Options

It's true that this isn't quite as general - there's no way to specify a particular limit for an individual Tags call, but I couldn't think of a good use case for that. If we did need that functionality, I think that the best way to do it might be to have kind of value in the context, essentially allowing the client to reach through the layers and configure the HTTP layer individually.

I'd be interested in your take on whether being able to configure different limits for different Tags calls on the same connection is useful/essential functionality.

[jcarter3]

I'm trying to decide that for myself 😄 The context here is that I'm working on building a new registry implementation to experiment with and was trying to leverage these interfaces for the storage/data layer so I could reuse the memory implementation and whatnot. Largely, I find the abstractions pretty good, though I have to admit I haven't used iterators much before.

From my experience on Docker Hub, and with what I'm doing currently, I've done tag listing via:

• S3/object store paging
• RDBMS queries
• NoSQL queries (dynamodb)

In all three, while the page size may not be necessary, it can be helpful.

• S3 (maybe others?) are limited to a max of 1000 keys. It might be safe to always pull back the max, but if only 100 are needed, we're just doing extra work
• Postgres wouldn't have a limit like S3, but if a repo has 100,000 tags (or we have some with millions), we don't want to return all of them either. So again, maybe we only return 1000 and it's up to the caller to request again. But if only 100 are wanted, we're doing extra work.
• DynamoDB would also have a limit on response size.

Maybe this is more a question of who is responsible for the paging 🤔 If a registry implementation gets a request for 1000 tags, but the implementation of this interface only returns 100, does it request 10x to get the full 1000 or just return the 100 and tell the client to ask for more? I could probably get on board with just saying implementations should return a sane (non configurable) number (1000?) and make the caller adapt.

A lot of words, maybe I talked myself into thinking it's okay to leave it out - in the worst case we query more data than necessary and it just gets dropped. But then there also isn't a clear mechanism for signaling we're "done" vs "max results".

[tianon]

I think if this isn't exposed explicitly you end up with two abstraction layers both doing paging and ending up with weird behaviors/performance characteristics when they don't match, right?

I think @jonjohnsonjr had some thoughts here, maybe we can coax him into chiming in 👀🫶