Add content reader to decouple NFS file reads [RHELDST-26339] #643
Conversation
|
@rohanpm Can you please review whether this is on expected lines? |
There was a problem hiding this comment.
I think currently this is a bit more complicated than what the issue was asking for. It looks like it can be simplified.
Although the issue is not asking for this yet, if I were working on this I would probably hack one of the Source backends to actually start using HTTP requests for fetching content. Just as a local change, not something to be pushed. The koji backend may be a good candidate, since koji is designed to serve up content via HTTP.
Doing that would help ensure that the new API is genuinely usable for the intended purpose.
rajulkumar
force-pushed
the
decouple_NFS
branch
from
568444e to
bf47408
Compare
rajulkumar
force-pushed
the
decouple_NFS
branch
from
bf47408 to
edb8fe8
Compare
src/pushsource/_impl/model/base.py
Outdated
| :class:`~io.BufferedReader` | ||
| A non-seekable object of the push item content | ||
| """ | ||
| return PushItemReader(self.opener(self)) |
There was a problem hiding this comment.
As mentioned in my proposed doc addition, I think we need to make this also support a "return None" case, given there are many types of push items where this doesn't make sense.
My suggestion is that opener should be allowed to be None. If it is None, then content should also return None. Then set up non-openable types such as ContainerPushItem to initialize opener to None.
One thing I'm not sure of: should opener on the base class default to open_src_local and then be overridden to None in non-openable types? Or should it default to None in the base class and then be overridden to open_src_local in openable types?
At the moment, I'm leaning towards saying it should default to None in the base class.
The reason is that, when someone introduces a new PushItem subclass, I think it makes sense for the default behavior to be "don't support reading - unless the developer thinks about it and implements it". If it instead defaults to open_src_local on the base class then the default behavior will be "try to read, and crash if that doesn't make sense". This will probably lead to PushItem subclasses where content doesn't make sense but it has not been gracefully overriden to return None.
There was a problem hiding this comment.
Agree. Default as None and overriding it for specific case sounds right.
I have updated the PushItems accordingly, only ones that are used in pubtools-pulp within the scope of this change.
rajulkumar
force-pushed
the
decouple_NFS
branch
from
bc500f0 to
085e9b0
Compare
Reading content of the push items strongly depends on their presence as file on NFS or locally-mounted filesystem. However, with advent of new push item sources, it likely that the push items might not be stored as a file or be locally available. Hence, this PR adds an `opener` to the pushitems that should fetch the corresponding bit stream which will be accessible from `content()`. `opener` should be defined in the `Source` while creating the pushitems to get the bits from the specific content source. PushItem is not bound to have an `opener` and `content()` will return the bits as file object only when the `opener` is defined. This adds a provision to transfer the responsibility of fetching the content from the consumer that is using the content/pushitems to the `Source` and corresponding `pushitem` here, abstracting the reading mechanism from the user and provide flexibility to fetch from different locations/protocols.
rajulkumar
force-pushed
the
decouple_NFS
branch
from
085e9b0 to
fab5c48
Compare
The API to read content of the push items strongly depends on their presence as file on NFS or locally-mounted filesystem. However, with advent of new push item sources, it likely that the push items might not be stored as a file or be locally available.
Hence, add a new content reader interface that will be implemented by content readers based on the source or content type. The readers can then be used by specific push item types to get their bits from the respective source.
Here, the File content reader implements the interface and is used by default for all push items. This reader reads the bits from the path provided as push item source and returns a file-like object as push item content attribute.