Add libdeflate gunzip logic to handle gzipped files which decompress to files larger than 4G or which contain multiple concatenated gzip files.

[ghuls]

Add libdeflate gunzip logic to handle gzipped files which decompress to files larger than 4G or which contain multiple concatenated gzip files.

Some gzip files have 0 for the decompressed gzip length or contain multiple gzip files concatenated together so the size hint is only for the last gzip "file" instead of for the whole file.

https://github.com/ebiggers/libdeflate/blob/master/programs/gzip.c#L206-L273

[dcwatson]

I'm not sure what this would look like in the Python API. My goal was to make this a drop-in replacement for https://docs.python.org/3/library/gzip.html#gzip.decompress - it sounds like you want something that would return a list (or preferably a generator) of decompressed data chunks. I'm not totally opposed to adding something like that if you have a working implementation, but probably won't look into it myself.

[dcwatson]

Also I should note, supporting large files is explicitly not a goal of libdeflate itself, so it won't be for this wrapper either. But if someone wants to work on handling small concatenated gzip files, I'd be curious to see what that looks like.