Feature request: make cached httpfs a separate extension

[dentiny]

It's known httpfs has certain limitations, for example

• for caching, it only supports full file download, instead of block-based, which might account for large chunk of memory and it doesn't offer a way to limit memory consumption upper bound
• the download operation is synchronous and full-file-based, which increases query latency
• duckdb is an embedded database rather than a client-server architecture, so the cache cannot be reused across different processes

You choose to copy the file into the main repo, instead of using it as a standalone dependency; my groundlessly guess you plan to make changes to the httpfs source code itself, instead of having a cache wrapper around, which makes sense to me.

So I'm wondering would you consider making it it a separate extension, so it benefits the whole community rather than just pg_duckdb.

[dpxcc]

There have been previous discussions about including cached_httpfs as a feature in the main httpfs extension: #88 (comment)