Remove code related to http caching (#644)

We had implemented our own http caching in #88 when nothing like that
existed yet. Now there are two different threads that are implementing
something like this in a better way than we have done. So this PR
removes our custom caching code from our codebase. Instead in a future
release we'll make sure that pg_duckdb works well with these at least
one of these other caching solutions.

For reference the two projects that are currently implementing caching
are:
1. An official in-memory cache by DuckDB, which will eventually improved
to also support on-disk caching:
duckdb/duckdb#16463
2. A community provided extension called `cache_httpfs`:
https://github.com/dentiny/duck-read-cache-fs

---------

Co-authored-by: Yves <yves@motherduck.com>

Author: JelteF

docs/extensions.md

@@ -2,7 +2,7 @@
 
 The following extensions are installed by default:
 
-* httpfs - note that httpfs was forked to add [`duckdb.cache`](functions.md#cache)
+* httpfs
 * json
 
 Supported extensions for installation are:
@@ -16,7 +16,7 @@ Installing other extensions may work, but is at your own risk.
 
 By default known extensions are allowed to be automatically installed and loaded when a DuckDB query depends on them. This behaviour can be configured using the [`duckdb.autoinstall_known_extensions`](settings.md#duckdbautoinstall_known_extensions) and [`duckdb.autoload_known_extensions`](settings.md#duckdbautoload_known_extensions) settings.
 
-It's also possible to manually install an extension. This can be useful when this autoinstall/autoaload behaviour is disabled, or when DuckDB fails to realise an extension is necessary to execute the query. Installing an extension requires superuser.
+It's also possible to manually install an extension. This can be useful when this autoinstall/autoload behaviour is disabled, or when DuckDB fails to realise an extension is necessary to execute the query. Installing an extension requires superuser.
 
 ```sql
 SELECT duckdb.install_extension('extname');

docs/functions.md

@@ -26,14 +26,6 @@ All of the DuckDB [json functions and aggregates](https://duckdb.org/docs/data/j
 | :--- | :---------- |
 |[`approx_count_distinct`](https://duckdb.org/docs/sql/functions/aggregates.html#approximate-aggregates)|Gives the approximate count of distinct elements using HyperLogLog|
 
-## Cache Management Functions
-
-| Name | Description |
-| :--- | :---------- |
-| [`duckdb.cache`](#cache) | Caches a Parquet or CSV file to disk |
-| [`duckdb.cache_info`](#cache_info) | Returns metadata about cached files |
-| [`duckdb.cache_delete`](#cache_delete) | Deletes a file from the cache |
-
 ## DuckDB Administration Functions
 
 | Name | Description |
@@ -257,48 +249,6 @@ Further information:
 | :--- | :--- | :---------- |
 | path | text | The path, either to a remote httpfs location or a local location (if enabled) of the delta dataset to read. |
 
-#### <a name="cache"></a>`duckdb.cache(path TEXT, type TEXT) -> bool`
-
-Caches a parquet or CSV file to local disk. The file is downloaded synchronously during the execution of the function. Once cached, the cached file is used automatically whenever that URL is used in other httpfs calls, provided that the remote data has not changed. Data is stored based on the eTag of the remote file.
-
-Note that cache management is not automated. Cached data must be deleted manually.
-
-##### Required Arguments
-
-| Name | Type | Description |
-| :--- | :--- | :---------- |
-| path | text | The path to a remote httpfs location to cache. |
-| type | text | File type, either `parquet` or `csv` |
-
-#### <a name="cache_info"></a>`duckdb.cache_info() -> (remote_path text, cache_key text, cache_file_size BIGINT, cache_file_timestamp TIMESTAMPTZ)`
-
-Inspects which remote files are currently cached in DuckDB. The returned data is as follows:
-
-| Name | Type | Description |
-| :--- | :--- | :---------- |
-| remote_path | text | The original path to the remote httpfs location that was cached |
-| cache_key | text | The cache key (eTag) used to store the file |
-| cache_file_size | bigint | File size in bytes |
-| cache_file_timestamp | timestamptz | Creation time of the cached file |
-
-#### <a name="cache_delete"></a>`duckdb.cache_delete(cache_key text) -> bool`
-
-Deletes a file from the DuckDB cache using the `unique cache_key` of the file.
-
-Example: To delete any copies of a particular remote file:
-
-```sql
-SELECT duckdb.cache_delete(cache_key)
-FROM duckdb.cache_info()
-WHERE remote_path = '...';
-```
-
-##### Required Arguments
-
-| Name | Type | Description |
-| :--- | :--- | :---------- |
-| cache_key | text | The cache key (eTag) to delete |
-
 #### <a name="install_extension"></a>`duckdb.install_extension(extension_name TEXT, repository TEXT DEFAULT 'core') -> bool`
 
 Installs a DuckDB extension and configures it to be loaded automatically in

sql/pg_duckdb--0.3.0--0.4.0.sql

@@ -1,6 +1,10 @@
 -- Add "url_style" column to "secrets" table
 ALTER TABLE duckdb.secrets ADD COLUMN url_style TEXT;
 
+DROP FUNCTION duckdb.cache_delete;
+DROP FUNCTION duckdb.cache_info;
+DROP FUNCTION duckdb.cache;
+DROP TYPE duckdb.cache_info;
 
 DROP FUNCTION duckdb.install_extension(TEXT);
 CREATE FUNCTION duckdb.install_extension(extension_name TEXT, source TEXT DEFAULT 'core') RETURNS void

src/pgduckdb_duckdb.cpp

@@ -287,19 +287,6 @@ DuckDBManager::LoadExtensions(duckdb::ClientContext &context) {
 			continue;
 		}
 
-		/*
-		 * Skip the httpfs extension. It conflicts with our cached_httpfs
-		 * extension which is loaded by default, but people might still try to
-		 * install it manually. We could also throw an error by doing so, but
-		 * it seems nicer to just skip it, because it is clear what they meant
-		 * to do and this way we don't need to educate people about the
-		 * existence of our cached_httpfs extension (which might be removed in
-		 * the near future anyway).
-		 */
-		if (extension.name == "httpfs") {
-			continue;
-		}
-
 		DuckDBQueryOrThrow(context, "LOAD " + extension.name);
 	}
 }
@@ -319,10 +306,6 @@ DuckDBManager::RefreshConnectionState(duckdb::ClientContext &context) {
 		UpdateSecretSeq(secret_table_last_seq);
 	}
 
-	auto http_file_cache_set_dir_query =
-	    duckdb::StringUtil::Format("SET http_file_cache_dir TO '%s';", CreateOrGetDirectoryPath("duckdb_cache"));
-	DuckDBQueryOrThrow(context, http_file_cache_set_dir_query);
-
 	if (duckdb_disabled_filesystems != NULL && !superuser()) {
 		/*
 		 * DuckDB does not allow us to disable this setting on the

src/pgduckdb_options.cpp

@@ -262,87 +262,6 @@ DuckdbInstallExtension(const std::string &extension_name, const std::string &rep
 	SPI_finish();
 }
 
-static bool
-CanCacheRemoteObject(std::string remote_object) {
-	return (remote_object.rfind("https://", 0) == 0) || (remote_object.rfind("http://", 0) == 0) ||
-	       (remote_object.rfind("s3://", 0) == 0) || (remote_object.rfind("s3a://", 0) == 0) ||
-	       (remote_object.rfind("s3n://", 0) == 0) || (remote_object.rfind("gcs://", 0) == 0) ||
-	       (remote_object.rfind("gs://", 0) == 0) || (remote_object.rfind("r2://", 0) == 0);
-}
-
-static bool
-DuckdbCacheObject(Datum object, Datum type) {
-	auto object_path = DatumToString(object);
-	if (!CanCacheRemoteObject(object_path)) {
-		elog(WARNING, "(PGDuckDB/DuckdbCacheObject) Object path '%s' can't be cached.", object_path.c_str());
-		return false;
-	}
-
-	auto object_type = DatumToString(type);
-	if (object_type != "parquet" && object_type != "csv") {
-		elog(WARNING, "(PGDuckDB/DuckdbCacheObject) Cache object type should be 'parquet' or 'csv'.");
-		return false;
-	}
-
-	/* Use a separate connection to cache the objects, so we are sure not to
-	 * leak the value change of enable_http_file_cache in case of an error */
-	auto con = DuckDBManager::CreateConnection();
-	auto &context = *con->context;
-
-	DuckDBQueryOrThrow(context, "SET enable_http_file_cache TO true;");
-
-	auto object_type_fun = object_type == "parquet" ? "read_parquet" : "read_csv";
-	auto cache_object_query =
-	    duckdb::StringUtil::Format("SELECT 1 FROM %s('%s');", object_type_fun, object_path.c_str());
-	DuckDBQueryOrThrow(context, cache_object_query);
-
-	return true;
-}
-
-typedef struct CacheFileInfo {
-	std::string cache_key;
-	std::string remote_path;
-	int64_t file_size;
-	Timestamp file_timestamp;
-} CacheFileInfo;
-
-static std::vector<CacheFileInfo>
-DuckdbGetCachedFilesInfos() {
-	std::string ext(".meta");
-	std::vector<CacheFileInfo> cache_info;
-	for (auto &p : std::filesystem::recursive_directory_iterator(CreateOrGetDirectoryPath("duckdb_cache"))) {
-		if (p.path().extension() == ext) {
-			std::ifstream cache_file_metadata(p.path());
-			std::string metadata;
-			std::vector<std::string> metadata_tokens;
-			while (std::getline(cache_file_metadata, metadata, ',')) {
-				metadata_tokens.push_back(metadata);
-			}
-			if (metadata_tokens.size() != 4) {
-				elog(WARNING, "(PGDuckDB/DuckdbGetCachedFilesInfos) Invalid '%s' cache metadata file",
-				     p.path().c_str());
-				break;
-			}
-			cache_info.push_back(CacheFileInfo {metadata_tokens[0], metadata_tokens[1], std::stoll(metadata_tokens[2]),
-			                                    std::stoll(metadata_tokens[3])});
-		}
-	}
-	return cache_info;
-}
-
-static bool
-DuckdbCacheDelete(Datum cache_key_datum) {
-	auto cache_key = DatumToString(cache_key_datum);
-	if (!cache_key.size()) {
-		elog(WARNING, "(PGDuckDB/DuckdbGetCachedFilesInfos) Empty cache key");
-		return false;
-	}
-	auto cache_filename = CreateOrGetDirectoryPath("duckdb_cache") + "/" + cache_key;
-	bool removed = !std::remove(cache_filename.c_str());
-	std::remove(std::string(cache_filename + ".meta").c_str());
-	return removed;
-}
-
 namespace pg {
 
 std::string
@@ -377,59 +296,23 @@ DECLARE_PG_FUNCTION(pgduckdb_raw_query) {
 	PG_RETURN_BOOL(true);
 }
 
+/*
+ * We need these dummy cache functions so that people are able to load the
+ * new pg_duckdb.so file with an old SQL version (where these functions still
+ * exist). People should then upgrade the SQL part of the extension using the
+ * command described in the error message. Once we believe no-one is on these old
+ * version anymore we can remove these dummy functions.
+ */
 DECLARE_PG_FUNCTION(cache) {
-	Datum object = PG_GETARG_DATUM(0);
-	Datum type = PG_GETARG_DATUM(1);
-	bool result = pgduckdb::DuckdbCacheObject(object, type);
-	PG_RETURN_BOOL(result);
+	elog(ERROR, "duckdb.cache is not supported anymore, please run 'ALTER EXTENSION pg_duckdb UPDATE'");
 }
 
 DECLARE_PG_FUNCTION(cache_info) {
-	pgduckdb::RequireDuckdbExecution();
-	ReturnSetInfo *rsinfo = (ReturnSetInfo *)fcinfo->resultinfo;
-	Tuplestorestate *tuple_store;
-	TupleDesc cache_info_tuple_desc;
-
-	auto result = pgduckdb::DuckdbGetCachedFilesInfos();
-
-	cache_info_tuple_desc = CreateTemplateTupleDesc(4);
-	TupleDescInitEntry(cache_info_tuple_desc, (AttrNumber)1, "remote_path", TEXTOID, -1, 0);
-	TupleDescInitEntry(cache_info_tuple_desc, (AttrNumber)2, "cache_file_name", TEXTOID, -1, 0);
-	TupleDescInitEntry(cache_info_tuple_desc, (AttrNumber)3, "cache_file_size", INT8OID, -1, 0);
-	TupleDescInitEntry(cache_info_tuple_desc, (AttrNumber)4, "cache_file_timestamp", TIMESTAMPTZOID, -1, 0);
-
-	// We need to switch to long running memory context
-	MemoryContext oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory);
-	tuple_store = tuplestore_begin_heap(rsinfo->allowedModes & SFRM_Materialize_Random, false, work_mem);
-	MemoryContextSwitchTo(oldcontext);
-
-	for (auto &cache_info : result) {
-		Datum values[4] = {0};
-		bool nulls[4] = {0};
-
-		values[0] = CStringGetTextDatum(cache_info.remote_path.c_str());
-		values[1] = CStringGetTextDatum(cache_info.cache_key.c_str());
-		values[2] = cache_info.file_size;
-		/* We have stored timestamp in *seconds* from epoch. We need to convert this to PG timestamptz. */
-		values[3] = (cache_info.file_timestamp * 1000000) - pgduckdb::PGDUCKDB_DUCK_TIMESTAMP_OFFSET;
-
-		HeapTuple tuple = heap_form_tuple(cache_info_tuple_desc, values, nulls);
-		tuplestore_puttuple(tuple_store, tuple);
-		heap_freetuple(tuple);
-	}
-
-	rsinfo->returnMode = SFRM_Materialize;
-	rsinfo->setResult = tuple_store;
-	rsinfo->setDesc = cache_info_tuple_desc;
-
-	return (Datum)0;
+	elog(ERROR, "duckdb.cache_info is not supported anymore, please run 'ALTER EXTENSION pg_duckdb UPDATE'");
 }
 
 DECLARE_PG_FUNCTION(cache_delete) {
-	pgduckdb::RequireDuckdbExecution();
-	Datum cache_key = PG_GETARG_DATUM(0);
-	bool result = pgduckdb::DuckdbCacheDelete(cache_key);
-	PG_RETURN_BOOL(result);
+	elog(ERROR, "duckdb.cache_delete is not supported anymore, please run 'ALTER EXTENSION pg_duckdb UPDATE'");
 }
 
 DECLARE_PG_FUNCTION(pgduckdb_recycle_ddb) {

test/regression/expected/extensions.out

@@ -3,8 +3,8 @@ SET duckdb.allow_community_extensions = true;
 SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, installed, installed_from FROM duckdb_extensions() WHERE loaded and extension_name != 'jemalloc' $$);
  extension_name | loaded | installed | installed_from 
 ----------------+--------+-----------+----------------
- cached_httpfs  | t      | t         | 
  core_functions | t      | t         | 
+ httpfs         | t      | t         | 
  icu            | t      | t         | 
  json           | t      | t         | 
  parquet        | t      | t         | 
@@ -35,8 +35,8 @@ SELECT last_value FROM duckdb.extensions_table_seq;
 SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, installed, installed_from FROM duckdb_extensions() WHERE loaded and extension_name != 'jemalloc' $$);
  extension_name | loaded | installed | installed_from 
 ----------------+--------+-----------+----------------
- cached_httpfs  | t      | t         | 
  core_functions | t      | t         | 
+ httpfs         | t      | t         | 
  icu            | t      | t         | 
  json           | t      | t         | 
  parquet        | t      | t         | 
@@ -61,8 +61,8 @@ SELECT last_value FROM duckdb.extensions_table_seq;
 SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, installed, installed_from FROM duckdb_extensions() WHERE loaded and extension_name != 'jemalloc' $$);
  extension_name | loaded | installed | installed_from 
 ----------------+--------+-----------+----------------
- cached_httpfs  | t      | t         | 
  core_functions | t      | t         | 
+ httpfs         | t      | t         | 
  icu            | t      | t         | 
  json           | t      | t         | 
  parquet        | t      | t         | 
@@ -85,8 +85,8 @@ SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, installed, installe
  extension_name | loaded | installed | installed_from 
 ----------------+--------+-----------+----------------
  aws            | t      | t         | core
- cached_httpfs  | t      | t         | 
  core_functions | t      | t         | 
+ httpfs         | t      | t         | 
  icu            | t      | t         | 
  json           | t      | t         | 
  parquet        | t      | t         | 
@@ -106,8 +106,8 @@ SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, installed, installe
  extension_name | loaded | installed | installed_from 
 ----------------+--------+-----------+----------------
  aws            | t      | t         | core
- cached_httpfs  | t      | t         | 
  core_functions | t      | t         | 
+ httpfs         | t      | t         | 
  icu            | t      | t         | 
  json           | t      | t         | 
  parquet        | t      | t         | 
@@ -130,9 +130,9 @@ SELECT * FROM duckdb.query($$ SELECT extension_name, loaded, installed, installe
  extension_name | loaded | installed | installed_from 
 ----------------+--------+-----------+----------------
  aws            | t      | t         | core
- cached_httpfs  | t      | t         | 
  core_functions | t      | t         | 
  duckpgq        | t      | t         | community
+ httpfs         | t      | t         | 
  icu            | t      | t         | 
  json           | t      | t         | 
  parquet        | t      | t         | 

test/regression/expected/non_superuser.out

@@ -40,10 +40,6 @@ ERROR:  DuckDB execution is not allowed because you have not been granted the du
 SET duckdb.force_execution = false;
 SELECT * FROM duckdb.raw_query($$ SELECT * FROM pgduckdb.public.t $$);
 ERROR:  DuckDB execution is not allowed because you have not been granted the duckdb.postgres_role
-SELECT * FROM duckdb.cache_info();
-ERROR:  DuckDB execution is not allowed because you have not been granted the duckdb.postgres_role
-SELECT * FROM duckdb.cache_delete('some file');
-ERROR:  DuckDB execution is not allowed because you have not been granted the duckdb.postgres_role
 CALL duckdb.recycle_ddb();
 ERROR:  DuckDB execution is not allowed because you have not been granted the duckdb.postgres_role
 SET duckdb.force_execution = true;

test/regression/sql/non_superuser.sql

@@ -32,8 +32,6 @@ SELECT * FROM t;
 -- actually open a duckdb connection.
 SET duckdb.force_execution = false;
 SELECT * FROM duckdb.raw_query($$ SELECT * FROM pgduckdb.public.t $$);
-SELECT * FROM duckdb.cache_info();
-SELECT * FROM duckdb.cache_delete('some file');
 CALL duckdb.recycle_ddb();
 SET duckdb.force_execution = true;