What if I lost my catalog?

[dridk]

Suppose I lost my catalog ( sqlite file) but not the data .
Would it be possible to restore the catalog from data? even partially ?
What about keeping some extra metadata in storage to make possible a catalog restoration ?

[Mytherin]

If your database is not encrypted, you could likely recover most of the data.

• After Add support for schema + table sub-paths, and make data paths relative to the schema + table paths by default #126, data files are written to schema_name/table_name/{files.parquet} - so you know which schemas and tables existed in the catalog - and the files associated with each table.
• Data files are labeled using uuidv7, so you know the data files that have been written and roughly in which order they have been written.
• Data files contain the full schema for the table at the point in time from which they are written, so you can track the schema evolution by following along the schemas by looking at the tables in-order. The most recently written data file likely has the latest schema definition.
• Statistics can be re-read from data files at any point.
• Delete files are written as ducklake-{uuid}-delete.parquet} and have a well-defined structure (filename/row_id). The filename indicates the corresponding data file the deletes apply to - so you can recover which rows were deleted relatively easily.
• By default, partitioned files are written using the hive-partitioning filename convention, so they are written to partition_value=<val>/ subdirectories - this allows you to recover partitioning information.

What is lost is "catalog-only" information, which is:

• Which snapshots made which exact changes
• If files are completely dropped from the table, you would not be aware of that.
• You would not know if data/delete files are "orphan files" or if they were actually committed.
• If data inlining is enabled, any data that is inlined is lost.
• Row-ids that are not explicitly written to the file (i.e. those derived with row_id_start).
• Any views that are defined
• Table/column comments
• Defined NOT NULL constraints

If your data is encrypted - you will have lost all of your data, as the encryption keys are stored in the catalog and the data is not readable without those encryption keys.

Storing the data necessary for a full recovery at all points would effectively require all commits to write all metadata to the data path - which would negate a lot of the benefits of DuckLake. That being said - you could periodically upload snapshots of the DuckLake catalog (e.g. as a DuckDB or SQLite database file) to the data path to serve as back-ups to allow at least "recovery as of a certain time". You could also use any back-up tools available for whichever metadata catalog you are using to back-up your DuckLake catalog (e.g. if you are using Postgres, you can use Postgres' backup utilities).