How to recover from metadata corruption in Ducklake

[guillesd]

If the metadata catalog is corrupted (or information is deleted) and the data files are orphaned, what is the way to proceed?
In #324 it was mentioned that ducklake_add_data_files could be used to re-add files, but this actually doesn't work with files that where previously managed by DuckLake (see reproducer in the issue).

[gacolitti]

The reverse situation is also something that would be great to get official guidance on. Namely, what to do when files tracked by the metadata are lost or corrupted, and the metadata still expects them to exist. Is there any way to recover from this situation?

Is the only option to restore from an earlier backed up metadata that doesn't include references to files that no longer exist as discussed here: https://ducklake.select/docs/stable/duckdb/guides/backups_and_recovery?

[guillesd]

I was actually checking if Iceberg has some guidance on this and they don't. Basically in the case of a missing file, best thing that can happen is that you have some system of recovery in your storage (i.e. s3 backups as specified in the ducklake guide).

Worst case scenario = you don't have this. Then you need to roll back to a snapshot previous to this. If this affects only a table in your Ducklake I would even consider dropping that table and recreating that as to not affect the rest of the lake.