What should I do when some data files are missing?

[minukhwang177]

Hi, I'm using azure blob storage as ducklake's data storage. These are connected with blob fuse and works like native folder.
Sadly sometimes files are gone when pod restarts, and this causes ducklake's merge function fails continously.

I will fix missing file problem, but how can I fix ducklake's metadata? I make some empty parquet file with same name, ducklake was failed because it's fotter length is invalid.

Can I jsut delete rows from metadata table?

[guillesd]

This is very tricky. Corrupt metadata is very difficult to fix

[minukhwang177]

ok, I see. Let me try some actions to fix it.
I think this could happen any time, and do some tricky thing every time is not a good solution.
What do you think about add options to ignore missing files during merge? I know it not a wonderful solution, that all I can think at this moment.