Card duplication detection during import doesn't always work

[TheLastProject]

In #2337, @KlausD-GH mentioned getting duplicates of all cards when importing the same Catima export twice. I've been able to confirm this with an export they sent on the latest release.

I have not yet had the time to research what exactly causes this, but my guess would be that it is related to images (as this is less commonly tested part of deduplication).

Low severity (as duplicates aren't data loss, but a bit annoying) and low commonness (as most users won't import the same import twice). But fixing this would still be a good thing to do.

(Note to self: this export can be found back in a mail sent on February 13, 2025)

[gar-onn]

Should it be an idea to add an uuid to every card so that import-export duplicates can be managed more easily?

[TheLastProject]

I don't think that will really help this case.

The issue here is specifically: the code incorrectly fails to be able to tell that a local card and a card in the backup file are the same when they are. Simply ignoring any card in the backup with the same ID would just bring back bug #512 in full glory.

A more complex UUID (like 6a0649a2-3089-4208-8544-a1126adff8cb or whatever) would help differentiate cards if they're created on different devices, to help in syncing cases like #791, but I don't think it will help with this specific issue.

[rndph]

I am a new user to Catima and was surprised that importing a just created export with images results in duplicates, even though #1422 states that it should not.

Here is a simple test export with one card and an image:
catima.zip

Just import it twice and it results in the same card being there twice.
Tested with Catima version 2.34.5 (installed via Google Play Store).

[TheLastProject]

Thanks for providing an affected export publicly so others could work on this issue too :)

[TakFog]

Does anyone already verified if the problem is the image compression done during the different steps of the process? I noticed that a compression is done while exporting (in the bitmapToByteArray) and also while saving (in saveCardImage) and these recompressions may change the checksum.
At the end, the image in the zip is neither the original one nor the one in the app after the import.
I don't have an android environment available atm so I can't check if I'm right.

[TheLastProject]

I think we concluded the image compression is most likely the issue. I at least can't trigger this bug in an export without images, yet the small test case with image shared before triggers it reliably

[matuspetro]

Hello @TheLastProject , I would like to be assigned to this issue.
During testing, I discovered that while comparing images, imported bitmaps are always null. So function not return that imported card as duplicate. Problem is that images are loaded at a later stage, after solving duplicates.

I would therefore suggest discarding image comparison and detecting duplicates based on other information. Then, update the images from the new import. Or its possible to add some popup to let user decide.

[TheLastProject]

Nice find @matuspetro

I would therefore suggest discarding image comparison and detecting duplicates based on other information.

Well, what other information would allow you to know what's a duplicate? And what is the correct, new version?

Then, update the images from the new import.

It seems dangerous to assume the data in the import is always newer, we want to prevent data loss.

Or its possible to add some popup to let user decide.

I'd rather not complicate the import logic as it is, as it's extremely deprecated: #513. If we have a working background import, that would be nice yeah.

So for now it's probably best to see if we can load the images first before comparing.

[matuspetro]

Okej, I will fix it. Can you assigned me to this issue?

[TheLastProject]

Assigned (didn't see your edit before)