Download vs Data explorer preview differences in record counts

[Uchechukwu-Onye-Igbo]

Hello,

We had an external user point out the following difference in record counts which is usually only a difference of 100k or so:

https://opendata.nhsbsa.net/dataset/english-prescribing-data-epd/resource/6fe3818a-bff7-43d6-8ce5-655c3104f262 (EPD WITHOUT SNOMED)

https://opendata.nhsbsa.net/dataset/english-prescribing-dataset-epd-with-snomed-code/resource/36c67aba-0ba4-402e-9182-f30a8cd9b138 (EPD with SNOMED)

I discussed this with our DW team and they pointed out that when downloaded the EPD with SNOMED file had 18,029,396 rows. I therefore checked the datastore page and this was showing as upload to datastore completed without issue two years ago – I clicked upload to datastore again just to see if this would make a difference and now this is showing:

I know we have checked for issues with pending upload to datastore files but how are we meant to identify instances wherein the above is happening?

Downloads are giving different values than compared to the data explorer preview – this again would need to be identified, rectified and communicated out.

Thank you so much,

[NHSBSA-Open-Data]

2026-01-15 18:24GMT @luccasmmg wrote via email:

Hi Kate, two thoughts

1 - Does the DW team knows how these files were uploaded? its possible that due to how big they were someone uploaded them directly to S3 bypassing CKAN which caused the system to not be aware that there was a change and therefore not trigger datapusher
2 - I will try thinking harder on this, but from a first idea would be to go to s3 check the last time a file was updated and see if it matches the last updated in the given resource, it would be great if you knew about any other files were this was showing up, just so i could manually check this quickly and validate this idea

Regards, Luccas

[NHSBSA-Open-Data]

@luccasmmg I have queried with DW but not sure if they would be able to answer this - the activity stream on ODP only goes back to March 2025, so I couldn't see if there were any strange goings on using that either.

I have not identified any further files where this may be happening, as of yet, sorry.

Thank you,
Kate

[Uchechukwu-Onye-Igbo]

hi @luccasmmg i forwarded you the email

[NHSBSA-Open-Data]

@luccasmmg Is there any update on this please?

Thank you,
Kate

[luccasmmg]

@NHSBSA-Open-Data an update of what i did

1. I grabbed every resource in the data portal
2. I went into the r2 bucket directly, and checked if the last_updated date in the bucket was at least 24 hours after the CKAN last_updated
3. That returned me 1280 results
4. So i had a script to download the file directly on r2 count the number of lines on it and compare to select count(*) from {resource id} on bigquery
5. This returned me a list of resources which im sending on gchat
6. Besides that i found a list of files that are not present on the datastore (Meaning that there is no mismatch but no preview)

Im sending that on gchat as well

What im currently now is trying to see if i run step 4 (Downloading every file directly on r2) im getting extra results