Migrate image/oldimage queries to file/filerevision/filetypes

[lgelauff]

Summary

Wikimedia is removing the image and oldimage tables from wikireplicas on 28 May 2026 (cloud-announce, T28741). This PR rewrites the two affected query functions in labs.py to use the new file, filerevision, and filetypes tables before that deadline.

Closes #504.

What changed

• labs.py: get_files() and get_file_info() rewritten to use file/filerevision/filetypes. All output key aliases (img_*, oi_archive_name, rec_img_*) preserved — no downstream changes required. Categorylinks query updated to use linktarget join (same fix as Fix categorylinks query for MediaWiki link target normalization #469).
• rdb.py: Added nullable file_id (BIGINT) column to the Entry model.
• loaders.py: make_entry() now stores file_id on new entries and uses .get() or None for MIME fields to handle NULL gracefully.
• conftest.py: Fixture dicts updated with file_id and oi_archive_name fields to match real labs.py output. Added REUPLOAD_FILE_INFO fixture.
• New tests: test_make_entry_reupload, test_get_files_parity (xfail — Toolforge only), test_get_files_info_by_name.
• tools/migrate_beta_db.py: Atomic migration script for the beta SQLite DB (adds file_id column + index).
• tools/revert_beta_db.py: Revert script using the SQLite 12-step schema-alteration procedure for full safety.

Implementation notes

GROUP BY non-determinism fixed: The old query used GROUP BY img_name with ORDER BY oi_timestamp ASC, which was non-deterministic when a file had multiple oldimage rows. Replaced with a correlated MIN(fr_id) subquery that deterministically selects the earliest non-deleted revision per file (original uploader / original timestamp).

Regression fix — suppressed latest revision: file.file_latest can point to a suppressed revision in filerevision (unlike the old image table which always held live data). The main JOIN therefore includes AND fr.fr_deleted = 0 to skip suppressed revisions.

get_files_legacy() is intentionally kept: A verbatim copy of the old image/oldimage query, retained solely to power test_get_files_parity (marked xfail locally, passes on Toolforge with TOOLFORGE=1). While both table sets are live until 28 May, this test asserts the new query returns the same filenames as the old one. Do not remove get_files_legacy() in this PR — it and the parity test are removed together in a follow-up after 28 May.

MIME NULL handling: The new LEFT JOIN filetypes can theoretically return NULL for img_major_mime/img_minor_mime if file.file_type has no match in filetypes. Files with unknown MIME are kept and stored as NULL rather than silently dropped. This surfaces a pre-existing bug in autodisqualify_by_filetype() (out of scope, tracked as follow-up).

Index strategy: Toolforge runs MariaDB 10.6.22, which does not support partial unique indexes. The migration therefore creates a plain non-unique index on file_id; import-time uniqueness is enforced at the application layer.

Migration

Beta uses SQLite; prod uses MySQL/MariaDB. Run before deploying.

Beta (SQLite) — use the included script:

python3 tools/migrate_beta_db.py

Prod (MySQL/MariaDB):

ALTER TABLE entries ADD COLUMN file_id BIGINT DEFAULT NULL;
ALTER TABLE entries ADD INDEX ix_entry_file_id (file_id);

A revert script (tools/revert_beta_db.py) is included for the beta SQLite DB. Both scripts are idempotent and tested locally (roundtrip: pre-migration schema/indexes/data identical to post-revert).

Test plan

• pytest montage/tests/ — 8 passed, 1 xfailed
• Standalone wikireplica query test (tools/test_labs_queries_pr505.py) — all checks passed on Toolforge beta (parity, row shape, file_id populated, attribution for reuploaded file verified)
• Migration script applied to montage-beta DB — succeeded
• montage-beta deployed and running on PR branch
• Run a category import on beta and verify file_id is populated on newly imported entries
• Verify original uploader is correct for a known reuploaded file via the UI
• Run parity test on Toolforge: tools/test_labs_queries_pr505.py — all checks passed (11866 files, parity confirmed, attribution verified)

---

Planning pipeline modelled after work by @alissayarmantho in ashishg/dp#1966 — thank you Alissa.

🤖 Generated with Claude Code

[lgelauff]

Deployment notes for reviewers

Separate PR needed: setuptools pin

This PR branch includes a setuptools==81.0.0 pin in requirements.txt that is not related to the migration — it was added to fix a Python 3.13 Toolforge deployment issue (pkg_resources was removed in setuptools 82). This should be extracted into a separate PR targeting master before or alongside this one, and removed once python-graph-core is replaced (tracked in #421).

---

Production deployment steps

Run these in order when deploying to prod (montage, not montage-beta):

1. Check prod is not in active use

   https://montage.toolforge.org/v1/logs/audit
   

   Check create_date on recent entries — proceed only if no active campaigns.

2. SSH into prod

   ssh <username>@login.toolforge.org
   become montage

3. Pull the new code

   cd ~/www/python/src
   git pull origin master

4. Run the DB migration (MySQL/MariaDB — run before restarting the service)

   sql local

   USE s52584__montage;
   ALTER TABLE entries ADD COLUMN file_id BIGINT DEFAULT NULL;
   ALTER TABLE entries ADD INDEX ix_entry_file_id (file_id);
   EXIT;

   Both statements are safe to run on a live DB — ALTER TABLE ... ADD COLUMN with a default of NULL does not lock rows in MariaDB 10.6.

5. Rebuild the frontend (if frontend changed)

   toolforge webservice node18 shell -m 2G
   cd ~/www/python/src/frontend
   npm install
   npm run toolforge:build
   exit

6. Install any new Python packages

   toolforge webservice python3.13 shell
   pip install -r ~/www/python/src/requirements.txt
   exit

7. Restart the service

   toolforge webservice python3.13 restart

8. Verify

   • Check https://montage.toolforge.org/meta/ — confirm restart timestamp is recent
   • Do a test category import and verify files appear correctly

[lgelauff]

Beta validation complete (2026-04-19)

Validated end-to-end on montage-beta running MariaDB (10.6.22, utf8mb4_unicode_ci) — same stack as production.

Query parity

• get_files() (new) and get_files_legacy() (old) return identical results for a real WLM category (WLM France 2025: 9,844 files, 0 discrepancy).
• compare_queries.py confirms 0 files only in new, 0 files only in legacy.

Import pipeline

• Category import via UI completed successfully (9,844/9,844 entries, zero gaps).
• Two bugs found and fixed during testing:
  • Duplicate linktarget rows caused duplicate entries → fixed with SELECT DISTINCT in get_files().
  • MariaDB utf8mb4_unicode_ci case-insensitive unique index caused IntegrityError on case-variant filenames → fixed with case-insensitive dedup in add_entries() in rdb.py.

Migration SQL

• migrate_prod_db.sql (add file_id column + index) runs cleanly on MariaDB.
• revert_prod_db.sql (drop column + index) is cleanly reversible — verified by running revert + re-apply cycle on beta.

Production readiness

• PR is ready to merge. After merge: run migrate_prod_db.sql on prod DB, deploy to prod.