Skip to content

feat(harvester): add Memovs audiovisual archives API#4024

Open
rerowep wants to merge 1 commit into
rero:stagingfrom
rerowep:wep-harvesting-audiovisual-archives
Open

feat(harvester): add Memovs audiovisual archives API#4024
rerowep wants to merge 1 commit into
rero:stagingfrom
rerowep:wep-harvesting-audiovisual-archives

Conversation

@rerowep

@rerowep rerowep commented Feb 3, 2026

Copy link
Copy Markdown
Contributor

Add new API harvester for Médiathèque Valais audiovisual archives (archives.memovs.ch). This harvester enables automatic ingestion of Film, Photo, and Audio metadata from the Memovs digital archives.

Implementation includes:

  • ApiMemovs class with pagination and date filtering support
  • JSON transformation layer mapping Memovs metadata to RERO-ILS
  • Document type detection (Film, Audio, Photo, Other)
  • Holdings creation with electronic locations
  • Support for creators, contributors, subjects, and descriptions
  • Integration with existing CLI and task infrastructure

Configuration added as VS-MEMO in apisources.yml with endpoint https://archives.memovs.ch/docs/api/ and code 'memovs'.

@rerowep rerowep self-assigned this Feb 3, 2026
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 2 times, most recently from a072a85 to 6377ef8 Compare February 4, 2026 13:09
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 5 times, most recently from e41e4c2 to 6170f6c Compare February 5, 2026 10:49
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch from 6170f6c to f2ff57a Compare February 9, 2026 15:06
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 3 times, most recently from 02670f4 to 07a5ba6 Compare February 19, 2026 15:37
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 2 times, most recently from 9de0c23 to dd9acfe Compare February 21, 2026 22:23
@coveralls

coveralls commented Feb 22, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 91.229% (-0.08%) from 91.312% — rerowep:wep-harvesting-audiovisual-archives into rero:staging

@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch from dd9acfe to 0861a59 Compare March 18, 2026 13:47
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 6 times, most recently from da8348f to 817eb99 Compare May 13, 2026 06:49
@rerowep rerowep marked this pull request as ready for review May 13, 2026 06:52
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 9 times, most recently from 0473432 to 617a1bc Compare May 18, 2026 10:50
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 8 times, most recently from 6222b76 to b08d551 Compare May 27, 2026 12:59
Comment thread rero_ils/modules/api_harvester/memovs/api.py Outdated
Comment thread rero_ils/modules/api_harvester/utils.py Outdated
Comment thread tests/api_harvester/memovs/test_memovs_api.py
Comment thread tests/api_harvester/memovs/test_memovs_api.py
Comment thread rero_ils/alembic/4b2a1c3d8e9f_apiharvester_location_itemtype_pids.py Outdated
Comment thread rero_ils/alembic/3fa8b1c2d0e7_fix_records_metadata_version_upgrade.py Outdated
Comment thread rero_ils/modules/api_harvester/memovs/dojson/json/__init__.py Outdated
Comment thread rero_ils/modules/api_harvester/memovs/dojson/json/model.py Outdated
Comment thread rero_ils/modules/api_harvester/memovs/dojson/json/model.py Outdated
Comment thread rero_ils/modules/api_harvester/memovs/dojson/json/model.py
Comment thread rero_ils/modules/api_harvester/memovs/dojson/json/model.py Outdated
Comment thread rero_ils/modules/api_harvester/memovs/dojson/__init__.py Outdated
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch from b08d551 to 74eabef Compare May 28, 2026 09:26
@rerowep rerowep requested a review from PascalRepond May 28, 2026 09:42
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch 2 times, most recently from 8d40235 to 3494bf2 Compare May 30, 2026 13:46
Add new API harvester for Médiathèque Valais audiovisual archives
(archives.memovs.ch) enabling automatic ingestion of Film, Photo,
and Audio metadata from the Memovs digital archives into RERO-ILS.

Harvester (api_harvester/memovs/):
* ApiMemovs class with pagination, date filtering, and orphan deletion
* Celery tasks for scheduled harvest and cleanup of removed records
* Configuration as VS-MEMO in apisources.yml with location/item_type PIDs
* Alembic migration for default location and item_type PID settings

BIBFRAME-to-RERO-ILS transformation (dojson/json/model.py):
* Document type detection from bf:content/bf:media/bf:carrier (Film, Audio, Photo, Other)
* Provision activity from bf:provisionActivity with date normalisation
* Electronic locators for landing page and thumbnail
* MEF entity resolution for contributions, subjects, genre/form and places
* Fallback to identifiedBy when MEF ref cannot be resolved
* Physical description from bf:extent split into extent, otherPhysicalDetails and colorContent
* Warning logged when bf:contribution agent carries a URL as rdfs:label

Document and item creation (api.py):
* Holdings created per document with electronic locations
* Items created per bf:hasPart call number, stale items deleted on update
* Local fields synced from bf:note annotations (vsavmat, vsavgeo, vsavfonds...)

Other changes:
* feat(documents): add rpt (Reporter) to contribution roles schema
* refactor(documents): load _CONTRIBUTION_ROLE dynamically from JSON schema

Co-authored-by: Peter Weber <peter.weber@rero.ch>
@rerowep rerowep force-pushed the wep-harvesting-audiovisual-archives branch from 3494bf2 to de758ad Compare June 9, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants