4 - API (tests): Add ('Recorded') End-to-End Tests

[RobertRosca]

PR no. 4 of the big refactor.

This PR adds some 'black box' end to end tests which perform API calls to the server to do the tests, and compare the results with expected values. They (almost) never test functions directly, the point of these tests is to be here throughout the refactoring to verify things still (mostly and hopefully) behave the same as before as far as clients/the frontend is concerned.

Details

Lines changed are not as scary as they look, majority of this is external API calls/data that the server needs to be able to respond and stored responses to compare against.

Data

During the tests a few sources of data are used to give the server what it needs to respond and to give the tests something to compare against:

• DAMNIT data:
  • Proposal 9669 data used - same data that the frontend demo uses, but as sqlite file
  • syrup used to make comparing the api responses easier (alternative is a billion assert expected[x] == result[x] checks)
• External API calls handled by cassette/py-vcr which records and replays calls
  • Configured to only store required info and to scrub most sensitive details out of the saved responses
    • Keycloak - basically nothing apart from metadata, mocking handled elsewhere
    • MyMdC - all query responses, kinda anonymised (clear plain text info replaced, internal ids remain as they have to be stable, they're not that sensitive, and there's not that good of a way to replace them)
  • Used cassette as it makes managing/updating the recorded data easy

Mock MyMdC Client

Decided to update the mocked client implementation, before there was some hand-written mymdc.json file that it used to respond, how the mock client (1) responds with the cassette data and (2) is just used directly in the tests.

[CammilleCC]

The CI test workflow would fail if the KeyCloak URL env var is not linked. Could you please add it to the repo accordingly and change the test-api.yml with something similar?:

DW_API_AUTH__SERVER_METADATA_URL: ${{ vars.KEYCLOAK_METADATA_URL }}

Please use the test KeyCloak instance.

[RobertRosca]

Done, thanks. Changed test-api.yml to source it from the repo variable (commit 3967e92e):

DW_API_AUTH__SERVER_METADATA_URL: ${{ vars.KEYCLOAK_METADATA_URL }}

and set the repo Actions variable KEYCLOAK_METADATA_URL to the test KeyCloak instance's discovery URL — https://keycloak-console.desy.de/auth/realms/XFEL/.well-known/openid-configuration — which matches the host the e2e cassettes were recorded against, so vcrpy replays them instead of hitting the network. Re-running CI now.

[CammilleCC]

I mispoke: the KeyCloak instance doesn't matter, it's the same metadata server thoughout.

LGTM