align functionality between wis2box-api (transform) and wis2box-management (publish)

[maaikelimper]

The latest version of wis2box is using 'wis2box-1.0b6'-branch of the wis2box-api to allow us to implement the synop-form in the wis2box-webapp.

We need to review the workflow in wis2box to allow us to move back to using the wis2box-api:main branch. The current situation should not be maintained as we end up having to align code in multiple places.

[maaikelimper]

Proposal:
wis2box-managment is responsible for publishing and storage
wis2box-api is responsible for data-transformations (csv2bufr, synop2bufr, bufr2bufr) using oapi-processes

wis2box-api:

• receives data and converts data via processes
• send DataPublishRequest via internal mqtt broker to wis2box-management container

wis2box-management:

• listen to events on the internal mqtt broker (unchanged)
• EventName='s3:ObjectCreated:Put' : data-mappings determine what plugin in wis2box.data to be used (unchanged)
• EventName='DataPublishRequest': trigger plugin "wis2box.data.request.DataPublish"
• plugins in wis2box.data make API-request in transform-function to receive output_data

[maaikelimper]

After further investigation I noted that the transform has to been to run as async proccesses both for performance reasons and to keep the memory-space separated between transformations. I've tested locally that I can successfully run wis2box-transformations as async jobs:

Using async jobs will also require an additional process to cleanup old jobs

[maaikelimper]

Note: need to prepare a benchmark to compare impact of performance of moving data-transformation to API

[maaikelimper]

we discussed a sensible default for cleaning up jobs in a cronjob: DELETE request on all successfully completed jobs older than one hour

[maaikelimper]

the message sent by the wis2box-api requesting data publication is defined by https://github.com/wmo-im/wis2box/wiki/PubSubInternalMessageBusSpec

[tomkralidis]

@tomkralidis to review wis2box-api and wis2box PRs on 19 January

[maaikelimper]

I had to rebase PR wis2box-api:wis2box-1.0b7 to match latest commit on main

I noted that on main we use the latest version of pywis-topics and wcmp when building the image:

https://github.com/wmo-im/pywis-topics/archive/main.zip
https://github.com/wmo-im/pywcmp/archive/master.zip

@tomkralidis Could we pin these to specific versions instead ?

[tomkralidis]

Yes, release tags will be put forth when we release (they are unpinned during a development cycle).

[tomkralidis]

2023-02-13: @tomkralidis to review wmo-im/wis2box-api#45 (job cleanup/management)

[tomkralidis]

2024-03-05

• BUFR tools (synop2bufr) need updating and integration (@RoryPTB)