Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wis2box-api for wis2box-api-1.0b7 #45

Merged
merged 67 commits into from
Feb 27, 2024
Merged

wis2box-api for wis2box-api-1.0b7 #45

merged 67 commits into from
Feb 27, 2024

Conversation

maaikelimper
Copy link
Collaborator

No description provided.

maaikelimper and others added 30 commits September 13, 2023 15:50
@maaikelimper
Preparing the API for wis2box-1.0b5 (#28)
00192a6 
* Change to docker base image to use wmo-im eccodes image

* new processes: synop2bufr, csv2bufr, bufr2bufr, oscar2feature

* update station-info to match on topics

* add docker-compose-stack for integrated testing of new publishing-processes
@maaikelimper
Preparing the API for wis2box-1.0b5 (#28)
4d2cd0c 
* Change to docker base image to use wmo-im eccodes image

* new processes: synop2bufr, csv2bufr, bufr2bufr, oscar2feature

* update station-info to match on topics

* add docker-compose-stack for integrated testing of new publishing-processes
@maaikelimper
Merge branch 'wis2box-1.0b5' of https://github.com/wmo-im/wis2box-api …
acf70d8 
…into wis2box-1.0b5
@maaikelimper
rename bufr2geojson
563aa97
@maaikelimper
Maaike wip (#31)
6ba50c5 
* csv2bufr v0.7.4
@maaikelimper
align synop2bufr message with csv2bufr (#33)
0694a71
@maaikelimper
pyoscar update and timestamp fix
1ec4958
@maaikelimper
deal with origin/a/wis2
a11e808
@maaikelimper
fix duplicate topic
e2a811b
@tomkralidis
Update station_info.py
0e290f0
@maaikelimper
set collection-id and links in station-info (#34)
460b4e7
@tomkralidis
bump to bufr2geojson 0.5.1
ff44818
@david-i-berry
Merge pull request #35 from wmo-im/bufr2geojson-bump
6d86462 
bufr2geojson bump
@maaikelimper
add missing WIS2BOX_API_URL
bd5b3f6
@maaikelimper
fix wis2box-api-error
58c91f6
@maaikelimper
pin elasticsearch
84f3533
@maaikelimper
remove redundant line
943556a
@maaikelimper
clean-up requirements
8f2d719
@david-i-berry
Merge pull request #37 from wmo-im/fix-wis2box-api-error
4f587d8 
Fix wis2box api error
@maaikelimper
remove wis2/ from data_id
b43eaff
@tomkralidis @maaikelimper
fixes based on latest WTH updates (#39)
69f2874 
* fixes based on latest WTH updates

* load openapi and use at init

---------

Co-authored-by: Maaike <[email protected]>
@maaikelimper
return input wsi (instead of primary) (#40)
0121ab5
@maaikelimper
Wis2box 1.0b6 bufr2geojson (#42)
e989756 
* use internal url bufr2geojson

* bufr2geojson to use internal minio-url

* fixes
@maaikelimper
Merge branch 'main' into wis2box-1.0b6
f6ca3ff
@maaikelimper @tomkralidis
detect updates and duplicates (#44)
faf087e 
* detect updates and duplicates

* flake8

---------

Co-authored-by: Tom Kralidis <[email protected]>
@maaikelimper
add update-link
f8802d0
@maaikelimper
publish DataNotificationRequest
cc32004
@maaikelimper
remove storage dep., use DataNotifcationRequest
4a024f7
@maaikelimper
return data, include storage_path
f6349be
@maaikelimper
update test, fix datetime
e446b70
tomkralidis
tomkralidis requested changes Jan 22, 2024
View reviewed changes
Copy link
Collaborator

@tomkralidis tomkralidis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor changes requests. Overall comment to order imports as follows;

  • standard library
  • 3rd party packages
  • local modules
import os
import sys

import paho.mqtt
import requests

import local.foo_module
import local.z_module

docker/pygeoapi-config.yml
@@ -14,6 +14,10 @@ server:
attribution: ${WIS2BOX_BASEMAP_ATTRIBUTION}
templates:
path: /pygeoapi/pygeoapi/templates
manager:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a cron that runs that runs a script (to be developed) to flush passing jobs older than n days (with a sensible default of days) and keep failing jobs for a given amount of days.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a cronjob removing successful jobs after one hour

docker_compose_test/docker-compose.yml Outdated Show resolved Hide resolved
docker_compose_test/requirements.txt Outdated
@@ -0,0 +1,3 @@
requests
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alphabetize

docker_compose_test/tests/integration/test_wis2box.py
# integration tests assume that the workflow in
# .github/workflows/wis2box_test.yml has been executed

import json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alphabetize imports

docker_compose_test/tests/integration/test_wis2box.py Outdated Show resolved Hide resolved
docker_compose_test/tests/integration/test_wis2box.py Outdated Show resolved Hide resolved
david-i-berry and others added 19 commits January 25, 2024 14:13
@david-i-berry
New process manager based on TinyDB to use multiprocessing.Process ra…
002bc06 
…ther than dummy.
@david-i-berry
Use of pools (need to apply flake checks)
15945dc
@david-i-berry
CRLF -> LF in entrypoints.
8cd73d8
@david-i-berry
Update to psql process manager.
512adbd
@david-i-berry
Working version, expect flake8 errors!
495bf32 
Pushed to test GH action.
@david-i-berry
Updated copyright info, MongoDB manager used as a template but everyt…
05ce808 
…hing has been changed.
@david-i-berry
Fix to use mp.Process.
b1c8434
@david-i-berry
Fix to use mp.Process.
b61c17b
@david-i-berry
Updates to test csv2bufr integration and new warning handling.
86f2db8
@maaikelimper
fix flake8
42e4278
@maaikelimper
Merge branch 'main' into wis2box-1.0b7
46a3db8
@maaikelimper
add pytwis-topics and pywcmp latest
b386650
@maaikelimper
use pywis_topics for pywis-topics-validate
8a6b186
@maaikelimper
back to TinyDB
27cea1e
@maaikelimper
add sleep to ensure stations are cached
77b21e0
@maaikelimper
update station_list.csv
8689949
@maaikelimper
csv2bufr v0.8.0
602c19e
@maaikelimper
paho-mqtt<2 for test
bfa2bdf
@maaikelimper
data-mappings should not be required with latest wis2box-management
d86eafb
@tomkralidis tomkralidis mentioned this pull request Feb 13, 2024
Closed
@maaikelimper maaikelimper merged commit 149c737 into main Feb 27, 2024
6 checks passed
@maaikelimper maaikelimper deleted the wis2box-1.0b7 branch February 27, 2024 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tomkralidis tomkralidis tomkralidis approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@maaikelimper @tomkralidis @david-i-berry