Skip to content

Commit a53dd57

Browse files
committed
add a readme file for parameters mapping workflow
1 parent 1b493df commit a53dd57

File tree

1 file changed

+40
-0
lines changed

1 file changed

+40
-0
lines changed
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Parameters Mapping or how to add a metadata header for CSV files
2+
3+
1. check that the parameters to add are listed in [data-services](https://github.com/aodn/data-services/tree/master/PARAMETERS_MAPPING)
4+
* ```parameters.csv``` list of available parameters and their ids
5+
* ```qc_flags.csv```
6+
* ```qc_scheme.csv```
7+
* ```unit_view.csv``` list the available units and their ids (cf names, longnames and id)
8+
9+
1. New parameters
10+
* needs to follow the IMOS vocabulary [BENE PLEASE UPDATE]
11+
12+
1. map the parameters for your dataset collection
13+
* update ```parameters_mapping.csv```. This is the file where all the information from the other files is brought together, and where a variable name as written in the column name of the csv is matched to a unique id for each parameters find in ```parameters.csv```, units find in ```unit_view.csv```, ...
14+
15+
1. Create view in Parameters mapping harvester: update the liquibase to update/include new views in the [harvester](https://github.com/aodn/harvesters/tree/master/workspace/PARAMETERS_MAPPING)
16+
* start your stack restoring the paramaters_mapping schema and the schema you are working on
17+
```RestoreDatabaseSchemas: - schema: parameters_mapping, - schema: working_schema```
18+
* open pgadmin and access your stack-db to test the sql query that will be used to create/update the view in the parameters_mapping harvester, as it is easier to get a better understanding of the query before updating the liquibase via Talend
19+
* start your pipeline box and Talend
20+
* update liquidbase in the second components ```Create parameters_mapping views```
21+
* the query will crash because of 6 views are calling their respective dataset collection schema:
22+
`aatams_biologging_shearwater_metadata_summary`;
23+
`aatams_biologging_snowpetrel_metadata_summary`;
24+
`aatams_sattag_dm_metadata_summary`;
25+
`aatams_sattag_nrt_metadata_summary`;
26+
`aodn_nt_sattag_hawksbill_metadata_summary`;
27+
`aodn_nt_sattag_oliveridley_metadata_summary`
28+
* write the new view you are working on at the top of the liquidbase script, so Talend can run and create before crashing at `aatams_biologging_shearwater_metadata_summary`
29+
* check stack database that the views are created as expected
30+
31+
1. merge the changes made in
32+
* [data-services](https://github.com/aodn/data-services/tree/master/PARAMETERS_MAPPING)
33+
* [harvester](https://github.com/aodn/harvesters/tree/master/workspace/PARAMETERS_MAPPING) to test on RC before merging to production
34+
35+
1. test on RC, check the csv files a user can download from the portal
36+
37+
# Other information
38+
The [PARAMETERS_MAPPING harvester](https://github.com/aodn/harvesters/tree/master/workspace/PARAMETERS_MAPPING) runs on a cron job daily , Monday to Friday.
39+
It harvests the content of these 5 files into the parameters_mapping DB schema and create a _metadata_summary view for each of the collection listed (it is not IMOS specific, for example we have a mapping for the AODN _WAVE_DM + NRT collections)
40+

0 commit comments

Comments
 (0)