|
| 1 | +# Parameters Mapping or how to add a metadata header for CSV files |
| 2 | + |
| 3 | +1. check that the parameters to add are listed in [data-services](https://github.com/aodn/data-services/tree/master/PARAMETERS_MAPPING) |
| 4 | + * ```parameters.csv``` list of available parameters and their ids |
| 5 | + * ```qc_flags.csv``` |
| 6 | + * ```qc_scheme.csv``` |
| 7 | + * ```unit_view.csv``` list the available units and their ids (cf names, longnames and id) |
| 8 | + |
| 9 | +1. New parameters |
| 10 | + * needs to follow the IMOS vocabulary [BENE PLEASE UPDATE] |
| 11 | + |
| 12 | +1. map the parameters for your dataset collection |
| 13 | + * update ```parameters_mapping.csv```. This is the file where all the information from the other files is brought together, and where a variable name as written in the column name of the csv is matched to a unique id for each parameters find in ```parameters.csv```, units find in ```unit_view.csv```, ... |
| 14 | + |
| 15 | +1. Create view in Parameters mapping harvester: update the liquibase to update/include new views in the [harvester](https://github.com/aodn/harvesters/tree/master/workspace/PARAMETERS_MAPPING) |
| 16 | + * start your stack restoring the paramaters_mapping schema and the schema you are working on |
| 17 | + ```RestoreDatabaseSchemas: - schema: parameters_mapping, - schema: working_schema``` |
| 18 | + * open pgadmin and access your stack-db to test the sql query that will be used to create/update the view in the parameters_mapping harvester, as it is easier to get a better understanding of the query before updating the liquibase via Talend |
| 19 | + * start your pipeline box and Talend |
| 20 | + * update liquidbase in the second components ```Create parameters_mapping views``` |
| 21 | + * the query will crash because of 6 views are calling their respective dataset collection schema: |
| 22 | + `aatams_biologging_shearwater_metadata_summary`; |
| 23 | + `aatams_biologging_snowpetrel_metadata_summary`; |
| 24 | + `aatams_sattag_dm_metadata_summary`; |
| 25 | + `aatams_sattag_nrt_metadata_summary`; |
| 26 | + `aodn_nt_sattag_hawksbill_metadata_summary`; |
| 27 | + `aodn_nt_sattag_oliveridley_metadata_summary` |
| 28 | + * write the new view you are working on at the top of the liquidbase script, so Talend can run and create before crashing at `aatams_biologging_shearwater_metadata_summary` |
| 29 | + * check stack database that the views are created as expected |
| 30 | + |
| 31 | +1. merge the changes made in |
| 32 | + * [data-services](https://github.com/aodn/data-services/tree/master/PARAMETERS_MAPPING) |
| 33 | + * [harvester](https://github.com/aodn/harvesters/tree/master/workspace/PARAMETERS_MAPPING) to test on RC before merging to production |
| 34 | + |
| 35 | +1. test on RC, check the csv files a user can download from the portal |
| 36 | + |
| 37 | +# Other information |
| 38 | +The [PARAMETERS_MAPPING harvester](https://github.com/aodn/harvesters/tree/master/workspace/PARAMETERS_MAPPING) runs on a cron job daily , Monday to Friday. |
| 39 | +It harvests the content of these 5 files into the parameters_mapping DB schema and create a _metadata_summary view for each of the collection listed (it is not IMOS specific, for example we have a mapping for the AODN _WAVE_DM + NRT collections) |
| 40 | + |
0 commit comments