From fd3621a524c069e9c573bb28810494788ee06dbe Mon Sep 17 00:00:00 2001 From: James Wood Date: Wed, 5 Feb 2025 14:14:08 -0800 Subject: [PATCH 1/2] Initial docs --- EOSDISIntegrations.md | 68 +++++++++++++++++++++++++++++++++++++++++++ README.md | 4 +++ 2 files changed, 72 insertions(+) create mode 100644 EOSDISIntegrations.md diff --git a/EOSDISIntegrations.md b/EOSDISIntegrations.md new file mode 100644 index 00000000..e045ac54 --- /dev/null +++ b/EOSDISIntegrations.md @@ -0,0 +1,68 @@ + +# NASA EOSDIS Integrations + +## Overview + +This document provides detailed instructions for DAACs to integrate their collections with the PODAAC l2ss-py Harmony service. + +### Adding new collections to L2ss-py Harmony Service + +1. Add UMM-V for new collections to both OPS and UAT. Must define latitude and longitude. Time is optional but preferred if used in the collection. + + a. The latitude and longitude variables require a Type of "COORDINATE" and a subtype of "LATITUDE" or "LONGITUDE" + +2. Add the new association to the l2ss-py service in UAT and OPS + + a. Go to the UAT Earthdata [Metadata Management Tool](https://mmt.uat.earthdata.nasa.gov) + + b. Login with your Earthdata credentials + + c. Go to Services (On the left-hand side), and click the "All Services" link + + d. Repeat steps above for OPS + +3. [UAT] Manually test the new association (Optional) + + a. (Test with UAT Harmony) Test the new association with Harmony + + i. Test subsetting in EDSC UAT + + ii. Test the collection in Harmony UAT with curl + + b. (Test l2ss-py locally) Test a granule from the collection with l2ss-py locally to ensure the collection is working as expected + +4. (Autotest) Within 3 days, check how the collection is doing in the l2ss-py-autotest + + a. Check the errors in the PRs + + b. add more... + +5. If there are errors with l2ss-py-autotest PRs then: + + a. Try to determine if the error is in the l2ss-py code or in the l2ss-py-autotest. + + b. Fork either the l2ss-py repo or the l2ss-py-autotest repo based on previous step. + + b. Create a new branch from `develop` named `feature/` + + c. Make the necessary changes in the branch and test locally until the error is fixed. + + d. Make a PR to the l2ss-py or l2ss-py-autotest repo into the `develop` branch + + e. Ask for a review from the PODAAC team. Add `jamesfwood` and `sliu` as Reviewers. + + f. [PODAAC] The PODAAC team will review the PR and merge it into the `develop` branch + + g. [PODAAC] The PODAAC team will make a new release of l2ss-py and deploy it to UAT + + h. **[UAT]** Within 3 days, the l2ss-py-autotest will run and retest the collection in UAT. If the collection passes the autotest, then everything is good in UAT. If the collection fails the autotest, then the DAAC will need to repeat step 5. + + i. Once everything is in the l2ss-py release, the PODAAC team will merge the release to main and deploy it to OPS. This step may take some time before it is completed. + + j. **[OPS]** Within 3 days of deploying the release to OPS, the l2ss-py-autotest will run and retest the collection in OPS. If the collection passes the autotest, then proceed to next step. If the collection fails the autotest, then the DAAC will need to repeat step 5. + +6. Manually test the collection with the l2ss-py Harmony service in OPS + + a. curl the collection from the l2ss-py Harmony service in OPS + + b. Test subsetting the collection in EDSC OPS diff --git a/README.md b/README.md index 266a1f71..dd236c7e 100644 --- a/README.md +++ b/README.md @@ -115,3 +115,7 @@ In order to fully test l2ss-py with Harmony, you can run Harmony locally. This r ``` localhost:3000/YOUR_COLLECTION_ID/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?format=application%2Fx-netcdf4&subset=lat(-10%3A10)&subset=lon(-10%3A10)&maxResults=2 ``` + +## NASA EOSDIS Integration + +Detailed instructions for integrating with EOSDIS can be found [here](EOSDISIntegrations.md). From 5ec35da55dfbca154e76010b48307f7ae7516f43 Mon Sep 17 00:00:00 2001 From: James Wood Date: Thu, 6 Feb 2025 17:37:09 -0800 Subject: [PATCH 2/2] Added more details --- EOSDISIntegrations.md | 73 ++++++++++++++++++++++++++++++------------- 1 file changed, 51 insertions(+), 22 deletions(-) diff --git a/EOSDISIntegrations.md b/EOSDISIntegrations.md index e045ac54..52c16a6a 100644 --- a/EOSDISIntegrations.md +++ b/EOSDISIntegrations.md @@ -7,21 +7,34 @@ This document provides detailed instructions for DAACs to integrate their collec ### Adding new collections to L2ss-py Harmony Service -1. Add UMM-V for new collections to both OPS and UAT. Must define latitude and longitude. Time is optional but preferred if used in the collection. +1. #### Add UMM-V for new collections to both OPS and UAT. Must define latitude and longitude. Time is optional but preferred if used in the collection. - a. The latitude and longitude variables require a Type of "COORDINATE" and a subtype of "LATITUDE" or "LONGITUDE" + a. The latitude and longitude variables require a Type of "COORDINATE" and a SubType of "LATITUDE" or "LONGITUDE" respectively. -2. Add the new association to the l2ss-py service in UAT and OPS +2. #### Add the new association to the l2ss-py service in UAT and OPS - a. Go to the UAT Earthdata [Metadata Management Tool](https://mmt.uat.earthdata.nasa.gov) + a. Go to the UAT Earthdata [Metadata Management Tool UAT](https://mmt.uat.earthdata.nasa.gov) b. Login with your Earthdata credentials c. Go to Services (On the left-hand side), and click the "All Services" link - d. Repeat steps above for OPS + d. Search for "podaac l2" in the Search Bar -3. [UAT] Manually test the new association (Optional) + e. Click the `PODAAC L2 Cloud Subsetter` service from provider `POCLOUD` + + f. Click the 3 dots on the upper right side and then click `Collection Associations` + + g. Click `Add Collection Associations` + + h. Use the Search Field to find the collection you want to add. + + i. Check the box next to the collections you want to add and click `Associate Selected Collections` + + j. Repeat steps above for OPS, go to the OPS Earthdata [Metadata Management Tool OPS](https://mmt.earthdata.nasa.gov) + + +3. #### Manually test the new association (OPTIONAL) a. (Test with UAT Harmony) Test the new association with Harmony @@ -31,38 +44,54 @@ This document provides detailed instructions for DAACs to integrate their collec b. (Test l2ss-py locally) Test a granule from the collection with l2ss-py locally to ensure the collection is working as expected -4. (Autotest) Within 3 days, check how the collection is doing in the l2ss-py-autotest +4. #### (Autotest) Within 3 days, check how the collection is doing in the l2ss-py-autotest + + a. Check for errors in the PRs in [l2ss-py-autotest](https://github.com/podaac/l2ss-py-autotest/pulls) + + i. Go to the Pull requests tab and Filter using your `short name` or `concept id`. Remove `is:open` to check closed PRs also. + + ii. If you see it in the list as `Open` then it either hasn't been tested yet or it failed. + + iii. If you find it `Closed` and `Merged` then it passed and you can skip step 5. + + b. If you find the PR as `Open` with a red X (failed), click the PR and then click the `Checks` tab + + c. Look on the left side to see the status of the PR and see which part failed, which will have a red X next to it. Usually it is the `Tested with Harmony` that fails. + + d. Click the failed item and it will show which tests failed. There are 2 types of tests, the `test_spatial_subset` and the `test_temporal_subset` + + e. Click the `Raw output` button to see the details of the tests. - a. Check the errors in the PRs + f. Use this information to determine if the error is in the l2ss-py code or in the l2ss-py-autotest. - b. add more... + g. If your collection doesn't use `time` at all, then you can create a PR to skip the `test_temporal_subset` from this collection. Update the appropriate file in this [directory](https://github.com/podaac/l2ss-py-autotest/tree/main/tests/skip). Add the collection concept id to the appropriate file. -5. If there are errors with l2ss-py-autotest PRs then: +5. #### If there are errors in your l2ss-py-autotest collection PRs then: a. Try to determine if the error is in the l2ss-py code or in the l2ss-py-autotest. b. Fork either the l2ss-py repo or the l2ss-py-autotest repo based on previous step. - b. Create a new branch from `develop` named `feature/` + c. Create a new branch from `develop` named `feature/fix-` - c. Make the necessary changes in the branch and test locally until the error is fixed. + d. Make the necessary changes in the branch and test locally until the error is fixed. - d. Make a PR to the l2ss-py or l2ss-py-autotest repo into the `develop` branch + e. Make a PR to the l2ss-py or l2ss-py-autotest repo into the `develop` branch - e. Ask for a review from the PODAAC team. Add `jamesfwood` and `sliu` as Reviewers. + f. Ask for a review from the PODAAC team. Add `jamesfwood` and `sliu` as Reviewers. - f. [PODAAC] The PODAAC team will review the PR and merge it into the `develop` branch + g. [PODAAC] The PODAAC team will review the PR and merge it into the `develop` branch - g. [PODAAC] The PODAAC team will make a new release of l2ss-py and deploy it to UAT + h. [PODAAC] The PODAAC team will make a new release of l2ss-py and deploy it to UAT - h. **[UAT]** Within 3 days, the l2ss-py-autotest will run and retest the collection in UAT. If the collection passes the autotest, then everything is good in UAT. If the collection fails the autotest, then the DAAC will need to repeat step 5. + i. Within 3 days, the l2ss-py-autotest will run and retest the collection in UAT. If the collection passes the autotest, then everything is good in UAT. If the collection fails the autotest, then the DAAC will need to repeat step 5. - i. Once everything is in the l2ss-py release, the PODAAC team will merge the release to main and deploy it to OPS. This step may take some time before it is completed. + j. [PODAAC] Once everything is in the l2ss-py release, the PODAAC team will merge the release to main and deploy it to OPS. This step may take some time since it may be waiting for other changes to be completed. - j. **[OPS]** Within 3 days of deploying the release to OPS, the l2ss-py-autotest will run and retest the collection in OPS. If the collection passes the autotest, then proceed to next step. If the collection fails the autotest, then the DAAC will need to repeat step 5. + k. Within 3 days of deploying the release to OPS, the l2ss-py-autotest will run and retest the collection in OPS. If the collection passes the autotest, then proceed to next step. If the collection fails the autotest, then the DAAC will need to repeat step 5. -6. Manually test the collection with the l2ss-py Harmony service in OPS +6. #### Test the collection with the l2ss-py Harmony service in OPS - a. curl the collection from the l2ss-py Harmony service in OPS + a. Test subsetting the collection in EDSC OPS - b. Test subsetting the collection in EDSC OPS + b. curl the collection from the l2ss-py Harmony service in OPS