As a user, I want to search for ESA data sets and context products

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

Search Engineer

💪 Motivation

...so that I can search for ESA data in our keyword search interface

📖 Additional Details

We need a script to:

• query the API to get all ESA context products, bundles, and collections
• download all the files
• load them into Solr

Acceptance Criteria

Given an index containing all ESA data loaded in the OpenSearch registry
When I perform a keyword search query for "bepicolombo"
Then I expect the context product for the BepiColombo investigation and the associated bundles/collections to be returned

⚙️ Engineering Details

No response

🎉 I&T

No response

[nutjob4life]

Hi @jordanpadams, couple questions:

Can this same query work?

(
    (
        product_class eq "Product_Context" or
        product_class eq "Product_Bundle" or
        product_class eq "Product_Collection"
    )
    and ops:Harvest_Info.ops:node_name like "PSA"
)

Secondly, is the idea like registry-legacy-soly#135 but this time download all the ops:Data_File_Info.ops:file_refs into a directory tree? Or is there also a pysolr step?

[jordanpadams]

@nutjob4life this is basically a follow-on to NASA-PDS/registry-legacy-solr#135 where we actually load the data to support the interface. No implementation needed.

[nutjob4life]

Oh! So just run harvest -c harvest.cfg?

The harvest.cfg generated by the script in NASA-PDS/registry-legacy-solr#135 looks like this:

<?xml version='1.0' encoding='UTF-8'?>
<harvest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://github.com/NASA-PDS/harvest/blob/main/src/main/resources/conf/configuration.xsd">
  <registry auth="/path/to/auth/file">app://localhost.xml</registry>
  <load>
    <directories>
      <path>/Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/operations/download</path>
    </directories>
  </load>
  <fileInfo processDataFiles="true" storeLabels="true">
    <fileRef replacePrefix="/Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/operations/download" with="https://url/to/archive"/>
  </fileInfo>
  <autoGenFields/>
</harvest>

which I'm certain is wrong, specifically the <registry auth=… and the with= attribute in the <fileRef>. What really should be there? Does ESA PSA have a node URL? (Sorry, I'm a total "harvest" newb.)