Indexing strategy for large datasets

[hectorcorrea]

The way we are currently indexing data from PDC Describe into PDC Discovery is to store the entire file list as a string field in Solr (see pdc_describe_json_ss and files_ss in https://github.com/pulibrary/pdc_discovery/blob/main/config/traject/pdc_describe_indexing_config.rb).

This approach has worked great for small and medium datasets, but when datasets have 30K or 60K files the single field approach seems to be causing problems (see #711).

We should reviews our strategy for indexing large datasets so that this is not a problem, particularly since the number of files will continue increasing with newer datasets.

cc: @matthewjchandler

[kelynch]

This might be being investigated in #724

[kelynch]

Blocked until we determine if this is necessary based on #724

[hectorcorrea]

This is small incremental step that we could do to improve our indexing: #741

[hectorcorrea]

Also, the work on this ticket will indeed be necessary for when we start processing datasets with hundreds of thousands of files.