Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terrascope missing products check: avoid cloudferro storage dependency #1046

Open
jdries opened this issue Feb 12, 2025 · 0 comments
Open

terrascope missing products check: avoid cloudferro storage dependency #1046

jdries opened this issue Feb 12, 2025 · 0 comments

Comments

@jdries
Copy link
Contributor

jdries commented Feb 12, 2025

The missing products check on terrascope can fail when storage issues occur at cloud provider.
This can be avoided by only depending on the catalog, and avoiding metadata retrieval from object storage.
Using the STAC catalog, we can get all the necessary metadata from the service rather than from object storage lookup.

File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 152, in _load_collection_cached
    metadata = self._resolve_merged_by_common_name(
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 775, in _resolve_merged_by_common_name
    missing = check_missing_products(
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 1202, in check_missing_products
    expected_tiles = query_jvm_opensearch_client(
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/layercatalog.py", line 1140, in query_jvm_opensearch_client
    products = open_search_client.getProducts(
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
    return_value = get_return_value(
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o1734.getProducts.
: java.io.IOException: Server returned HTTP response code: 504 for URL: https://zipper.creodias.eu/get-object?path=/Sentinel-2/MSI/L1C_N0500/2020/07/20/S2A_MSIL1C_20200720T102031_N0500_R065_T32TPT_20230422T170513.SAFE/manifest.safe
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$.$anonfun$loadMetadata$1(OpenSearchResponses.scala:583)
	at org.openeo.opensearch.package$.attempt$1(package.scala:46)
	at org.openeo.opensearch.package$.withRetries(package.scala:58)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$.loadMetadata(OpenSearchResponses.scala:536)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$.org$openeo$opensearch$OpenSearchResponses$CreoFeatureCollection$$getFilePathsFromManifest(OpenSearchResponses.scala:610)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$$anon$12.$anonfun$apply$40(OpenSearchResponses.scala:871)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$$anon$12.$anonfun$apply$38(OpenSearchResponses.scala:869)
	at scala.util.Either.map(Either.scala:353)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$$anon$12.apply(OpenSearchResponses.scala:851)
	at io.circe.Decoder.decodeJson(Decoder.scala:128)
	at io.circe.Decoder.decodeJson$(Decoder.scala:128)
	at org.openeo.opensearch.OpenSearchResponses$CreoFeatureCollection$$anon$12.decodeJson(OpenSearchResponses.scala:848)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant