Zero-config dynamically-generated queryables, Performance fixes #351
Conversation
…cause it looks odd
|
Hi @Zaczero. Looks really good, just a few linting errors. |
|
It would be nice to have @rhysrevans3 look at this as he is doing work on the queryables. |
|
@jonhealy1 This looks good to me and is probably a neater solution for @Zaczero am I right in saying this doesn't change the filter extension query? So doesn't effect the queryables mapping that is used in to_es |
|
@rhysrevans3 I sent you an invite to be a maintainer of this project so you can do an official pr review. |
There was a problem hiding this comment.
These look good!
@jonhealy1 I think my queryables pull request could still be useful. It takes a different approach, extracting the fields from the extensions where possible and does something similar to this pull request for the other properties. I'd be happy to attempt to merge the two methods in my pull request once this one is accepted if we want both?
| @@ -6,7 +6,7 @@ | |||
| desc = f.read() | |||
|
|
|||
| install_requires = [ | |||
| "fastapi-slim", | |||
There was a problem hiding this comment.
Just for my education what does fastapi have that fastapi-slim doesn't have for us?
There was a problem hiding this comment.
I guess fastapi-slim isn't really going to still be a thing in the future. fastapi/fastapi#11525 (comment)
There was a problem hiding this comment.
fastapi-slim is, I think, a barebones version of fastapi but now the default fastapi is barebones.
| Returns: | ||
| str: The index name derived from the collection id. | ||
| """ | ||
| cleaned = collection_id.translate(_ES_INDEX_NAME_UNSUPPORTED_CHARS_TABLE) |
There was a problem hiding this comment.
Not seen this before very cool!
| "cloud_cover": { | ||
| "description": "Cloud Cover", | ||
| "$ref": "https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fields/properties/eo:cloud_cover", |
There was a problem hiding this comment.
I know this isn't your change but is this right? But shouldn't the key be eo:cloud_cover? My understanding of the queryables the extension isn't removed?
There was a problem hiding this comment.
I don't know about this personally. Should we create an issue to look into this?
|
This looks amazing. I'd like to follow this same approach to generate aggregations as well |
Related Issue(s):
Description:
This PR consists of self-contained commits (except the first commit that provides database_logic deduplication), making it easy to change or remove individual patches. It addresses several small issues, improves the performance of certain methods, and adds support for dynamically-generated queryables. This enhancement doesn't require any new configuration as queryables are generated on the fly based on the os/es mappings. The implementation is designed for extensibility, with built-in logic for augmenting fields metadata with additional information. Currently, it only includes the _DEFAULT_QUERYABLES configuration, which was simply copied from the pre-PR code.
Example queryables response:
{"$schema":"https://json-schema.org/draft/2019-09/schema","$id":"https://stac-api.example.com/queryables","type":"object","title":"Queryables for STAC API","description":"Queryable names for the STAC API Item Search filter.","properties":{"bbox":{"title":"Bbox","type":"number"},"collection":{"description":"Collection","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/then/properties/collection","title":"Collection","type":"string"},"geometry":{"description":"Geometry","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/1/oneOf/0/properties/geometry","title":"Geometry","type":"object"},"id":{"description":"ID","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/item.json#/definitions/core/allOf/2/properties/id","title":"Id","type":"string"},"stac_extensions":{"title":"Stac Extensions","type":"string"},"stac_version":{"title":"Stac Version","type":"string"},"type":{"title":"Type","type":"string"},"constellation":{"title":"Constellation","type":"string"},"created":{"description":"Creation Timestamp","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/datetime.json#/properties/created","title":"Created","type":"string","format":"date-time"},"datetime":{"description":"Acquisition Timestamp","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/datetime.json#/properties/datetime","title":"Datetime","type":"string","format":"date-time"},"end_datetime":{"title":"End Datetime","type":"string","format":"date-time"},"eopf:datatake_id":{"title":"Eopf:Datatake Id","type":"string"},"eopf:instrument_configuration_id":{"title":"Eopf:Instrument Configuration Id","type":"number"},"instruments":{"title":"Instruments","type":"string"},"platform":{"title":"Platform","type":"string"},"processing:datetime":{"title":"Processing:Datetime","type":"string","format":"date-time"},"processing:facility":{"title":"Processing:Facility","type":"string"},"processing:level":{"title":"Processing:Level","type":"string"},"processing:version":{"title":"Processing:Version","type":"string"},"product:timeliness":{"title":"Product:Timeliness","type":"string"},"product:timeliness_category":{"title":"Product:Timeliness Category","type":"string"},"product:type":{"title":"Product:Type","type":"string"},"published":{"title":"Published","type":"string","format":"date-time"},"sar:center_frequency":{"title":"Sar:Center Frequency","type":"number"},"sar:frequency_band":{"title":"Sar:Frequency Band","type":"string"},"sar:instrument_mode":{"title":"Sar:Instrument Mode","type":"string"},"sar:observation_direction":{"title":"Sar:Observation Direction","type":"string"},"sar:pixel_spacing_azimuth":{"title":"Sar:Pixel Spacing Azimuth","type":"number"},"sar:pixel_spacing_range":{"title":"Sar:Pixel Spacing Range","type":"number"},"sar:polarizations":{"title":"Sar:Polarizations","type":"string"},"sar:resolution_azimuth":{"title":"Sar:Resolution Azimuth","type":"number"},"sar:resolution_range":{"title":"Sar:Resolution Range","type":"number"},"sat:absolute_orbit":{"title":"Sat:Absolute Orbit","type":"integer"},"sat:orbit_cycle":{"title":"Sat:Orbit Cycle","type":"number"},"sat:orbit_state":{"title":"Sat:Orbit State","type":"string"},"sat:platform_international_designator":{"title":"Sat:Platform International Designator","type":"string"},"sat:relative_orbit":{"title":"Sat:Relative Orbit","type":"integer"},"start_datetime":{"title":"Start Datetime","type":"string","format":"date-time"},"updated":{"description":"Creation Timestamp","$ref":"https://schemas.stacspec.org/v1.0.0/item-spec/json-schema/datetime.json#/properties/updated","title":"Updated","type":"string","format":"date-time"},"view:azimuth":{"title":"View:Azimuth","type":"number"},"view:incidence_angle":{"title":"View:Incidence Angle","type":"number"},"auth:schemes.oidc.openIdConnectUrl":{"title":"Auth:Schemes.Oidc.Openidconnecturl","type":"string"},"auth:schemes.oidc.type":{"title":"Auth:Schemes.Oidc.Type","type":"string"},"auth:schemes.s3.type":{"title":"Auth:Schemes.S3.Type","type":"string"},"storage:schemes.cdse-s3.description":{"title":"Storage:Schemes.Cdse-S3.Description","type":"string"},"storage:schemes.cdse-s3.platform":{"title":"Storage:Schemes.Cdse-S3.Platform","type":"string"},"storage:schemes.cdse-s3.requester_pays":{"title":"Storage:Schemes.Cdse-S3.Requester Pays","type":"boolean"},"storage:schemes.cdse-s3.title":{"title":"Storage:Schemes.Cdse-S3.Title","type":"string"},"storage:schemes.cdse-s3.type":{"title":"Storage:Schemes.Cdse-S3.Type","type":"string"},"storage:schemes.creodias-s3.description":{"title":"Storage:Schemes.Creodias-S3.Description","type":"string"},"storage:schemes.creodias-s3.platform":{"title":"Storage:Schemes.Creodias-S3.Platform","type":"string"},"storage:schemes.creodias-s3.requester_pays":{"title":"Storage:Schemes.Creodias-S3.Requester Pays","type":"boolean"},"storage:schemes.creodias-s3.title":{"title":"Storage:Schemes.Creodias-S3.Title","type":"string"},"storage:schemes.creodias-s3.type":{"title":"Storage:Schemes.Creodias-S3.Type","type":"string"}},"additionalProperties":false}PS. I think the auto-generated "title" should be removed completely, but I included it because I found it to be common practice in some STAC projects. I'm not sure how you feel about it.
PR Checklist:
pre-commit run --all-files)make test)