Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HSEARCH-4950 Vector search using Elasticsearch/OpenSearch #3871

Merged
merged 19 commits into from
Jan 17, 2024
Merged

HSEARCH-4950 Vector search using Elasticsearch/OpenSearch #3871

merged 19 commits into from
Jan 17, 2024

Conversation

marko-bekhta
Copy link
Member

https://hibernate.atlassian.net/browse/HSEARCH-4950

Opening as a draft as we need to address an additional parameter for the elasticsearch knn predicate before finishing up the integration.

yrodiere
yrodiere reviewed Dec 18, 2023
View reviewed changes
.../org/hibernate/search/backend/elasticsearch/lowlevel/index/mapping/impl/PropertyMapping.java Outdated Show resolved Hide resolved
...rnate/search/backend/elasticsearch/search/predicate/impl/AbstractElasticsearchPredicate.java Outdated Show resolved Hide resolved
...bernate/search/backend/elasticsearch/search/predicate/impl/ElasticsearchSearchPredicate.java Outdated Show resolved Hide resolved
...bernate/search/backend/elasticsearch/search/projection/impl/AccumulatingSourceExtractor.java Show resolved Hide resolved
...bernate/search/backend/elasticsearch/search/predicate/impl/ElasticsearchSearchPredicate.java Outdated Show resolved Hide resolved
...bernate/search/backend/elasticsearch/search/predicate/impl/ElasticsearchSearchPredicate.java Outdated Show resolved Hide resolved
...a/org/hibernate/search/backend/elasticsearch/dialect/model/impl/OpenSearch2ModelDialect.java Show resolved Hide resolved
...main/java/org/hibernate/search/integrationtest/backend/tck/indexnull/IndexNullAsValueIT.java Outdated Show resolved Hide resolved
...g/hibernate/search/integrationtest/backend/tck/search/predicate/KnnPredicateSpecificsIT.java Outdated Show resolved Hide resolved
...nate/search/integrationtest/backend/tck/testsupport/types/ByteVectorFieldTypeDescriptor.java Show resolved Hide resolved
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch 2 times, most recently from ac135d1 to a4d1449 Compare December 20, 2023 09:47
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch 7 times, most recently from 49fa30c to e56cfb7 Compare December 22, 2023 14:20
@marko-bekhta marko-bekhta marked this pull request as ready for review December 22, 2023 15:41
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch from e56cfb7 to 2181b8e Compare January 5, 2024 16:34
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch from 2181b8e to 46b53da Compare January 8, 2024 16:58
@marko-bekhta marko-bekhta requested a review from yrodiere January 8, 2024 17:50
yrodiere
yrodiere reviewed Jan 10, 2024
View reviewed changes
...arch/backend/elasticsearch/search/predicate/impl/AbstractElasticsearchNestablePredicate.java Outdated Show resolved Hide resolved
documentation/src/main/asciidoc/public/reference/_search-dsl-predicate.adoc Outdated Show resolved Hide resolved
documentation/src/main/asciidoc/public/reference/_search-dsl-predicate.adoc Outdated Show resolved Hide resolved
...tation/src/test/java/org/hibernate/search/documentation/search/predicate/PredicateDslIT.java Outdated Show resolved Hide resolved
documentation/src/main/asciidoc/public/reference/_search-dsl-predicate.adoc Show resolved Hide resolved
...ava/org/hibernate/search/backend/elasticsearch/validation/impl/PropertyMappingValidator.java Show resolved Hide resolved
...ne/src/main/java/org/hibernate/search/engine/search/predicate/dsl/KnnPredicateFieldStep.java Outdated Show resolved Hide resolved
.../src/main/java/org/hibernate/search/engine/search/predicate/dsl/KnnPredicateOptionsStep.java Outdated Show resolved Hide resolved
...e/src/main/java/org/hibernate/search/engine/search/predicate/dsl/KnnPredicateVectorStep.java Outdated Show resolved Hide resolved
.../backend/elasticsearch/search/predicate/dsl/impl/ElasticsearchKnnPredicateFieldStepImpl.java Outdated Show resolved Hide resolved
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch 2 times, most recently from da083fb to c36fb12 Compare January 12, 2024 14:00
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch 3 times, most recently from 2c56832 to 847527f Compare January 12, 2024 16:25
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch from 847527f to 5ae4212 Compare January 15, 2024 10:54
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch 4 times, most recently from 14b7f2f to aeeb486 Compare January 16, 2024 18:03
yrodiere
yrodiere approved these changes Jan 17, 2024
View reviewed changes
Copy link
Member

@yrodiere yrodiere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see where you handled these new checks, but if all tests pass... alright, let's merge :) I've been nitpicking long enough.
Thanks for working on this @marko-bekhta! Please merge as soon as CI passes.

documentation/src/main/asciidoc/public/reference/_search-dsl-predicate.adoc Show resolved Hide resolved
@marko-bekhta
HSEARCH-4950 Add dense/knn vector types to the Elasticsearch backend
2a73e25
@marko-bekhta
HSEARCH-4950 Add dense/knn vector types schema validation
780460d
@marko-bekhta
HSEARCH-4950 Make sure that test data is compatible with all similari…
e8a6079 
…ty functions
@marko-bekhta
HSEARCH-4950 Exclude a few more tests
a0397a5
@marko-bekhta
HSEARCH-4950 Do not allow multivalued vector fields in the Elasticsea…
e98bfa4 
…rch backend
@marko-bekhta
HSEARCH-4950 Add knn predicate implementation for the Elasticsearch d…
4110609 
…istribution only
@marko-bekhta
HSEARCH-4950 Adjust tests to account for optional/required number of …
e163728 
…candidates knn predicate option
@marko-bekhta
HSEARCH-4950 Pass json-arrays to vector field projection extractors
bdc2c7e
@marko-bekhta
HSEARCH-4950 Add knn predicate implementation for the an OpenSearch d…
4183878 
…istribution
@marko-bekhta
HSEARCH-4950 Handle indexNullAs for an Elasticsearch backend
892c256
@marko-bekhta
HSEARCH-4950 Move number of candidates to an Elasticsearch extension
46e7ce3
@marko-bekhta
HSEARCH-4950 Hide knn clauses in a predicate request context
642e120
@marko-bekhta
HSEARCH-4950 Update nested context checks
eb152c9
@marko-bekhta
HSEARCH-4950 Fail faster for older dialects not supporting vectors
bc7e6ae
@marko-bekhta
HSEARCH-4950 Prevent adding knn in some cases for an Elasticsearch ba…
33966f7 
…ckend
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch from 8ee07df to 6f407ad Compare January 17, 2024 10:22
@marko-bekhta marko-bekhta force-pushed the feat/HSEARCH-4950-Vector-search-using-Elasticsearch-OpenSearch branch from 6f407ad to f4ba6bd Compare January 17, 2024 10:39
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions

74.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

@marko-bekhta
Copy link
Member Author

Thanks @yrodiere !!! 😃 I'll merge this in and then if we find anything needs adjusting I'll create a follow-up PR

@marko-bekhta marko-bekhta merged commit d79dcf7 into hibernate:main Jan 17, 2024
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yrodiere yrodiere yrodiere approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@marko-bekhta @yrodiere