Upgrade faiss to 12b92e9 #1509

jmazanec15 · 2024-03-05T21:53:17Z

Description

Upgrades faiss from 32f0e8c to 12b92e9.

As part of this, we are removing the jni/patches/faiss/0002-Custom-patch-to-support-sqfp16-neon.patch patch because it is already present in faiss.

In future, we will need to remove this line: https://github.com/opensearch-project/k-NN/blob/main/scripts/build.sh#L100 in order to enable neon support. However, this will require gcc 9 in the distribution build. So we wont change that until that change is made.

Also absent in this change is updating faiss version in Faiss.java (see #1515 for more details)

Issues Resolved

#1508

Testing

Manually confirmed 0002-Custom-patch-to-support-AVX2-Linux-CI.patch works:

git apply --ignore-space-change --ignore-whitespace --3way ../../patches/faiss/0002-Custom-patch-to-support-AVX2-Linux-CI.patch
Applied patch to 'faiss/impl/code_distance/code_distance-avx2.h' cleanly.

Check List

Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2024-03-05T23:16:39Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.11%. Comparing base (231ad93) to head (7718804).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1509      +/-   ##
============================================
+ Coverage     85.09%   85.11%   +0.01%     
- Complexity     1280     1281       +1     
============================================
  Files           168      168              
  Lines          5232     5232              
  Branches        495      495              
============================================
+ Hits           4452     4453       +1     
  Misses          572      572              
+ Partials        208      207       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Changes submodule commit to 12b92e9. Signed-off-by: John Mazanec <[email protected]>

Signed-off-by: John Mazanec <[email protected]>

ryanbogan

LGTM!

Upgrades faiss to facebookresearch/faiss@12b92e9. Cleanup outdated patches. Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit 1303182)

* Manually install zlib for win CI (#1513) Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit 231ad93) * Upgrade faiss to 12b92e9 (#1509) Upgrades faiss to facebookresearch/faiss@12b92e9. Cleanup outdated patches. Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit 1303182) * Disable sdc table for HNSWPQ read-only indices (#1518) Passes flag to disable sdc table for the HNSWPQ indices. This table is only used by HNSWPQ during graph creation to compare nodes already present in graph. When we call load index, the graph is read only. Hence, we wont be doing any ingestion and so the table can be disabled to save some memory. Along with this, added a unit test and a couple test helper methods for generating random data. Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit c9262f5) --------- Co-authored-by: John Mazanec <[email protected]>

* Optimize Faiss Query With Filters: Reduce iteration and memory for id filter (#1402) * Optimize Faiss Query With Filters. Reduce iteration copy for docid set iterator Signed-off-by: luyuncheng <[email protected]> * Optimize Faiss Query With Filters. Reduce iteration copy for docid set iterator. Use Bitmap And Batch to do id filter. and you sparse or fixed bitset do exact ANN search Signed-off-by: luyuncheng <[email protected]> * Using int64_t instead of long type for GetLongArrayElements Signed-off-by: luyuncheng <[email protected]> * Add IDSelectorJlongBitmap Signed-off-by: luyuncheng <[email protected]> * 1. Add IDSelectorJlongBitmap and UT for it 2. Move FilterIdsSelectorType to a util class Signed-off-by: luyuncheng <[email protected]> * 1. Add IDSelectorJlongBitmap and UT for it 2. Move FilterIdsSelectorType to a util class 3. Spotless apply Signed-off-by: luyuncheng <[email protected]> * Rebase remote-tracking branch 'origin/main' into Filter Signed-off-by: luyuncheng <[email protected]> * tidy Signed-off-by: luyuncheng <[email protected]> * Add Changelog Signed-off-by: luyuncheng <[email protected]> * fix javadoc tasks Signed-off-by: luyuncheng <[email protected]> * fix bwc javadoc Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector Signed-off-by: luyuncheng <[email protected]> * Rebase faiss_wrapper.cpp Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector For description Select different FilterIdsSelectorType Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector For description Select different FilterIdsSelectorType Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector as Byte.SIZE Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector For comments Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> * Increment 2.12.0-SNAPSHOT to 2.13.0-SNAPSHOT in BWC workflow (#1505) Signed-off-by: Varun Jain <[email protected]> * Manually install zlib for win CI (#1513) Signed-off-by: John Mazanec <[email protected]> * Upgrade faiss to 12b92e9 (#1509) Upgrades faiss to facebookresearch/faiss@12b92e9. Cleanup outdated patches. Signed-off-by: John Mazanec <[email protected]> * Disable sdc table for HNSWPQ read-only indices (#1518) Passes flag to disable sdc table for the HNSWPQ indices. This table is only used by HNSWPQ during graph creation to compare nodes already present in graph. When we call load index, the graph is read only. Hence, we wont be doing any ingestion and so the table can be disabled to save some memory. Along with this, added a unit test and a couple test helper methods for generating random data. Signed-off-by: John Mazanec <[email protected]> * Support distance type radius search for Lucene engine Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve comments Signed-off-by: Junqiu Lei <[email protected]> * Resolve comments Signed-off-by: Junqiu Lei <[email protected]> * Add RNNQueryFactory class Signed-off-by: Junqiu Lei <[email protected]> * Add javadoc Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> Signed-off-by: Varun Jain <[email protected]> Signed-off-by: John Mazanec <[email protected]> Signed-off-by: Junqiu Lei <[email protected]> Co-authored-by: luyuncheng <[email protected]> Co-authored-by: Varun Jain <[email protected]> Co-authored-by: John Mazanec <[email protected]>

…ject#1498) * Optimize Faiss Query With Filters: Reduce iteration and memory for id filter (opensearch-project#1402) * Optimize Faiss Query With Filters. Reduce iteration copy for docid set iterator Signed-off-by: luyuncheng <[email protected]> * Optimize Faiss Query With Filters. Reduce iteration copy for docid set iterator. Use Bitmap And Batch to do id filter. and you sparse or fixed bitset do exact ANN search Signed-off-by: luyuncheng <[email protected]> * Using int64_t instead of long type for GetLongArrayElements Signed-off-by: luyuncheng <[email protected]> * Add IDSelectorJlongBitmap Signed-off-by: luyuncheng <[email protected]> * 1. Add IDSelectorJlongBitmap and UT for it 2. Move FilterIdsSelectorType to a util class Signed-off-by: luyuncheng <[email protected]> * 1. Add IDSelectorJlongBitmap and UT for it 2. Move FilterIdsSelectorType to a util class 3. Spotless apply Signed-off-by: luyuncheng <[email protected]> * Rebase remote-tracking branch 'origin/main' into Filter Signed-off-by: luyuncheng <[email protected]> * tidy Signed-off-by: luyuncheng <[email protected]> * Add Changelog Signed-off-by: luyuncheng <[email protected]> * fix javadoc tasks Signed-off-by: luyuncheng <[email protected]> * fix bwc javadoc Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector Signed-off-by: luyuncheng <[email protected]> * Rebase faiss_wrapper.cpp Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector For description Select different FilterIdsSelectorType Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector For description Select different FilterIdsSelectorType Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector as Byte.SIZE Signed-off-by: luyuncheng <[email protected]> * UpdatedFilterIdsSelector For comments Signed-off-by: luyuncheng <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> * Increment 2.12.0-SNAPSHOT to 2.13.0-SNAPSHOT in BWC workflow (opensearch-project#1505) Signed-off-by: Varun Jain <[email protected]> * Manually install zlib for win CI (opensearch-project#1513) Signed-off-by: John Mazanec <[email protected]> * Upgrade faiss to 12b92e9 (opensearch-project#1509) Upgrades faiss to facebookresearch/faiss@12b92e9. Cleanup outdated patches. Signed-off-by: John Mazanec <[email protected]> * Disable sdc table for HNSWPQ read-only indices (opensearch-project#1518) Passes flag to disable sdc table for the HNSWPQ indices. This table is only used by HNSWPQ during graph creation to compare nodes already present in graph. When we call load index, the graph is read only. Hence, we wont be doing any ingestion and so the table can be disabled to save some memory. Along with this, added a unit test and a couple test helper methods for generating random data. Signed-off-by: John Mazanec <[email protected]> * Support distance type radius search for Lucene engine Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve comments Signed-off-by: Junqiu Lei <[email protected]> * Resolve comments Signed-off-by: Junqiu Lei <[email protected]> * Add RNNQueryFactory class Signed-off-by: Junqiu Lei <[email protected]> * Add javadoc Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> * Resolve feedback Signed-off-by: Junqiu Lei <[email protected]> --------- Signed-off-by: luyuncheng <[email protected]> Signed-off-by: Varun Jain <[email protected]> Signed-off-by: John Mazanec <[email protected]> Signed-off-by: Junqiu Lei <[email protected]> Co-authored-by: luyuncheng <[email protected]> Co-authored-by: Varun Jain <[email protected]> Co-authored-by: John Mazanec <[email protected]>

jmazanec15 added skip-changelog Maintenance Add support for new versions of OpenSearch/Dashboards from upstream v2.13.0 labels Mar 5, 2024

jmazanec15 force-pushed the issue-1508 branch 20 times, most recently from fa6738f to dd9edb3 Compare March 6, 2024 19:21

Upgrade faiss to 12b92e9

bec3130

Changes submodule commit to 12b92e9. Signed-off-by: John Mazanec <[email protected]>

jmazanec15 force-pushed the issue-1508 branch from dd9edb3 to f8337c9 Compare March 6, 2024 21:05

jmazanec15 added backport 2.x and removed v2.13.0 labels Mar 6, 2024

jmazanec15 force-pushed the issue-1508 branch from 14d66f8 to 359bf01 Compare March 6, 2024 21:29

jmazanec15 removed the skip-changelog label Mar 6, 2024

jmazanec15 marked this pull request as ready for review March 6, 2024 21:32

jmazanec15 requested review from heemin32, navneet1v, VijayanB, vamshin, naveentatikonda, junqiu-lei, martin-gaievski and ryanbogan as code owners March 6, 2024 21:32

jmazanec15 added 4 commits March 6, 2024 13:41

Add comment about misleading version

76fbd69

Signed-off-by: John Mazanec <[email protected]>

Removing 0002 patch as is no longer needed

cfd7042

Signed-off-by: John Mazanec <[email protected]>

Rename AVX2 patch for support

49ecfc1

Signed-off-by: John Mazanec <[email protected]>

Update changelog

7718804

Signed-off-by: John Mazanec <[email protected]>

jmazanec15 force-pushed the issue-1508 branch from fd9fd8c to 7718804 Compare March 6, 2024 21:42

naveentatikonda approved these changes Mar 6, 2024

View reviewed changes

ryanbogan approved these changes Mar 6, 2024

View reviewed changes

jmazanec15 merged commit 1303182 into opensearch-project:main Mar 6, 2024
51 checks passed

opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 6, 2024

Upgrade faiss to 12b92e9 (#1509)

7422fac

Upgrades faiss to facebookresearch/faiss@12b92e9. Cleanup outdated patches. Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit 1303182)

opensearch-trigger-bot bot mentioned this pull request Mar 6, 2024

[Backport 2.x] Upgrade faiss to 12b92e9 #1516

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade faiss to 12b92e9 #1509

Upgrade faiss to 12b92e9 #1509

jmazanec15 commented Mar 5, 2024 •

edited

Loading

codecov bot commented Mar 5, 2024 •

edited

Loading

ryanbogan left a comment

Upgrade faiss to 12b92e9 #1509

Upgrade faiss to 12b92e9 #1509

Conversation

jmazanec15 commented Mar 5, 2024 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Mar 5, 2024 • edited Loading

Codecov Report

ryanbogan left a comment

Choose a reason for hiding this comment

jmazanec15 commented Mar 5, 2024 •

edited

Loading

codecov bot commented Mar 5, 2024 •

edited

Loading