Add Splade v1 (#144)

* Add SPLADE v1

* WIP SPLADE Export errors

* add ONNX model to HF hub and use that

* Update sentences in Converting_SPLADE_to_ONNX.ipynb

* Remove unnecessary files and directories

* Rename var in TextEmbedding class to use EMBEDDING_MODEL_TYPE

* Add SPLADE to list of text embeddings

* Add SPLADE model support for text embedding

* Fix deprecation warning in embedding.py

* Add test for batch embedding with sparse embeddings

* Refactor import statement in test_sparse_embeddings.py

* Rename nbs

* Update vocab size in SPLADE model

* Fix canonical vector lookup in test_text_onnx_embeddings.py

* review refactoring

* restore list_supported_models in OnnxTextEmbedding

* Remove unused method _preprocess_onnx_input() in SpladePP class

* Update SPLADE_PP_en_v1 source in splade_pp.py

* Refactor onnx_model.py to change base model behavior

* extend tests to sparse values as well as indicies

* chore: pre-commit hooks

---------

Co-authored-by: generall <[email protected]>
Co-authored-by: Anush008 <[email protected]>

Author: 3 people

.gitignore

@@ -168,11 +168,5 @@ local_cache/*/*
 docs/experimental/*.parquet
 docs/experimental/*.bin
 qdrant_storage/*
-fooling_around/fast-multilingual-e5-large/config.json
-fooling_around/fast-multilingual-e5-large/model_optimized.onnx
-fooling_around/fast-multilingual-e5-large/model_optimized.onnx.data
-fooling_around/fast-multilingual-e5-large/ort_config.json
-fooling_around/fast-multilingual-e5-large/sentencepiece.bpe.model
-fooling_around/fast-multilingual-e5-large/special_tokens_map.json
-fooling_around/fast-multilingual-e5-large/tokenizer_config.json
-fooling_around/fast-multilingual-e5-large/tokenizer.json
+fooling_around/*
+experiments/models/*

docs/Getting Started.ipynb

@@ -81,8 +81,8 @@
     "\n",
     "embeddings_generator = embedding_model.embed(documents)  # reminder this is a generator\n",
     "embeddings_list = list(embedding_model.embed(documents))\n",
-    "  # you can also convert the generator to a list, and that to a numpy array\n",
-    "len(embeddings_list[0]) # Vector of 384 dimensions"
+    "# you can also convert the generator to a list, and that to a numpy array\n",
+    "len(embeddings_list[0])  # Vector of 384 dimensions"
    ]
   },
   {
@@ -185,7 +185,7 @@
     }
    ],
    "source": [
-    "multilingual_large_model = TextEmbedding(\"intfloat/multilingual-e5-large\") # This can take a few minutes to download"
+    "multilingual_large_model = TextEmbedding(\"intfloat/multilingual-e5-large\")  # This can take a few minutes to download"
    ]
   },
   {
@@ -206,7 +206,9 @@
     }
    ],
    "source": [
-    "np.array(list(multilingual_large_model.embed([\"Hello, world!\", \"你好世界\", \"¡Hola Mundo!\", \"नमस्ते!\"]))).shape # Vector of 1024 dimensions"
+    "np.array(\n",
+    "    list(multilingual_large_model.embed([\"Hello, world!\", \"你好世界\", \"¡Hola Mundo!\", \"नमस्ते!\"]))\n",
+    ").shape  # Vector of 1024 dimensions"
    ]
   },
   {