Training your own Dense Retriever from scratch

[jonsaadfalcon]

Hello, is it possible to train your own dense retriever from scratch? If so, what would be the expected commands?

[lintool]

What dense retrieval model are you trying to train? Training itself is out of scope wrt Pyserini, and is usually handled by code in other repos... e.g., consider https://github.com/texttron/tevatron

[xhluca]

I'm not sure about what @jonsaadfalcon wants to accomplish, but in my case I have various projects where I want to train DPR to perform retrieval on very specific dataset. Moreover, due to low lexical overlap, un-trainable methods like BM25 are not necessarily appropriate for those use cases, but DPR seems to perform slightly better.

Tevatron looks like a promising project, but after going over the docs, I am still not sure how it's supposed to be used along libraries like Huggingface and Pytorch inside Python (as opposed to CLI-only usage). It seems that there are abstraction for Python usage, but I'm not sure if they are meant to be used by the public.

@MXueguang Happy to hear your thoughts!

[MXueguang]

Hi @xhluca,
You can customize trainer, model, datasets etc based on modules in Tevatron and then train your customized model.
E.g. https://github.com/texttron/tevatron/blob/b8f33900895930f9886012580e85464a5c1f7e9a/examples/splade/train_splade.py#L27
It customized the trainer, and model but use the rest of the pipeline in Tevatron.

[xhluca]

Thanks for sharing this! Can we use the resulting trained model as a hugging face model?

[MXueguang]

If there is no any additional parameters for the model, it should be able to directly used by huggingface

[xhluca]

Thanks, I'll take a closer look :)