Add NvidiaChatGenerator component #1483
Assignees
Labels
Comments
julian-risch
added
feature request
|
I wonder if this would even be necessary because my understanding is that Nvidia NIM can be hosted locally or used through their API, so this could all be accomplished through the OpenAIChatGenerator (they even recommend using OpenAI python library here). There doesn't seem to be anything "unique" to the Nvidia NIM API that would make it necessary to make a generator for those unique features alone. Since it is OpenAI API compliant, this would seem redundant. At least with Llama.cpp or Ollama chat generators that we have, they have some additional unique features/parameters that makes it a good idea to use their dedicated chat generators as opposed to OpenAIChatGenerator with an api_base_url |
julian-risch
removed
the
contributions wanted!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
The current Nvidia integration comprises a
NvidiaTextEmbedder,NvidiaDocumentEmbedder,NvidiaRankerandNvidiaGeneratorbut noNvidiaChatGenerator.We should implement a
NvidiaChatGeneratorcomponent and add a code example to https://haystack.deepset.ai/integrations/nvidiaDescribe the solution you'd like
nvidia/src/haystack_integrations/utils/nvidia/nim_backend.pycontains ageneratemethod that uses the /chat/completions endpoint.We should implement a
chat_generateinnim_backend.py. After thatgeneratecan probably be refactored to callchat_generatewith a list containing a single ChatMessage where the"role"is set to"user".Note that we are implementing a
run_asyncin allChatGeneratorstoo. An initial implementation ofNvidiaChatGeneratorwithoutrun_asyncwould be okay but better if we add it too. #1379Describe alternatives you've considered
Additional context
Adding this component was suggested on discord.
The text was updated successfully, but these errors were encountered: