Add Vertex AI inference providers by yangcao77 · Pull Request #41 · redhat-ai-dev/lightspeed-configs

yangcao77 · 2026-07-16T16:10:37Z

Summary

Adds vertexai inference providers to lightspeed-core-configs/lightspeed-stack.yaml
Fixes Vertex AI credentials not reaching the container: adds a GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH volume mount in compose/compose.yaml (with a harmless empty placeholder fallback via the new env/gcp-credentials-placeholder.json so the mount never fails when unset), and updates env/default-values.env accordingly.
Documentsdocs/PROVIDERS.mdfor expanded Vertex AI credential setup instructions.

Test plan

Started the stack locally (make local-up) against quay.io/lightspeed-core/lightspeed-stack:dev-20260710-2c3e52e; container reached healthy.
Verified GET /v1/models returns models for vertex new provider
Verified POST /v1/query returns a valid response using provider: vertexai.

qodo-code-review · 2026-07-16T16:11:42Z

PR Summary by Qodo

Add Gemini (OpenAI-compatible) and Vertex AI inference providers

✨ Enhancement ⚙️ Configuration changes 📝 Documentation 🕐 20-40 Minutes

AI Description

• Add Gemini and Vertex AI providers to the default Lightspeed inference configuration.
• Mount Google application credentials into the container via compose with a safe placeholder
 fallback.
• Document Gemini setup (OpenAI base_url workaround) and expanded Vertex AI credential steps.

Diagram

graph TD
  E["env/default-values.env"] --> C["compose/compose.yaml"] --> S(["Lightspeed Core container"]) --> L["lightspeed-core-configs/lightspeed-stack.yaml"]
  C --> K["GCP creds (mounted)"] --> S
  S --> G{{"Gemini OpenAI API"}}
  S --> V{{"Vertex AI API"}}
  subgraph Legend
    direction LR
    _file["Config file"] ~~~ _svc(["Container/service"]) ~~~ _ext{{"External API"}}
  end

High-Level Assessment

The following are alternative approaches to this PR:

1. Add a first-class `gemini` provider type to the inference schema

➕ Avoids the conceptual mismatch of configuring Gemini as openai-type
➕ Allows Gemini-specific settings/validation (regions, auth modes, model naming)
➖ Requires upstream schema/code changes rather than a config-only enablement
➖ Longer lead time and higher coordination cost

2. Standardize on Vertex AI for Gemini and omit the consumer Gemini API path

➕ Single enterprise-ready auth mechanism (IAM/ADC) and project-based billing
➕ Avoids maintaining two different Gemini onboarding paths
➖ Higher setup friction for local/dev users without GCP project access
➖ Does not cover users who only have a Gemini API key

3. Provide a helper profile/overlay config for Google providers instead of enabling by default

➕ Keeps the default stack minimal for users who don't need Google providers
➕ Reduces chance of confusion around credential mounting variables
➖ Extra step for users who do want Gemini/Vertex AI
➖ Documentation and examples must point to multiple config entry points

Recommendation: Current approach is appropriate for a config-driven enablement: it uses the existing openai provider type for Gemini via the documented OpenAI-compatible endpoint, while adding a proper vertexai provider with explicit credential mounting. If Gemini usage becomes widespread or needs provider-specific options/validation, consider the first-class gemini provider type as a follow-up.

Files changed (5) +54 / -1

Enhancement (1) +10 / -0

lightspeed-stack.yamlEnable Gemini (OpenAI-compatible) and Vertex AI inference providers +10/-0

Enable Gemini (OpenAI-compatible) and Vertex AI inference providers
• Adds a Gemini provider configured as 'openai' with a custom base_url pointing at Google's OpenAI-compatible endpoint and GEMINI_API_KEY. Adds a Vertex AI provider configured via VERTEX_AI_PROJECT and VERTEX_AI_LOCATION env expansions.
lightspeed-core-configs/lightspeed-stack.yaml

Documentation (1) +30 / -0

PROVIDERS.mdDocument Gemini provider via OpenAI-compatible endpoint and clarify Vertex AI setup +30/-0

Document Gemini provider via OpenAI-compatible endpoint and clarify Vertex AI setup

• Adds a Gemini section explaining the 'openai'-type provider workaround using a custom base_url and GEMINI_API_KEY. Expands Vertex AI documentation with compose-based credential mounting instructions and required roles/API enablement notes.

docs/PROVIDERS.md

Other (3) +14 / -1

compose.yamlMount Google application credentials into the container with a safe fallback +3/-0

Mount Google application credentials into the container with a safe fallback

• Adds a volume mount that maps a host credentials JSON (via GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH) into the container at /app-root/gcp-credentials.json. Uses an empty placeholder file when the host path is unset so compose startup does not fail.

compose/compose.yaml

default-values.envAdd Gemini and Vertex AI env defaults for key-based and file-based auth +10/-1

Add Gemini and Vertex AI env defaults for key-based and file-based auth

• Introduces GEMINI_API_KEY and replaces direct GOOGLE_APPLICATION_CREDENTIALS host-path usage with GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH plus an in-container GOOGLE_APPLICATION_CREDENTIALS default. Adds guidance comments to prevent misconfiguration of host vs container paths.

env/default-values.env

gcp-credentials-placeholder.jsonAdd empty placeholder credentials file for compose mounts +1/-0

Add empty placeholder credentials file for compose mounts
• Adds an empty JSON file used as a harmless default mount target when GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH is not set.
env/gcp-credentials-placeholder.json

qodo-code-review · 2026-07-16T16:24:12Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. ~~GitOps generator leaks providers~~ ✓ Resolved 🐞 Bug ≡ Correctness

Description

scripts/generate-gitops-manifests.sh only replaces the single - type: sentence_transformers line,
so adding Gemini and Vertex blocks immediately after it in lightspeed-stack.yaml causes the
generated GitOps ConfigMap to include those providers unintentionally. This can make generated
deployments expose/initialize Gemini/Vertex providers and fail at startup or request-time when the
new credentials/env vars are not configured.

Code

lightspeed-core-configs/lightspeed-stack.yaml[R31-40]

+    - type: openai
+      id: gemini
+      api_key_env: GEMINI_API_KEY
+      extra:
+        base_url: https://generativelanguage.googleapis.com/v1beta/openai/
+    - type: vertexai
+      id: vertexai
+      extra:
+        project: ${env.VERTEX_AI_PROJECT:=}
+        location: ${env.VERTEX_AI_LOCATION:=global}

Relevance

⭐⭐ Medium
Only related history: PR36 rejected changing awk next behavior; no direct precedent on “provider
leakage” concerns.
PR-#36

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The updated lightspeed-stack.yaml now contains gemini/vertexai blocks directly after
sentence_transformers. The generator’s awk hook only nexts the single sentence_transformers
line, then prints subsequent lines unchanged, so the new provider blocks will be included in the
generated ConfigMap output.

lightspeed-core-configs/lightspeed-stack.yaml[28-40]
scripts/generate-gitops-manifests.sh[56-92]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`scripts/generate-gitops-manifests.sh` injects providers by matching the `sentence_transformers` line, but it does not remove subsequent providers from the source file. Since `lightspeed-core-configs/lightspeed-stack.yaml` now includes `gemini` and `vertexai` immediately after `sentence_transformers`, the generated GitOps manifest will accidentally include those providers.

## Issue Context
The generator’s awk block triggers on `- type: sentence_transformers`, prints vLLM/OpenAI providers, then continues printing the rest of `lightspeed-stack.yaml` unchanged. With the new provider blocks placed directly after the trigger line, they will be emitted into the generated ConfigMap.

## Fix Focus Areas
- scripts/generate-gitops-manifests.sh[56-92]
- lightspeed-core-configs/lightspeed-stack.yaml[28-40]

## Suggested fix
Update the generator so it replaces the *entire* `inference.providers` section (or explicitly skips the original providers following the match) rather than only skipping the one `sentence_transformers` line. For example, have awk enter a “skip providers” mode after the match until it reaches the next top-level key (e.g., `user_data_collection:`), and print only the injected providers for GitOps output.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. Invalid GCP creds placeholder 🐞 Bug ☼ Reliability

Description

compose/compose.yaml defaults the credentials bind-mount to an {} placeholder file, while
default-values.env points GOOGLE_APPLICATION_CREDENTIALS at that mounted path. When Vertex AI is
used, Google auth will read an invalid credentials JSON, causing authentication/configuration
failures unless GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH is explicitly set to a real credentials
file.

Code

compose/compose.yaml[R26-28]

+      # Only used when the vertexai provider is configured. Falls back to a
+      # harmless empty placeholder so the mount doesn't fail otherwise.
+      - ${GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH:-../env/gcp-credentials-placeholder.json}:/app-root/gcp-credentials.json:Z,ro

Relevance

⭐⭐ Medium
No prior review evidence about placeholder creds; team often tweaks compose mounts, but this
specific pattern not seen.
PR-#16
PR-#36

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
compose/compose.yaml mounts /app-root/gcp-credentials.json from
${GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH} but falls back to a repo file when unset;
default-values.env points GOOGLE_APPLICATION_CREDENTIALS at that mount; and the fallback file
content is {}. This combination forces Google auth to read an invalid JSON file unless overridden.
compose/compose.yaml[21-33]
env/default-values.env[21-31]
env/gcp-credentials-placeholder.json[1-1]
lightspeed-core-configs/lightspeed-stack.yaml[28-40]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The compose setup always mounts a file at `/app-root/gcp-credentials.json`, defaulting to `env/gcp-credentials-placeholder.json` which contains `{}`. At the same time, `env/default-values.env` sets `GOOGLE_APPLICATION_CREDENTIALS=/app-root/gcp-credentials.json`, forcing Google auth to read the placeholder when users haven’t configured real credentials.

## Issue Context
This creates a confusing failure mode: Vertex AI usage will fail due to an invalid credentials JSON even though the environment variable is set and a file exists.

## Fix Focus Areas
- compose/compose.yaml[21-33]
- env/default-values.env[21-31]
- env/gcp-credentials-placeholder.json[1-1]

## Suggested fix
Avoid setting `GOOGLE_APPLICATION_CREDENTIALS` by default when the host credentials path is unset. Options:
1) Remove `GOOGLE_APPLICATION_CREDENTIALS` from `env/default-values.env` (leave it empty), and set it only in `env/values.env` when using Vertex.
2) Alternatively, set it in `compose/compose.yaml` using conditional interpolation (only set when `GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH` is non-empty), and keep the placeholder mount purely to prevent mount errors.
Also consider updating docs wording to reflect whichever behavior you choose.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

qodo-code-review · 2026-07-16T16:24:37Z

      - ../lightspeed-core-configs/rhdh-profile.py:/app-root/rhdh-profile.py:Z
      - ../llama-stack-configs/config.yaml:/app-root/config.yaml:Z
      - ../rag-content:/rag-content:Z
+      # Only used when the vertexai provider is configured. Falls back to a


2. Invalid gcp creds placeholder 🐞 Bug ☼ Reliability

compose/compose.yaml defaults the credentials bind-mount to an {} placeholder file, while default-values.env points GOOGLE_APPLICATION_CREDENTIALS at that mounted path. When Vertex AI is used, Google auth will read an invalid credentials JSON, causing authentication/configuration failures unless GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH is explicitly set to a real credentials file.

Agent Prompt

## Issue description The compose setup always mounts a file at `/app-root/gcp-credentials.json`, defaulting to `env/gcp-credentials-placeholder.json` which contains `{}`. At the same time, `env/default-values.env` sets `GOOGLE_APPLICATION_CREDENTIALS=/app-root/gcp-credentials.json`, forcing Google auth to read the placeholder when users haven’t configured real credentials. ## Issue Context This creates a confusing failure mode: Vertex AI usage will fail due to an invalid credentials JSON even though the environment variable is set and a file exists. ## Fix Focus Areas - compose/compose.yaml[21-33] - env/default-values.env[21-31] - env/gcp-credentials-placeholder.json[1-1] ## Suggested fix Avoid setting `GOOGLE_APPLICATION_CREDENTIALS` by default when the host credentials path is unset. Options: 1) Remove `GOOGLE_APPLICATION_CREDENTIALS` from `env/default-values.env` (leave it empty), and set it only in `env/values.env` when using Vertex. 2) Alternatively, set it in `compose/compose.yaml` using conditional interpolation (only set when `GOOGLE_APPLICATION_CREDENTIALS_HOST_PATH` is non-empty), and keep the placeholder mount purely to prevent mount errors. Also consider updating docs wording to reflect whichever behavior you choose.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Signed-off-by: Stephanie <yangcao@redhat.com>

These providers are now injected by generate-gitops-manifests.sh for the GitOps deployment and added via lightspeed-stack.local.yaml for local testing, so the tracked baseline stays in sync with origin/main. Co-authored-by: Cursor <cursoragent@cursor.com>

…ocal.yaml override Extends the existing GitOps script injection pattern (already used for vllm/openai) to also add gemini (config.yaml) and vertexai (lightspeed-stack.yaml), keeping the tracked baseline files in sync with origin/main. Adds a config.local.yaml override mechanism (mirroring lightspeed-stack.local.yaml) so gemini/vertexai/openai/vllm can still be tested locally, and updates CONTRIBUTING.md/PROVIDERS.md accordingly. Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Stephanie <yangcao@redhat.com>

Jdubrick

lgtm

qodo-code-review Bot reviewed Jul 16, 2026

View reviewed changes

yangcao77 changed the title ~~Add Gemini and Vertex AI inference providers~~ WIP: Add Gemini and Vertex AI inference providers Jul 16, 2026

yangcao77 added the hold/do-not-merge label Jul 16, 2026

yangcao77 and others added 2 commits July 21, 2026 15:35

enable google provider

1815d70

Signed-off-by: Stephanie <yangcao@redhat.com>

yangcao77 force-pushed the google-provider branch from df06e85 to e5d5512 Compare July 21, 2026 19:58

yangcao77 changed the title ~~WIP: Add Gemini and Vertex AI inference providers~~ Add Gemini and Vertex AI inference providers Jul 21, 2026

yangcao77 removed the hold/do-not-merge label Jul 21, 2026

yangcao77 requested review from Jdubrick and michael-valdron July 21, 2026 20:03

remove gemini, as we expect Lcore to support gemini in their config

3a9eff4

Signed-off-by: Stephanie <yangcao@redhat.com>

yangcao77 force-pushed the google-provider branch from 7f0bc5e to 3a9eff4 Compare July 21, 2026 21:04

Jdubrick reviewed Jul 21, 2026

View reviewed changes

Comment thread scripts/generate-gitops-manifests.sh

yangcao77 added 2 commits July 21, 2026 17:13

add vertex models

697cc46

Signed-off-by: Stephanie <yangcao@redhat.com>

add gemini 3

4333bea

Signed-off-by: Stephanie <yangcao@redhat.com>

Jdubrick approved these changes Jul 21, 2026

View reviewed changes

yangcao77 merged commit 58a1d5d into redhat-ai-dev:main Jul 21, 2026
3 checks passed

yangcao77 changed the title ~~Add Gemini and Vertex AI inference providers~~ Add Vertex AI inference providers Jul 21, 2026

qodo-code-review Bot mentioned this pull request Jul 22, 2026

Notebooks use PGVector, not FAISS #43

Merged

2 tasks

Navigation Menu

Uh oh!

Add Vertex AI inference providers - #41

Add Vertex AI inference providers#41
yangcao77 merged 6 commits into
redhat-ai-dev:mainfrom
yangcao77:google-provider

yangcao77 commented Jul 16, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Jul 16, 2026

Uh oh!

qodo-code-review Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

qodo-code-review Bot Jul 16, 2026

Uh oh!

Uh oh!

Jdubrick left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

yangcao77 commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

qodo-code-review Bot commented Jul 16, 2026

PR Summary by Qodo

Uh oh!

qodo-code-review Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

Uh oh!

qodo-code-review Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jdubrick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yangcao77 commented Jul 16, 2026 •

edited

Loading

qodo-code-review Bot commented Jul 16, 2026 •

edited

Loading