Skip to content

README.md

Latest commit

 

History

History
113 lines (81 loc) · 4.47 KB

README.md

File metadata and controls

113 lines (81 loc) · 4.47 KB
Raw

Embed Docsite (RAG)

This service embeds the OpenFn Documentation to a vector database. It downloads, chunks, processes metadata, embeds and uploads the documentation to a vector database. Two write targets are supported: Pinecone (the default) and Postgres with the pgvector extension.

Setup

Every environment maintains its own vector store, so there is no shared index to point at. Run this service to populate your own before using search_docsite.

Pinecone (default)

  1. Create an account on Pinecone and set up a free cluster.
  2. Add PINECONE_API_KEY and OPENAI_API_KEY to your .env file.

The service creates the docsite index if it does not already exist.

Postgres (opt-in)

  1. Configure POSTGRES_URL in your .env file, pointing at a server with the vector extension (pgvector) available.
  2. Pass "target": "postgres" in the payload (see Payload Reference below).

The docsite_batches / docsite_chunks tables and the vector extension are created by a versioned migration that Apollo applies at startup (or run bun run migrate yourself); no manual psql step is needed. This service does not migrate, and returns a 503 if the schema is missing. See the main README's Database section for details.

Usage - Embedding OpenFn Documentation

The vector database used here is Pinecone. To obtain the env variables follow these steps:

  1. Create an account on [Pinecone] and set up a free cluster.
  2. Obtain the URL and token for the cluster and add them to the .env file.
  3. You'll also need an OpenAI API key to generate embeddings.

With the CLI, returning to stdout:

openfn apollo embed_docsite tmp/payload.json

To run directly from this repo (note that the server must be started):

bun py embed_docsite

The payload is optional. With no --input, the service indexes all documentation using the defaults below; to customise it, pass a payload file:

bun py embed_docsite --input tmp/payload.json

A full run downloads the entire docs site and embeds several thousand chunks, so allow upwards of ten minutes.

Implementation

The service uses the DocsiteProcessor to download the documentation and chunk it into smaller parts. The DocsiteIndexer formats metadata, creates a new collection, embeds the chunked texts (OpenAI) and uploads them into the vector database (Pinecone).

The chunked texts can be viewed in tmp/split_sections.

Docs checkout

general_docs and adaptor_docs are read from a shallow git clone of OpenFn/docs, kept at services/embed_docsite/docsite_cache/ (gitignored). The first call in a process clones; every call after fetches and reset --hards onto the latest main.

The clone is blobless and sparse (--filter=blob:none --sparse, checked out to only docs/ and adaptors/). If the refresh fails and a checkout already exists, the existing copy is served and a warning is logged — the run only fails if there is no checkout to fall back on. git must be on PATH.

adaptor_functions is a single JSON file from a different repo (OpenFn/adaptors), fetched over plain HTTP.

Payload Reference

The write target is independent of the read backend (DOCSITE_SEARCH_BACKEND), so a Postgres batch can be built while Pinecone still serves search traffic.

The input payload is a JSON object. All parameters are optional:

{
    "target": "pinecone",             // 'pinecone' | 'postgres'. Defaults to pinecone. Chooses the write destination.
    "docs_to_upload": ["adaptor_docs", "general_docs", "adaptor_functions"], // Select from 3 types of documentation to upload
    "docs_to_ignore": ["job-examples.md", "release-notes.md"], // Titles of documents that should not be indexed
    "chunk_target_length": 1000,      // Target chunk size in characters
    "chunk_min_length": 700,          // Minimum chunk size before merging with the next split

    // Pinecone target only:
    "collection_name": "docsite-20250225",  // Namespace (defaults to the current timestamp)
    "index_name": "docsite", // Name of the index in the vector database (an index contains collections; defaults to docsite)
    "max_total_collections": 3, // The max number of collections to keep in the vector database. This will delete older collections by date.

    // Postgres target only:
    "keep_batches": 2                 // Number of recent complete batches to retain when pruning
}