Feature Request: Configurable request parallelism for model definition

[yossig]

Problem

Calling LLMs from Flock functions like llm_complete, llm_filter, and llm_embedding is bound by long-running, high-latency IO to the LLM provider.
batch_size helps improve throughput, but in practice it has diminishing returns as batches grow (especially with smaller/faster models). At some point, increasing batch size no longer improves total throughput and may even hurt latency/efficiency depending on provider behavior.

This creates a throughput ceiling for large datasets and makes end-to-end runtime too long for many analytical workloads.

A dedicated parallelism control would allow fan-out of requests per DataChunk, significantly improving throughput (often by an order of magnitude depending on provider/model/setup), while keeping user control over resource usage and provider rate limits.

Proposal: Add a Model-Level parallelism Configuration

Suggested Interface Changes

• Introduce an optional integer argument (parallelism) to model definitions, e.g.:

  CREATE MODEL(
    'my-model',
    'gpt-4o',
    'openai',
    {"batch_size": 16, "parallelism": 4}
  );

• If not set, default to serial execution (parallelism=1).
• When set to N>1, Flock function batching logic should process up to N batches concurrently per data chunk, for the relevant functions:
  • llm_complete
  • llm_filter
  • llm_embedding

---

Additional points to consider

1. Function coverage - Important for llm_complete, llm_filter, and llm_embedding, could also be supported for agg functions, but probably less critical
2. Ordering - Output row ordering must be preserved deterministically.
3. Error handling - Clear handling of provider errors/timeouts/rate limits under parallel execution. Retry/backoff behavior should remain safe and predictable.

Implementation Notes

• The current provider handler layer already supports executing multiple HTTP requests in parallel using libcurl's multi-handle. For an easy win, the batching logic in scalar function implementations can be refactored to enqueue up to parallelism batch requests before invoking the provider handler's CollectCompletions, collect as they complete, then re-order by batch index
• A more robust and cleaner approach could be to introduce provider-level async/multiplexed request handling by adding a submission API plus completion polling/collection, then execute requests with bounded async HTTP concurrency controlled by parallelism.

[anasdorbani]

Hi @yossig, thanks for the detailed explanation. That would definitely be very helpful. We’ve already started implementing it in #167, which is why the handler supports parallelism by adding requests and collecting the completions at the end. We’ll be releasing it soon and will keep you posted. Thanks again, and we’d be very happy to hear if you have any other suggestions.

[yossig]

Thanks for taking this on, waiting for support for this to be merged and available.

Quick note after reviewing the PR you provided, and related to this feature request.
Beside the need to support concurrent async requests, it's also important to create a knob to control the degree of concurrency. This current approach does not seem to expose a bounded max in-flight control at the model level, as I suggested in this request. This means flock could generate unlimited bursts calls to the provider that can easily hit rate limits and/or timeouts on the other end and.

Thinking about it a bit more, this is also a bit tricky because while the reasonable place to have this knob to control parallelism / max in-flight requests would be in the model definition, the real limitation is globally on the provider level. But probably controlling it on the model level would be a reasonable enough compromise to be able and tune it per query.

So my suggestion would be:

• As I suggest in the FR: Add a model-level knob such as parallelism / max_in_flight_requests and pass that arg down to handler
• Enforce it centrally in the shared handler/transport layer rather than in each UDF (probably in BaseModelProviderHandler::ExecuteBatch - This can be by actually building a small internal queueing mechanism to feed in requests in batches, or get a quick win with a "poor man's version" and use CURLMOPT_MAX_HOST_CONNECTIONS or CURLMOPT_MAX_TOTAL_CONNECTIONS)
• This is probably a feature of it's own, but optionally consider add retry/backoff handling to failed requests

That would keep the model-config paradigm intact, make behavior consistent across functions, and provide a safer foundation for hosted APIs and local providers alike. Overall though: really appreciate the work here — feels like a strong base to continue from.

[anasdorbani]

Thanks @yossig for the clear explanation. The PR is a bit outdated since we’ve made many changes to improve how concurrency control is handled. The main challenge we’re working through is properly handling each provider’s API maximum request limits, but we should be able to figure out a solid approach. Your suggestions will definitely help shape the next version of the API request parallelism.

Thanks again, and feel free to share more ideas or even contribute to the project.