Add SHA256-CBOR hashing algorithm for token processor with extra keys…

[leipanhz]

Add KV-cache file prefetch plugin for inference requests (experimental feature)
Part 1: changes in KV-Cache (current PR)
Part 2: changes in llm-d-router

PR Description:
Introduces a new experimental feature that aims to proactively prefetch KV-cache blocks across different storage tiers before inference requests are processed by the GPU pod. The plugin extends the precise prefix cache scorer with engine key calculation to determine the storage location (file names) of KV-cache blocks that will be needed and arrange for them to be promoted to a closer storage tier to improve inference latency. The current implementation is intended for a shared file system that includes transparent access to a remote storage tier, such as IBM Storage Scale configured to off-load cold data to remote object storage. The prefetch plugin uses a concurrent worker thread pool architecture to efficiently prefetch multiple (configurable) files in parallel from remote storage to the shared file system. In a future version of the plugin this could be extended, for example, to prefetch KV-cache blocks from the file system to CPU memory on the worker node that the request is being routed to.

For this to work correctly, the plugin must be configured to use a hash algorithm for generating engine keys that matches the algorithm used by vLLM when offloading KV-cache blocks to storage. For this purpose, this work adds a configurable hashing algorithm SHA256-CBOR to the token processor as an alternative for vLLM compatibility. The SHA256-CBOR implementation supports extra keys (multimodal features) in block hash computation. In addition, this feature relies on logic derived from the llm-d-fs-connector to generate KV file names, so it currently only works with the llm-d-fs-connector.

Changes include:

New Prefetch Plugin (prefetch_prerequest_experimental.go):

• Implements PreRequest interface for pre-inference file prefetching
• Converts engine keys to filesystem paths using llm-d-fs-connector format
• Manages worker thread pool for concurrent file prefetching (configurable workers)
• Each worker reads configurable number of blocks (BlockSize x BlockCount bytes) from KV-cache files to trigger prefetch of the rest of the file from remote storage.
• Supports configurable prefetch parameters (block size, concurrency, queue size)

Precise Prefix Cache Scorer Enhancement (precise_prefix_cache.go):

• Add GetEngineKeysForRequest() method to extract engine keys from requests
• Support multimodal features in engine key computation

Add SHA256-CBOR hashing algorithm for token processor with extra keys support

• Add configuration to choose hashing function via the field name “hashAlgorithm”: FNV64a default, SHA256-CBOR for vLLM
• Implement SHA256-CBOR hashing matching vLLM engine-key computation
• Extend BlockExtraFeatures for multimodal content support

[github-actions]

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[github-actions]

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

[leipanhz]

Related issue: llm-d/llm-d-router#866