Adding Design Principles #596
Conversation
k8s-ci-robot
requested review from
ahg-g,
danehans,
Jeffwan,
kfswain,
shaneutt and
smarterclayton
k8s-ci-robot
added
cncf-cla: yes
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
k8s-ci-robot
added
the
do-not-merge/hold
| ## Encourage innovation via extensibility | ||
|
|
||
| This project is largely based on the idea that extensibility will enable innovation. With that in mind, we should make it as easy as possible for AI researchers to experiment with custom scheduling and routing logic. They should not need to know how to build a Kubernetes controller, or replicate a full networking stack. Instead, all the information needed to make a routing decision should be provided in an accessible format, with clear guidelines and examples of how to customize routing logic. |
There was a problem hiding this comment.
make it as easy as possible for AI researchers to experiment with custom scheduling...
Do you and others have ideas on how to achieve this goal? Should we make the scheduler extensible or define a scheduling API that EPP and potentially other extensions call?
There was a problem hiding this comment.
I think we're seeing this succeed with llm-d and the more recent #845.
k8s-ci-robot
removed
the
do-not-merge/hold
robscott
force-pushed
the
design-principles
branch
from
5127127 to
7e469c9
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: robscott, shaneutt The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
|
|
||
| ## Composable components and reducing reinvention | ||
| While it may be tempting to develop an entirely new AI-focused Gateway, many essential routing capabilities are already well established by Kubernetes. Our focus is on creating a layer of composable components that can be assembled together with other Kubernetes components. This approach empowers engineers to use our solution as a building block—combining established technologies like Gateway API with our extensible model to build higher level solutions. |
There was a problem hiding this comment.
Consider rewording, perhaps this is personal bias, but the first sentence reads as if to ward off the reader from attempting to implement something new.
The later sentences focus on the value of using what has already been built, which I think is what we are going for. Perhaps move the concept of the first sentence to the end as something like:
Should you encounter a limitation, consider how existing tooling may be extended or improved first. Suggestions always welcomed (and encouraged) at our: sync-link-goes-here.
| ## Encourage innovation via extensibility | ||
|
|
||
| This project is largely based on the idea that extensibility will enable innovation. With that in mind, we should make it as easy as possible for AI researchers to experiment with custom scheduling and routing logic. They should not need to know how to build a Kubernetes controller, or replicate a full networking stack. Instead, all the information needed to make a routing decision should be provided in an accessible format, with clear guidelines and examples of how to customize routing logic. |
There was a problem hiding this comment.
make it as easy as possible for AI researchers to experiment with custom scheduling...
Do you and others have ideas on how to achieve this goal? Should we make the scheduler extensible or define a scheduling API that EPP and potentially other extensions call?
|
@robscott checking in to see what your plans are for this PR. |
robscott
force-pushed
the
design-principles
branch
from
7e469c9 to
f2c526e
Compare
Sorry for the delay here, finally got some time to update this, PTAL. |
| [Gateway API]:https://github.com/kubernetes-sigs/gateway-api | ||
| [AI Inference]:https://www.arm.com/glossary/ai-inference | ||
|
|
||
|
|
| The pace of innovation in this ecosystem has been rapid. Focusing too heavily on the specifics of current techniques could result in the API becoming outdated quickly. Instead of making the API too descriptive about _how_ an objective should be achieved, this API should focus on the objectives that a Gateway and/or Endpoint Picker should strive to attain. Overly specific instructions or configuration can start as implementation specific APIs and grow into standards as the concepts become more stable and widespread. | ||
|
|
||
| ## Composable components and reducing reinvention | ||
| While it may be tempting to develop an entirely new AI-focused Gateway, many essential routing capabilities are already well established by Kubernetes. Our focus is on creating a layer of composable components that can be assembled together with other Kubernetes components. This approach empowers engineers to use our solution as a building block—combining established technologies like Gateway API with our extensible model to build higher level solutions. Should you encounter a limitation, consider how existing tooling may be extended or improved first. |
There was a problem hiding this comment.
nit: Add a space after the heading.
| At a high level, this defines how a Gateway provides information to an Endpoint Picker, and how the Endpoint Picker selects endpoint(s) that the Gateway should route to. | ||
|
|
||
| ### 2. Endpoint Picker -> Model Server Framework | ||
| This defines what an Endpoint Picker should expect from a compatible Model Server Framework with a focus on health checks and metrics. |
There was a problem hiding this comment.
nit: Add a space after the heading.
| When describing "core interfaces", we are referring to both of the following: | ||
|
|
||
| ### 1. Gateway -> Endpoint Picker | ||
| At a high level, this defines how a Gateway provides information to an Endpoint Picker, and how the Endpoint Picker selects endpoint(s) that the Gateway should route to. |
There was a problem hiding this comment.
nit: Add a space after the heading.
|
A few nits and CI is failing. |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/retest |
As the project continues to grow, it would be helpful to have some high level design principles for the project. These principles can help guide us when determining which features and work to prioritize.
/cc @ahg-g @smarterclayton @danehans @kfswain @Jeffwan @shaneutt