Skip to content

Commit 4c19a41

Browse files
danielholandadanielholanda
authored
Early preview of new server interface (#277)
1 parent 81708d4 commit 4c19a41

File tree

4 files changed

+720
-1
lines changed

4 files changed

+720
-1
lines changed

‎docs/lemonade/server_spec.md‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
# Lemonade Server Spec (Preview)
2+
3+
> This is a preview release. The API specification is subject to change.
4+
5+
This spec was inspired by the [LM Studio REST API](https://lmstudio.ai/docs/api-reference/rest-api), [Ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md), and [OpenAI API](https://platform.openai.com/docs/api-reference/introduction).
6+
7+
This spec focuses on enabling client applications by extending existing cloud-focused APIs (e.g., OpenAI) to also include the ability to load and unload models before completion requests are made. These extensions allow for a greater degree of UI/UX responsiveness in native applications by allowing applications to:
8+
- Pre-load models at UI-loading-time, as opposed to completion-request time.
9+
- Load models from the local system that were downloaded by other applications (i.e., a common system-wide models cache).
10+
- Unload models to save memory space.
11+
12+
## API Endpoints
13+
14+
- POST `/api/v0/completions` - Text Completions (prompt -> completion)
15+
- POST `/api/v0/load` - Load a model
16+
- POST `/api/v0/unload` - Unload a model
17+
- POST `/api/v0/params` - Set generation parameters
18+
- GET `/api/v0/health` - Check server health
19+
- GET `/api/v0/stats` - Performance statistics from the last request
20+
- GET `/api/v0/models` - List available models
21+
22+
> 🚧 We are in the process of developing this interface. Let us know what's important to you on Github or by email (turnkeyml at amd dot com).
23+
24+
## Start the REST API server
25+
26+
First, install lemonade with your desired backend (e.g., `pip install lemonade[llm-oga-cpu]`). Then, run the following command to start the server:
27+
28+
```bash
29+
lemonade server-preview
30+
```
31+
32+
## Endpoints
33+
34+
### `POST /api/v0/completions` <sub>![Status](https://img.shields.io/badge/status-partially_available-green)</sub>
35+
36+
Text Completions API. You provide a prompt and receive a streamed completion. This API will also load the model if it is not already loaded.
37+
38+
### Parameters
39+
40+
| Parameter | Required | Description | Status |
41+
|-----------|----------|-------------|--------|
42+
| `prompt` | Yes | The prompt to use for the completion. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
43+
| `model` | No | The model to use for the completion. If not specified, the server will use the default loaded model. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
44+
| All other params of `/api/v0/load` | No | Detailed loading options as defined in the `/api/v0/load` endpoint. | <sub>![Status](https://img.shields.io/badge/WIP-yellow)</sub> |
45+
| All other params of `/api/v0/params` | No | Detailed generation options as defined in the `/api/v0/params` endpoint. | <sub>![Status](https://img.shields.io/badge/WIP-yellow)</sub> |
46+
47+
### Example request
48+
49+
```bash
50+
curl http://localhost:1234/api/v0/completions \
51+
-H "Content-Type: application/json" \
52+
-d '{
53+
"model": "<HUGGINGFACE_CHECKPOINT>",
54+
"prompt": "the meaning of life is",
55+
}'
56+
```
57+
58+
### Response format
59+
60+
```json
61+
{
62+
"text": " to find your purpose, and once you have",
63+
}
64+
```
65+
66+
### `GET /api/v0/load` <sub>![Status](https://img.shields.io/badge/status-fully_available-green)</sub>
67+
68+
Explicitly load a model. This is useful to ensure that the model is loaded before you make a request.
69+
70+
### Parameters
71+
72+
| Parameter | Required | Description |
73+
|-----------|----------|-------------|
74+
| `model` | Yes | HuggingFace checkpoint to load. |
75+
| `device` | No | Device to load the model on. Defaults to `hybrid`. |
76+
| `dtype` | No | Data type to load the model on. Defaults to `int4`. |
77+
| `cache_dir` | No | Parent directory where models are stored. Defaults to `~/.cache/lemonade`. |
78+
79+
### Example request
80+
81+
```bash
82+
curl http://localhost:1234/api/v0/load \
83+
-H "Content-Type: application/json" \
84+
-d '{
85+
"model": "<HUGGINGFACE_CHECKPOINT>",
86+
"cache_dir": "/Users/your_username/models"
87+
}'
88+
```
89+
90+
### Response format
91+
92+
```json
93+
{
94+
"status": "success",
95+
"message": "Model loaded successfully"
96+
}
97+
```
98+
In case of an error, the status will be `error` and the message will contain the error message.
99+
100+
101+
### `POST /api/v0/unload` <sub>![Status](https://img.shields.io/badge/status-fully_available-green)</sub>
102+
103+
Explicitly unload a model. This is useful to free up memory and disk space while still leaving the server runnning (which takes minimal resources but a few seconds to start).
104+
105+
### Parameters
106+
107+
This endpoint does not take any parameters.
108+
109+
### Example request
110+
111+
```bash
112+
curl http://localhost:1234/api/v0/unload
113+
```
114+
115+
### Response format
116+
117+
```json
118+
{
119+
"status": "success",
120+
"message": "Model unloaded successfully"
121+
}
122+
```
123+
In case of an error, the status will be `error` and the message will contain the error message.
124+
125+
### `POST /api/v0/params` <sub>![Status](https://img.shields.io/badge/status-in_development-yellow)</sub>
126+
Set the generation parameters for text completion. These parameters will persist across requests until changed.
127+
128+
### Parameters
129+
130+
| Parameter | Required | Description |
131+
|-----------|----------|-------------|
132+
| `temperature` | No | Controls randomness in the output. Higher values (e.g. 0.8) make the output more random, lower values (e.g. 0.2) make it more focused and deterministic. Defaults to 0.7. |
133+
| `top_p` | No | Controls diversity via nucleus sampling. Keeps the cumulative probability of tokens above this value. Defaults to 0.95. |
134+
| `top_k` | No | Controls diversity by limiting to the k most likely next tokens. Defaults to 50. |
135+
| `min_length` | No | The minimum length of the generated text in tokens. Defaults to 0. |
136+
| `max_length` | No | The maximum length of the generated text in tokens. Defaults to 2048. |
137+
| `do_sample` | No | Whether to use sampling (true) or greedy decoding (false). Defaults to true. |
138+
139+
### Example request
140+
141+
```bash
142+
curl http://localhost:1234/api/v0/params \
143+
-H "Content-Type: application/json" \
144+
-d '{
145+
"temperature": 0.8,
146+
"top_p": 0.95,
147+
"max_length": 1000
148+
}'
149+
```
150+
151+
### Response format
152+
153+
```json
154+
{
155+
"status": "success",
156+
"message": "Generation parameters set successfully",
157+
"params": {
158+
"temperature": 0.8,
159+
"top_p": 0.95,
160+
"top_k": 40,
161+
"min_length": 0,
162+
"max_length": 1000,
163+
"do_sample": true
164+
}
165+
}
166+
```
167+
In case of an error, the status will be `error` and the message will contain the error message.
168+
169+
### `GET /api/v0/health` <sub>![Status](https://img.shields.io/badge/status-fully_available-green)</sub>
170+
171+
Check the health of the server. This endpoint will also return the currently loaded model.
172+
173+
### Parameters
174+
175+
This endpoint does not take any parameters.
176+
177+
### Example request
178+
179+
```bash
180+
curl http://localhost:1234/api/v0/health
181+
```
182+
183+
### Response format
184+
185+
```json
186+
{
187+
"status": "ok",
188+
"model_loaded": "<HUGGINGFACE_CHECKPOINT>"
189+
}
190+
```
191+
### `GET /api/v0/stats` <sub>![Status](https://img.shields.io/badge/status-fully_available-green)</sub>
192+
193+
Performance statistics from the last request.
194+
195+
### Parameters
196+
197+
This endpoint does not take any parameters.
198+
199+
### Example request
200+
201+
```bash
202+
curl http://localhost:1234/api/v0/stats
203+
```
204+
205+
### Response format
206+
207+
```json
208+
{
209+
"time_to_first_token": 2.14,
210+
"tokens_per_second": 33.33,
211+
"input_tokens": 128,
212+
"output_tokens": 5,
213+
"decode_token_times": [0.01, 0.02, 0.03, 0.04, 0.05]
214+
}
215+
```
216+
217+
### `GET /api/v0/models` <sub>![Status](https://img.shields.io/badge/status-in_development-yellow)</sub>
218+
219+
List all available models.
220+
221+
### Parameters
222+
223+
| Parameter | Required | Description |
224+
|-----------|----------|-------------|
225+
| `cache_dir` | No | Parent directory where models are stored. Defaults to `~/.cache/lemonade`. |
226+
227+
### Example request
228+
229+
```bash
230+
curl http://localhost:1234/api/v0/models \
231+
-H "Content-Type: application/json" \
232+
-d '{
233+
"cache_dir": "/Users/your_username/models"
234+
}'
235+
```
236+
237+
### Response format
238+
239+
```json
240+
{
241+
"data": [
242+
{
243+
"checkpoint": "<HUGGINGFACE_CHECKPOINT>",
244+
"device": "cpu",
245+
"dtype": "bfloat16",
246+
},
247+
{
248+
"checkpoint": "<ANOTHER_HUGGINGFACE_CHECKPOINT>",
249+
"device": "cpu",
250+
"dtype": "bfloat16",
251+
}
252+
]
253+
}
254+
```

‎src/lemonade/cli.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
from lemonade.tools.humaneval import AccuracyHumaneval
2323
from lemonade.tools.perplexity import AccuracyPerplexity
2424
from lemonade.tools.chat import LLMPrompt, Serve
25+
from lemonade.tools.serve import ServerPreview
2526

2627

2728
def main():
@@ -44,6 +45,7 @@ def main():
4445
Cache,
4546
Version,
4647
SystemInfo,
48+
ServerPreview,
4749
]
4850

4951
# Import onnxruntime-genai recipes

0 commit comments

Comments
 (0)