Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CodeGate doesn't include reasoning for OpenRouter thinking models #977

Open
jhrozek opened this issue Feb 7, 2025 · 3 comments
Open

CodeGate doesn't include reasoning for OpenRouter thinking models #977

jhrozek opened this issue Feb 7, 2025 · 3 comments
Labels
bug

Comments

@jhrozek
Copy link
Contributor

jhrozek commented Feb 7, 2025

It's possible to follow the reasoning of the thinking models like DeepSeek through OpenRouter, see https://openrouter.ai/announcements/reasoning-tokens-for-thinking-models

However, that doesn't work through codegate it seems.
@jhrozek jhrozek added the bug label Feb 7, 2025
@jhrozek
Copy link
Contributor Author

jhrozek commented Feb 7, 2025 •
edited
Loading

Looks like we need to make sure that the "include_reasoning": true flag is included in the request. Hopefully, litellm will not strip the request or reply.

@jhrozek
Copy link
Contributor Author

jhrozek commented Feb 7, 2025

so on our side, the fix is simple, but won't work until BerriAI/litellm#8193 is fixed.

@jhrozek
Copy link
Contributor Author

jhrozek commented Feb 7, 2025

parking

@jhrozek jhrozek mentioned this issue Feb 10, 2025
Closed
jhrozek added a commit that referenced this issue Feb 19, 2025
@jhrozek
Pass through reasoning input flag and output chunks
4e6a204 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
@jhrozek jhrozek mentioned this issue Feb 19, 2025
Merged
blkt pushed a commit that referenced this issue Feb 19, 2025
@jhrozek
Pass through reasoning input flag and output chunks (#1096)
4e0c6f3 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 19, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
b851b11 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 19, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
d9788c0 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 20, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
1a1a047 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 21, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
ceed272 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 21, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
eae88d0 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 21, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
2a1b3f1 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Feb 25, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
5aa1213 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
blkt pushed a commit that referenced this issue Mar 2, 2025
@jhrozek @blkt
Pass through reasoning input flag and output chunks (#1096)
3f03576 
Some OpenRouter models allow to include the reasoning output if told to
do so with a special input flag. This seems to be specific to
OpenRouter.

A simple test is:
```
curl http://localhost:8989/openrouter/completions \
 -H "Authorization: Bearer $OPENROUTER_KEY" \
 -H "Content-Type: application/json" \
 -d '{
   "model": "deepseek/deepseek-r1",
   "messages": [
     {"role": "user", "content": "What is heavier, a kilogram of iron or a kilogram of feathers?"}
   ],
   "include_reasoning": true,
   "stream": true
 }'
```

Fixes: #977
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jhrozek @lukehinds