feat(cli): add terminal image display support using ink-picture #13883
Conversation
Add the ability to display images directly in the terminal when MCP tools return image data. This enables extensions like nanobanana to show generated images inline without requiring external viewers. Changes: - Add ink-picture dependency for cross-terminal image rendering - Create ImageDisplay component with self-contained TerminalInfoProvider - Add ImageData and ImageResult types to core tools module - Update MCP tool handler to detect and return image blocks - Integrate image display into ToolResultDisplay component Supported terminal protocols (auto-detected): - Kitty graphics protocol - iTerm2 inline images - Sixel graphics - Half-block/Braille/ASCII fallbacks BREAKING CHANGE: None
Summary of ChangesHello @ali-aljufairi, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces an experimental feature to render images directly in the terminal output of the CLI. By integrating the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an exciting proof-of-concept for displaying images directly in the terminal. The implementation using ink-picture is a great start. I've identified a few areas for improvement to enhance robustness and correctness before this feature is ready for merging. My main concerns are around handling of mixed-content responses from tools, which could lead to information loss in the UI, and a bug in the new ImageDisplay component where a prop is not being used. Addressing these points will make the feature more reliable.
| export const ImageDisplay: React.FC<ImageDisplayProps> = ({ | ||
| image, | ||
| width, | ||
| height, | ||
| }) => { | ||
| const { base64, filePath, mimeType, alt } = image; | ||
|
|
||
| // Determine the image source | ||
| let src: string | undefined; | ||
|
|
||
| if (base64) { | ||
| // Convert base64 to data URL | ||
| const mime = mimeType || 'image/png'; | ||
| src = `data:${mime};base64,${base64}`; | ||
| } else if (filePath) { | ||
| src = filePath; | ||
| } | ||
|
|
||
| if (!src) { | ||
| return ( | ||
| <Box> | ||
| <Text color="yellow">[Image: Unable to display - no valid source]</Text> | ||
| </Box> | ||
| ); | ||
| } | ||
|
|
||
| return ( | ||
| <TerminalInfoProvider> | ||
| <Box flexDirection="column"> | ||
| <Image | ||
| src={src} | ||
| width={width} | ||
| height={height} | ||
| alt={alt || 'Generated image'} | ||
| /> | ||
| {alt && ( | ||
| <Box marginTop={1}> | ||
| <Text dimColor>{alt}</Text> | ||
| </Box> | ||
| )} | ||
| </Box> | ||
| </TerminalInfoProvider> | ||
| ); | ||
| }; |
There was a problem hiding this comment.
The preserveAspectRatio prop is defined in ImageDisplayProps but is not being destructured from the props, and therefore not passed to the underlying Image component. This makes the prop non-functional, as it will always use the default behavior of ink-picture.
export const ImageDisplay: React.FC<ImageDisplayProps> = ({
image,
width,
height,
preserveAspectRatio,
}) => {
const { base64, filePath, mimeType, alt } = image;
// Determine the image source
let src: string | undefined;
if (base64) {
// Convert base64 to data URL
const mime = mimeType || 'image/png';
src = `data:${mime};base64,${base64}`;
} else if (filePath) {
src = filePath;
}
if (!src) {
return (
<Box>
<Text color="yellow">[Image: Unable to display - no valid source]</Text>
</Box>
);
}
return (
<TerminalInfoProvider>
<Box flexDirection="column">
<Image
src={src}
width={width}
height={height}
preserveAspectRatio={preserveAspectRatio}
alt={alt || 'Generated image'}
/>
{alt && (
<Box marginTop={1}>
<Text dimColor>{alt}</Text>
</Box>
)}
</Box>
</TerminalInfoProvider>
);
};
| if (imageBlock) { | ||
| // Collect any text content | ||
| const textParts = mcpContent | ||
| .filter((block): block is McpTextBlock => block.type === 'text') | ||
| .map((block) => block.text) | ||
| .join('\n'); | ||
|
|
||
| // Try to extract file path from text content (e.g., "• /path/to/image.png") | ||
| const filePathMatch = textParts.match( | ||
| /[•\-*]\s*([^\n]+\.(png|jpg|jpeg|gif|webp))/i, | ||
| ); | ||
| const filePath = filePathMatch ? filePathMatch[1].trim() : undefined; | ||
|
|
||
| const imageResult: ImageResult = { | ||
| image: { | ||
| // Prefer file path over base64 for better compatibility with ink-picture | ||
| filePath, | ||
| base64: filePath ? undefined : imageBlock.data, | ||
| mimeType: imageBlock.mimeType, | ||
| alt: textParts || undefined, | ||
| }, | ||
| }; | ||
| return imageResult; | ||
| } |
There was a problem hiding this comment.
The current logic for displaying images finds the first image block and immediately returns, causing any other non-text content blocks in the response to be ignored by the UI. This can lead to information loss if a tool returns an image alongside other important content like resource links.
To prevent this, consider processing the image only if it's the sole non-text content type. If other content types are present, you could fall back to the existing string-based display for all blocks to ensure no information is hidden from the user.
| // Try to extract file path from text content (e.g., "• /path/to/image.png") | ||
| const filePathMatch = textParts.match( | ||
| /[•\-*]\s*([^\n]+\.(png|jpg|jpeg|gif|webp))/i, | ||
| ); | ||
| const filePath = filePathMatch ? filePathMatch[1].trim() : undefined; |
There was a problem hiding this comment.
Extracting the file path from the text content using a regular expression is brittle as it relies on a specific text format. If the tool's text output for file paths changes, this logic will break.
A more robust approach would be for the MCP tool to provide the file path in a structured field within the image content block itself, avoiding the need for string parsing. While the server-side changes might be out of scope for this PR, this approach should be considered for a more resilient implementation.
1 participant
Add the ability to display images directly in the terminal when MCP tools return image data. This enables extensions like nanobanana to show generated images inline without requiring external viewers.
Changes:
Supported terminal protocols (auto-detected):
BREAKING CHANGE: None
Add the ability to display images directly in the terminal when MCP tools return image data. This enables extensions like nanobanana to show generated images inline without requiring external viewers.
Note: This is an experimental feature exploring terminal image display capabilities. We acknowledge that not all terminals support inline images, and there are known limitations.
Details
What this PR does:
ink-picturedependency for cross-terminal image renderingImageDisplaycomponent with self-containedTerminalInfoProviderImageDataandImageResulttypes to core tools moduleToolResultDisplaycomponentTerminal Protocol Support (via ink-picture):
Known Limitations & Issues:
Changes required in nanobanana extension (not in this PR):
To test this feature, the following modifications were made to the nanobanana MCP server locally:
types.ts- AddedimageDataandimageMimeTypefields toImageGenerationResponseimageGenerator.ts- Capture first image base64 data during generationindex.ts- Include image content block in MCP response:```typescript
content.push({
type: '''image''',
data: result.imageData,
mimeType: result.imageMimeType || '''image/png'''
});
```
Demo
2025-11-27_13-37-40.mov
Related Issues
How to Validate
Prerequisites:
Steps:
npm run build/nanobanana generate image of a sunsetExpected Results:
Demo:
Success Criteria for Future Development
Before this can be considered for merge:
Pre-Merge Checklist