Repository navigation
feat(capability): send screenshots as MCP image blocks instead of base64 text - #533
Merged
Merged
Conversation
… text
A WebDriver screenshot is `{"value": "<base64 png>"}` and a phone screen runs to
hundreds of kilobytes. Returned inside invokeCapability's JSON text block it was
paid for as text and still invisible to the model. A capability may now declare
`media: {mime, json_path}`; on a 2xx the string at that pointer is detached,
compressed the way the legacy screenshot tools compress theirs, and sent as a
second `image` content block, with a short note left in the body where it sat.
Error bodies and bodies with nothing at the path go back untouched.
Both infra products' take_screenshot carry it. The index is refused at load for a
mime no client renders or a path that is not a JSON pointer. maybeCompressBase64
moves to src/lib/image.ts so the registry can import it without pulling in the
server entry module; utils.ts re-exports it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
 |
- media.mime may only be image/png: the compressor re-encodes anything over 1 MB as PNG, so a JPEG declared would be attached as PNG under a JPEG label - the note, `image.bytes` and the telemetry report the size of the block as attached, after compression, not as received (received is kept as its own telemetry field) - after a fallback chain the step that answered decides whether there is an image to detach, since the 2xx body is that step's - checkMedia lives beside checkFallbacks in the loader and throws IndexError itself, so the try/catch rethrow goes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SavioBS629
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
take_screenshot(web and mobile) returns{"value": "<base64 png>"}. The registry parsed that JSON whole and put it in the tool result's text block, so a phone screenshot cost hundreds of KB of base64 as text and the model still could not see the picture. Raised as an eval finding against Device Control.What
media: {mime, json_path}. On a 2xx,invokeCapabilitylifts the string at that JSON pointer out of the body, compresses it with the samemaybeCompressBase64the legacy screenshot tools use, and returns it as a secondimagecontent block. The JSON block keeps a note in its place ("<image/png, 35 KB, attached to this result as an image>") and animage: {mime, bytes}field. Error bodies and bodies with nothing at the path are returned unchanged.take_screenshotrecords declaremediaand their intent/guidance no longer promise base64 text.mimemust be one a client renders (png/jpeg/gif/webp) andjson_pathmust be a JSON pointer.maybeCompressBase64moves tosrc/lib/image.ts(re-exported fromutils.ts) so the registry can import it without pulling in the server entry module.Remote
Nothing new for remote-mcp-server:
takeAppScreenshotalready ships an image block through the hosted server, the compressor already has its REMOTE_MCP branch (libvips cache off, single thread), and the response size ceiling is the same 1 MB.Verified
capabilityRegistryMedia.test.ts.take_screenshotcame back as text + image, the model described the page content,valuecarried the note andimage.byteswas 35651.🤖 Generated with Claude Code