Repository navigation
feat(agent): implement native multimodal autonomous sensory capabilities - #2304
Merged
Merged
Conversation
Re-bin
self-requested a review
March 21, 2026 05:35
Re-bin
added a commit
that referenced
this pull request
Mar 21, 2026
Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
tracy-e
added a commit
to tracy-e/nanobot
that referenced
this pull request
Mar 24, 2026
… onboard wizard Major upstream features merged (92 commits): - Streaming support with automatic fallback (cli, telegram) - WeChat (weixin) channel via ilinkai HTTP long-poll API - Command router refactor (nanobot/command/) — local commands migrated - Interactive onboard wizard (nanobot/cli/onboard.py) - Multimodal tool perception (PR HKUDS#2304) - ExecTool enable flag (PR HKUDS#1824) - /status command (PR HKUDS#1985) - Prompt cache optimization (PR HKUDS#1109) - Per-session concurrent locks (replacing global lock) - Mistral and OpenVINO providers - Cron run history tracking - WhatsApp outbound media support - Email IMAP retry hardening Local modifications preserved: - /clear, /compact, /skills, /model, /mcp commands (migrated to CommandRouter) - Runtime model switching and state persistence - Claude OAuth provider (claude_oauth_provider.py, claude_oauth_auth.py) - MemorySearchTool, SubagentConfig, memory_window (active) - Progress message suppression, ollama strip_model_prefix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sorker
pushed a commit
to sorker/nanobot
that referenced
this pull request
Mar 24, 2026
…rception Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
timhok
pushed a commit
to timhok/nanobot
that referenced
this pull request
Mar 29, 2026
…rception Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
MaoChen1980
pushed a commit
to MaoChen1980/nanobot-mg
that referenced
this pull request
May 18, 2026
…rception Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR implements "Native Multimodal Autonomous Sensory" capabilities, allowing the agent to natively fetch, understand, and process multimodal context blocks (e.g., images) directly via tool executions, without relying on external text-only transcriptions.
Key Enhancements
Standardized Content Blocks Exchange:
nanobot/agent/tools/base.pyandregistry.pyfromstrtoAnyto support returning lists of multimodal blocks.[{"type": "image_url", ...}, {"type": "text", ...}]structures.Sensory Output in Tools:
ReadFileTool(filesystem.py): Detects image MIME types. If an image is read, it encodes the raw bytes into base64 and returns a native image block instead of failing or reading as UTF-8.WebFetchTool(web.py): Prioritizes a streamingGETrequest to detect thecontent-type. If it detects an image, it captures it directly as a native image block, bypassing textual translation services (like Jina) that would otherwise strip the raw visual data.Sensory Injection & History Persistence:
ContextBuilder.add_tool_resultnow blindly accepts structured blocks and injects them natively into the LLM context.AgentLoop._save_turnto safely persist tool results. Base64 payloads inimage_urlblocks are automatically stripped and replaced with[image: <path>]placeholders before saving to.jsonlhistory, preventing context bloat while maintaining full multimodal perception for the live inference round.Prompt Awareness:
ContextBuilder._get_identityandSubagentManager._build_subagent_promptso the agent is consciously aware of its native multimodal perception capabilities and will proactively read visual resources when requested.This transforms the agent into a "general sensory entity" capable of direct visual reasoning over local files and web images!