Skip to content

feat(agent): implement native multimodal autonomous sensory capabilities - #2304

Merged
Re-bin merged 1 commit into
HKUDS:mainfrom
vandazia:feature/multimodal-sensory
Mar 21, 2026
Merged

Re-bin merged 1 commit into
HKUDS:mainfrom
vandazia:feature/multimodal-sensory

Conversation

@vandazia

Copy link
Copy Markdown
Contributor

This PR implements "Native Multimodal Autonomous Sensory" capabilities, allowing the agent to natively fetch, understand, and process multimodal context blocks (e.g., images) directly via tool executions, without relying on external text-only transcriptions.

Key Enhancements

  1. Standardized Content Blocks Exchange:

    • Upgraded tool return types in nanobot/agent/tools/base.py and registry.py from str to Any to support returning lists of multimodal blocks.
    • Tools can now return standard [{"type": "image_url", ...}, {"type": "text", ...}] structures.
  2. Sensory Output in Tools:

    • ReadFileTool (filesystem.py): Detects image MIME types. If an image is read, it encodes the raw bytes into base64 and returns a native image block instead of failing or reading as UTF-8.
    • WebFetchTool (web.py): Prioritizes a streaming GET request to detect the content-type. If it detects an image, it captures it directly as a native image block, bypassing textual translation services (like Jina) that would otherwise strip the raw visual data.
  3. Sensory Injection & History Persistence:

    • ContextBuilder.add_tool_result now blindly accepts structured blocks and injects them natively into the LLM context.
    • Updated AgentLoop._save_turn to safely persist tool results. Base64 payloads in image_url blocks are automatically stripped and replaced with [image: <path>] placeholders before saving to .jsonl history, preventing context bloat while maintaining full multimodal perception for the live inference round.
  4. Prompt Awareness:

    • Explicitly updated the guidelines in ContextBuilder._get_identity and SubagentManager._build_subagent_prompt so the agent is consciously aware of its native multimodal perception capabilities and will proactively read visual resources when requested.

This transforms the agent into a "general sensory entity" capable of direct visual reasoning over local files and web images!

@vandazia vandazia changed the title enhancement: implement native multimodal autonomous sensory capabilities feat(agent): implement native multimodal autonomous sensory capabilities Mar 20, 2026
@Re-bin
Re-bin self-requested a review March 21, 2026 05:35
Re-bin added a commit that referenced this pull request Mar 21, 2026
Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
@Re-bin
Re-bin merged commit 834f1e3 into HKUDS:main Mar 21, 2026
tracy-e added a commit to tracy-e/nanobot that referenced this pull request Mar 24, 2026
… onboard wizard

Major upstream features merged (92 commits):
- Streaming support with automatic fallback (cli, telegram)
- WeChat (weixin) channel via ilinkai HTTP long-poll API
- Command router refactor (nanobot/command/) — local commands migrated
- Interactive onboard wizard (nanobot/cli/onboard.py)
- Multimodal tool perception (PR HKUDS#2304)
- ExecTool enable flag (PR HKUDS#1824)
- /status command (PR HKUDS#1985)
- Prompt cache optimization (PR HKUDS#1109)
- Per-session concurrent locks (replacing global lock)
- Mistral and OpenVINO providers
- Cron run history tracking
- WhatsApp outbound media support
- Email IMAP retry hardening

Local modifications preserved:
- /clear, /compact, /skills, /model, /mcp commands (migrated to CommandRouter)
- Runtime model switching and state persistence
- Claude OAuth provider (claude_oauth_provider.py, claude_oauth_auth.py)
- MemorySearchTool, SubagentConfig, memory_window (active)
- Progress message suppression, ollama strip_model_prefix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sorker pushed a commit to sorker/nanobot that referenced this pull request Mar 24, 2026
…rception

Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
timhok pushed a commit to timhok/nanobot that referenced this pull request Mar 29, 2026
…rception

Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
MaoChen1980 pushed a commit to MaoChen1980/nanobot-mg that referenced this pull request May 18, 2026
…rception

Add native image content blocks for read_file and web_fetch, preserve the multimodal tool-result path through the agent loop, and keep session history compact with image placeholders. Also harden web_fetch against redirect-based SSRF bypasses and add regression coverage for image reads and blocked private redirects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants