[chengyongru]

Replace the single-stage MemoryConsolidator with a two-stage memory system — Consolidator (lightweight, per-turn) + Dream (heavyweight, cron-scheduled) — backed by git version control for full change history and rollback.

Motivation

The old MemoryConsolidator had three fundamental problems:

1. Coupled responsibilities — Compression and long-term memory updates happened in a single LLM call via tool_choice=forced on save_memory. Some providers don't support tool_choice; the LLM had to simultaneously summarize and rewrite the entire MEMORY.md, degrading quality as memory grew.

2. No incremental updates — Every consolidate required the LLM to output the full MEMORY.md content. Token cost grew linearly with memory size, and existing facts were easily lost.

3. No change history — Memory files were plain Markdown with no version tracking. Corrupt or unintended edits were irreversible.

What changed

Architecture

Consolidator (per-turn)              Dream (cron-scheduled)           GitStore (version control)
+----------------------------+       +----------------------------+   +---------------------------+
| token over budget → LLM    |       | Phase 1: analyze history    |   | dulwich-backed .git repo  |
| summarize evicted messages  |──────▶|   vs existing memory files  |   | auto_commit on Dream run  |
| → history.jsonl            |       | Phase 2: AgentRunner        |   | /dream-log: view changes  |
| (plain text, no tool_call)  |       |   + read_file/edit_file     |   | /dream-restore: rollback  |
+----------------------------+       |   → surgical incremental    |   +---------------------------+
                                     |     edit of memory files    |
                                     +----------------------------+

MemoryStore rewrite

• Decoupled from LLM calls into a pure file I/O layer
• history.jsonl replaces HISTORY.md with auto-incrementing cursor per entry
• Cursor-based tracking prevents re-processing
• Automatic compaction (max 1000 entries)

Consolidator simplified

• No longer calls save_memory via forced tool_choice
• Uses plain-text LLM summarization (tools=None) for full provider compatibility
• Prompt adds Solutions category for capturing trial-and-error workflows
• Priority ranking: user corrections > solutions > decisions > events > environment facts

Dream (new)

Two-phase processor, cron-scheduled (default: every 2 hours):

Phase	Description
Phase 1	Plain LLM call comparing history.jsonl against MEMORY.md / SOUL.md / USER.md, outputs [FILE] atomic fact lines
Phase 2	Delegates to AgentRunner with read_file / edit_file tools for surgical edits (up to 10 tool calls)

Cursor always advances to prevent re-processing.

GitStore (new)

• Wraps dulwich for zero-dependency git operations (no system git required)
• Auto-commits memory file changes after each Dream run
• .gitignore auto-generated to only track memory files
• Supports log(), diff_commits(), revert() for browsing and rollback

New commands

Command	Description
/dream	Manually trigger a Dream consolidation run
/dream-log [sha]	Show what the last Dream changed (diff of latest commit, or a specific commit)
/dream-restore [sha]	List recent commits, or revert memory files to a previous state

DreamConfig

class DreamConfig(Base):
    cron: str = "0 */2 * * *"      # Every 2 hours
    model: str | None = None       # Optional model override
    max_batch_size: int = 20       # Max history entries per run
    max_iterations: int = 10       # Max tool calls in Phase 2

CronService.register_system_job()

Supports idempotent registration of internal system jobs that survive gateway restarts.

Breaking changes

• HISTORY.md → history.jsonl: legacy history file is no longer read; existing content is not migrated
• SOUL.md, USER.md, MEMORY.md are now marked as "auto-managed by Dream — do not edit directly" in the system prompt and memory skill

Files changed

Core: memory.py (+586/-178), git_store.py (+299 new), loop.py, context.py, __init__.py
Config & Commands: schema.py, commands.py, builtin.py
Cron: cron/service.py
Skills: skills/memory/SKILL.md
Tests: +test_consolidator.py, +test_dream.py, +test_memory_store.py, +test_git_store.py, -test_memory_consolidation_types.py (+533/-478)

Test plan

• pytest tests/agent/test_memory_store.py — MemoryStore I/O, cursor, compaction
• pytest tests/agent/test_consolidator.py — Consolidator summarization logic
• pytest tests/agent/test_dream.py — Dream Phase 1/2, cursor advancement
• pytest tests/agent/test_git_store.py — GitStore init, commit, log, diff, revert
• pytest tests/agent/test_consolidate_offset.py — Session consolidation boundary selection
• pytest tests/agent/test_loop_consolidation_tokens.py — Token-budget-triggered consolidation
• Manual: send messages, verify history.jsonl entries are appended
• Manual: /dream triggers Dream, verify MEMORY.md/USER.md/SOUL.md updates
• Manual: /dream-log displays change log with diff
• Manual: /dream-restore lists commits and reverts to a previous state

[Re-bin]

I like this! @chengyongru

[Re-bin]

This PR is in a very good place now.

I pulled it down, merged the latest main, and went through it from both the code side and the product side. What I like here is that this does not just add “more memory features” - it turns memory into a cleaner system with clearer boundaries:

• short-term context is compressed by the Consolidator
• long-term memory is refined by Dream
• durable memory state lives in SOUL.md, USER.md, and memory/MEMORY.md
• historical archive moves to memory/history.jsonl
• memory changes become inspectable and reversible through built-in Dream commands

On top of that, I made a few follow-up improvements to keep the feature feeling natural and production-ready:

• hardened the legacy HISTORY.md -> history.jsonl migration path, including corner cases and partial-history preservation
• fixed the migration edge case where an empty pre-created history.jsonl could accidentally block legacy migration
• improved /dream-log and /dream-restore output so they are much more user-friendly and easier to trust
• added /dream, /dream-log, and /dream-restore into Telegram command registration and help output
• protected the system-managed Dream cron job so users can inspect it but not remove it by accident
• clarified Dream configuration semantics by moving toward intervalH and modelOverride, which read much closer to product language than implementation language
• renamed the internal gitstore module for a cleaner shape
• updated docs so the memory model, in-chat commands, and source-vs-stable release expectations are all easier to understand

What matters most to me is that the architecture stayed disciplined while the feature grew:

• the main loop only wires the system in; it does not absorb the memory logic
• migration stays in the store layer
• cron protection stays in the cron layer
• user-facing explanations stay in commands/docs
• versioning stays isolated behind GitStore

So this still feels like the simplest version of the right design, not the beginning of a pile of special cases.

From a user perspective, this is also the right kind of improvement: mostly invisible when you do not need it, but very powerful when you do. Existing users do not have to learn the internals to benefit from it, and the upgrade path is now much safer.

I also ran the test suite and the branch is green on my side:

• full test suite: 975 passed, 1 skipped
• targeted regression coverage was added for migration, Dream UX, cron protection, Telegram command exposure, and the new config semantics

At this point, I think this PR is merge-ready.