DEV Community

Trust Mutemasango
Trust Mutemasango

Posted on

What's Better in yodaman-ctx 1.3.5

Better answers to plain questions, two more languages read by structure, and indexing that cannot fail silently. Each change measured, and the limits that remain stated plainly.


yodaman-ctx (ctx for short) is a command-line tool that lets you search and ask questions about your code. It indexes a project once, then finds code by its exact name or by what it does, and answers questions with the files and lines it used. Everything runs on your own machine.

Version 1.3.5 works through the places where ctx was weakest. Every change to search was scored on the test ctx uses on itself: 56 questions about ctx's own code, each paired with the files that answer it. A change stayed only if it made results better. Where something could not be fixed, this article says so.


First, a sharper ruler

Before improving search, it helped to measure it more precisely.

ctx's test asked one question per result: did the answer's file come back? That is generous. A file can have dozens of pieces, and a search can return the right file but the wrong piece of it, which is not much help when you are reading the result.

The test can now also ask a stricter question: did the piece that actually contains the answer come back? Each of the 56 questions stores a short pattern that the answering code or text must contain, and a result only counts if it matches.

The difference is striking:

Before the changes below Right file in the top 5 Right piece in the top 5
56 questions 86% 59%

ctx found the right file most of the time, but the exact piece you need noticeably less often. That second number is the one that matters to someone reading results, and it is the one every change was judged against. The same check is available in ctx eval for your own projects.


1. Finding the file a question is really about

The weakness: when ctx missed, it usually found the right part of the project but a neighbouring file. Asked which tool the chat agent calls to search code, for example, it returned other chat files instead of the tool itself.

What changed: search now also reads each piece's file path, in both of its halves. Keyword search matches words in the path. The chat agent's search tool lives in agent/tools/retrieve-knowledge-tool.ts, so a question about the agent's tool now matches its path. Common parts of paths, such as src and ts, appear almost everywhere and count for almost nothing, so they do not crowd results.

Meaning search sees the path too: each piece is now understood together with the name of the file it comes from.

The result, each step measured against the one before:

Before Paths in keyword search Paths in meaning search too
Right file first (MRR) 0.589 0.625 0.677
Right file in the top 5 86% 88% 89%
Right piece first (chunk MRR) 0.366 0.393 0.428
Right piece in the top 5 59% 61% 66%

The keyword step improved six questions and made one slightly worse; one of the improvements was a question ctx had missed entirely, about the chat agent's search tool. The meaning-search step raised every average again, but question by question it was more mixed: at the file level, ten questions improved and five got worse.


2. Java and C#, method by method

The weakness: ctx read five languages by their structure (TypeScript, JavaScript, Python, Go and Rust) and split everything else by size. For Java and C#, that meant a class could be cut into pieces in the middle of a method.

What changed: ctx now reads Java and C# by their structure too. Because almost all code in those languages lives inside a class, ctx steps inside: each method, constructor or property becomes its own piece, labelled with the class it belongs to. Its documentation comment (/** ... */ in Java, /// in C#) and its annotations or attributes (@Override, [Obsolete]) stay attached. The class header and fields are kept together, and the closing brace joins the last method, so no line is lost. Both styles of C# namespace are handled.

New tests confirm that, for both languages, every non-blank line of a file ends up in exactly one piece, and the existing tests for the five other languages still pass unchanged.

Because this changes how files are split, and file paths now shape the meaning search, ctx shows its rebuild warning, with the exact command, for existing projects after you upgrade.


3. Matching forms of the same word

The weakness: keyword search matched words exactly, so "authentication" in a question did not match "authenticate" in the code.

What changed: each word is now also stored by its stem, the part that related forms share: "authentication" and "authenticate" both give "authent", and "refreshing" gives "refresh". The exact word is still stored too, so exact matches work as before.

The result is a small gain: every average rose (right piece in the top 5: 66% to 70%), but question by question it was an even split, seven better and seven worse. It was kept because the averages improved and it costs nothing extra to index.

One thing it cannot do: link words that share nothing, such as "authenticate" and checkLogin. That is what meaning search is for.


4. Indexing that cannot quietly fail

This problem turned up while running the test above.

The problem: if the embedding model could not load (an interrupted download, for example), every piece of a project failed to embed. ctx then saved the project with no pieces at all, marked it as up to date, and reported success. The first sign of trouble came later, as a confusing error when you searched.

What changed: if no piece can be embedded, indexing now stops with a clear message naming the cause, saves nothing, and leaves any existing index for the project exactly as it was. A single piece that fails on its own is still skipped with a warning.


All together

On the 56 questions Before 1.3.5
Right file in the top 5 86% 91%
Right piece in the top 5 59% 70%

What stays the same, and why

  • Meaning search on very large projects is slow. With realistic amounts of text per piece, a meaning search over 200,000 pieces takes about 2 to 4 seconds; for a project of 25,000 pieces it takes about a third of a second. Four ways of storing or reading the data were tried. Three made no measurable difference. The fourth, a purpose-built table in sqlite-vec, took about 40% less time, but it returned different results from the exact search, for reasons not yet understood, and it doubles the space the data takes. It was not shipped.

  • Some platforms load data into memory. ctx's fast meaning search relies on an SQLite extension called sqlite-vec, which ships for Linux and macOS (x64 and arm64) and Windows (x64). Its newest test release covers the same platforms, so on others, such as Windows on ARM, ctx still works but loads the meaning-search data into memory. This depends on the sqlite-vec project, not on ctx.


The short version

ctx 1.3.5 measures search by whether you get the exact piece of code you need, not just the right file. Measured that way, the piece you need is now in the top five for 70% of questions instead of 59%, because file names count in both halves of search and related word forms match. Java and C# are read method by method. And indexing can no longer fail silently. Each search change was kept only because the test showed it helped; the language support and the indexing fix are covered by new tests.

What did not improve is stated plainly: speed on very large projects, and platforms that depend on another project.

After upgrading, rebuild each project's index once with ctx index <path> --force; ctx tells you which projects need it.

ctx is MIT-licensed and available at github.com/Yoda-Man/ctx.

Top comments (0)