Superseded: this model is from an earlier Jeff release and is no longer updated. The current release is Jeff v1.3: mstrasser/jeff-adapter-nav. This v1.2 adapter loads only on the v1.2 base.

Jeff-Qwen3.5-0.8B-nav

Voice navigation: turns a spoken command, even a misheard one, into an item on the current app screen, a question, or none of these.

Give it what speech recognition heard, the current screen and every item on that screen. It picks the item the user meant, matching by meaning and by sound, or says the user is asking a question, or that they want something not on the screen. Training screens mostly had 40 to 120 items. It is a LoRA adapter for Jeff-Qwen3.5-0.8B v1.2, a small open decision model: it answers in one forward pass, with no generated text to parse.

Results

Test set: Nav held-out apps, 3,300 rows. Requests from about 10% of the roughly 340 generated apps, never trained on. It is included in this repo as test.jsonl.

Calibration error is the expected calibration error over 15 confidence bins, after each model's fitted temperature: the average gap between how confident the model is and how often it is right. Lower is better. The untrained Qwen3.5-0.8B is read out from its own scores for the option letters.

Against Qwen3.8-27B

Here Jeff + nav answers first. Below a confidence threshold, the query is also sent to Qwen3.8-27B and the 27B's answer is used. Each adapter's threshold is the fastest one that still beats the 27B alone by at least 1 point of accuracy on the adapter's calibration rows (not the test rows).

The threshold is 0.00: on the calibration rows the adapter alone already beat the 27B by more than a point, so no query is passed on. The route is 34.6× faster than the 27B alone.

How this was measured: a fixed random sample of 300 held-out test rows, the same rows for both routes, on an Apple M4 Max (128 GB) with both models in MLX, one at a time. Qwen3.8-27B ran with 8-bit weights, prompted, with step-by-step reasoning off. Times are the mean per query over the whole sample, from prompt to answer. This is a 300-row sample, not the full test set.

Speed and memory

One request at a time on an NVIDIA RTX PRO 6000: 675 requests per setting, 75 test prompts of each of the nine adapters, interleaved. Measured for the adapters as a group, not for each adapter, so treat these as typical figures:

Jeff's speed and memory on the Mac: not measured yet.

Where it goes wrong

On the full test set:

• By far the commonest mistake is picking a screen item when the user wanted something that is not on the screen (none_of_these). If acting on a wrong item is costly, check the confidence before acting.
• Questions (ask_question) were all recognised on this test set.
• Some test rows use different wordings and keys for the two fixed options (for example nothing_matches), as in the format pools used in training; a few single-row errors come from those.

Commonest confusions (right answer → chosen instead, rows):

Accuracy by kind of answer:

("An item on the screen" groups all rows whose answer is a screen item; per-item rows are too few to report.)

When to use it

• Your app takes voice commands and you want the item on the current screen the user meant, matched by meaning and by sound.
• You can list every item on the screen as an option. Training screens mostly had 40 to 120 items, some up to 200, and dialogues 4 to 8.
• You also need to tell a question, or a request the screen does not offer, apart from a navigation command.

When not to use it

• The user's words exactly match an item's name. Match those in code first; training left such cases out.
• You need the command carried out or an answer to the user's question. Jeff only picks an option.
• You want one app's best accuracy and have data from that app. A separate full fine-tune on one app's own screens (a different model) scored 95.2% on navigation screens with a 10-item shortlist; this general adapter is not yet measured.
• Your users mostly speak languages other than English. All training data is English.

How to use it

The adapter runs with Jeff's server on the Jeff-Qwen3.5-0.8B v1.2 base. One server holds the base and any number of adapters; each request picks one by name with "model": "nav".

git clone https://github.com/firelex/jeff && cd jeff
uv sync --no-default-groups --extra lora          # add --extra cuda on NVIDIA GPUs, --extra mac on Apple silicon
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B --revision v1.2 --local-dir Jeff-Qwen3.5-0.8B-v1.2
uv run --no-default-groups hf download mstrasser/Jeff-Qwen3.5-0.8B-nav --local-dir adapters/nav
JEFF_CHECKPOINT=Jeff-Qwen3.5-0.8B-v1.2 JEFF_ADAPTERS=adapters PORT=8765 \
  uv run --no-default-groups --extra lora jeff-serve
# add JEFF_BACKEND=mlx and --extra mac for Apple silicon

Every folder in adapters/ is served under its folder name. Add or replace an adapter while the server runs with POST /v1/adapters/reload. The server checks the base weights' checksum, so the adapter loads only on the exact v1.2 base.

from jeff import Client

nav = Client("http://localhost:8765", model="jeff-latest").with_model("nav")

Request format

State (the situation):

Questions:

• target (choice): Which item on the screen the user wants, or whether they are asking a question, or want something not listed. Options: ask_question and none_of_these first, in that order, then every item on the screen as o1, o2, … with a short text in the app's own style, such as "Settings", "client Brennan Roofing" or "invoice 2231 → payments".

Rules:

• Use the instructions below word for word; the adapter was trained mostly on them.
• Keep the two fixed options first, with these exact texts. ask_question is "Not a navigation request: the user is asking a question"; none_of_these is "None of these: the user wants something not listed".
• List every item on the screen; do not shortlist. Use at most 254 options in all. Item order does not matter; it was shuffled in training.
• Keep the state keys in this order, with the transcript last, so the unchanging part of the request can be prepared in advance.

Example

{
  "model": "nav",
  "state": {
    "current_screen": "Tallow Books › Clients",
    "previous_screen": "Tallow Books › Home",
    "dialogue_step": null,
    "voice_transcript_may_contain_errors": "um open bren and roofing supplies"
  },
  "questions": {
    "target": {
      "type": "choice",
      "instructions": "Which of these does the user want? The voice transcript comes from speech recognition and may contain misheard words, so match by meaning and by sound. If the user is asking a question rather than asking to go somewhere or do something, choose the question option. If none of the listed options is what they want, choose none of these.",
      "criteria": {
        "ask_question": "Not a navigation request: the user is asking a question",
        "none_of_these": "None of these: the user wants something not listed",
        "o1": "Home",
        "o2": "Search",
        "o3": "client Brennan Roofing",
        "o4": "client Brennon Roofing Supplies",
        "o5": "invoice 2231 → payments",
        "o6": "Settings"
      }
    }
  }
}

curl -s localhost:8765/v1/systemone -H 'content-type: application/json' -d @request.json

The answer holds a probability for each option of each question, the chosen option and a confidence. (A recorded response to this example is not published yet.)

Training

The replay slice mixes in some of the base model's own training data so the adapter keeps the base's general skills. The scored checkpoint is the one at the end of the epoch.

Data and provenance

Sources:

Which models wrote and checked the data. The spoken commands were written by Qwen3.8-Max (hosted); the apps by Qwen3.8-Flash-Next (local), and the apps' navigation systems by both. Rows were checked by Qwen3.8-Flash and DeepSeek-V4-Flash (hosted). Qwen3.8-Max is a hosted, closed-weight model, used through Alibaba Cloud's DashScope API to finish in time. Counts are on the 72,299 training rows; one row can pass through several models, so the counts do not add up to the total.

nav's rows record only the checking model; the hand-over file (READY-nav) records which models wrote each stage, listed below as the number of apps (of about 340) each model worked on:

Training data not published. The test set is included in this repo as test.jsonl, and the calibration set (1,994 rows, the rows the threshold for the comparison with Qwen3.8-27B is chosen on) as calibration.jsonl.

Test set: test.jsonl, 3,300 rows: Requests from about 10% of the roughly 340 generated apps, never trained on. It is byte for byte the set the adapter was scored on.

The terms of the hosted model providers are being checked for training and publication use.

Quality review. An independent reviewer checked the training, calibration and test files for shortcuts, duplicates, leaks between splits and junk before training. The report is at jeffhub.ai/adapters/nav/qa-report.

Limitations

• English only.
• Tied to the Jeff-Qwen3.5-0.8B v1.2 base. It will not work on any other base or version; the server refuses it if the base weights' checksum does not match. A v1.3 long-term-support base is coming, and the adapters will be retrained on it.
• Jeff chooses between the options you give it. It does not write text or reason in several steps.
• Calibration was fitted on this adapter's own calibration rows. On very different data, check it again.
• The general adapter has not been measured on any single real app yet.
• Everything listed under When not to use it above.

Links

• Adapter page: jeffhub.ai/adapters/nav
• Code and server: github.com/firelex/jeff
• Base model: mstrasser/Jeff-Qwen3.5-0.8B (revision v1.2)

Jeff is an independent project. It uses the same request format as Jev but is not affiliated with or endorsed by TypeSafe, the makers of Jev.