CPU Character Router: a failed abstention threshold

A recipe request with nowhere to go

“Suggest a soup recipe.” is outside a router whose three possible destinations are weather, calendar and notes. In this experiment, the saved model still accepts that request as notes, with a maximum softmax probability of about 0.486. A route score measures preference among the available labels; it does not certify that a request belongs to any of them.

I built this small model to inspect that boundary. It counts short character sequences, learns three sets of weights from 96 synthetic training requests, and uses a validation-selected threshold to decide when to abstain. The threshold 0.4 reaches 44 correct decisions among 47 accepted validation requests. On the separate 54-record test split, it accepts 52 requests, of which 43 are correct. All six test requests outside the three routes are accepted. I kept those failures and the original threshold instead of tuning against the test answers.

This package includes the weights, all 204 synthetic records, the 300-epoch training trace, the threshold table and every validation/test output. A reader can replay a request locally or inspect which decisions were counted. It is a NumPy CPU model release with a command-line interface; there is no hosted App or verified Transformers/Inference Provider integration. It never executes a weather, calendar or notes action.

The templates were written and visible to me, and each language has only three out-of-domain test examples. The result illustrates one failed rejection rule on this constructed sample. It does not estimate open-world safety, calibrated confidence or general bilingual understanding.

一条菜谱请求,被送进了笔记

“Suggest a soup recipe.”是一条菜谱请求,超出了天气、日历、笔记这三个去向。这个实验保存的模型却把它接受为笔记,最高softmax分数约0.486。分数只比较已有三个标签的相对偏好,不能证明请求属于其中任何一类。

我用这个小模型检查拒答边界:先把短字符片段变成特征,用96条合成训练请求学习三组权重,再从验证集选择拒答阈值。0.4阈值在验证集接受47条、答对44条;换到独立划分的54条测试样本,接受52条,其中43条正确。六条不属于三个去向的请求全部误放行。我保留了失败样例和原阈值,没有照着测试答案调参。

包里保留全部204条合成记录、权重、实际300轮训练曲线、阈值表和验证/测试的每条输出,可在CPU上用命令行重放。此次没有托管App,也未验收Transformers或推理供应商接入;模型不会真的执行天气、日历或笔记操作。

模板由我编写并可见,测试中每种语言只有三条域外样例。这说明的是这组构造样本上一条拒答规则的失败,不支持开放世界安全、概率校准或通用双语能力的结论。

A small character-ngram softmax classifier trained from scratch on author-constructed English and Chinese routing prompts. It predicts weather, calendar or notes and can abstain below a validation-selected threshold. The selected threshold failed on the held-out out-of-domain examples: all six were accepted.

This is an educational negative-result study with a runnable CPU model, fixed data, training code and every evaluated output. It is not a language model, a production router, a security filter or evidence of general bilingual understanding. No external/pretrained model or API was used, and no tool action is executed.

The observed result

Quantity Validation Held-out test
Known-route top-1 accuracy 48/48 44/48
Accepted correct / all accepted 44/47 43/52
Correct accepted / known cases 44/48 43/48
Accepted OOD / all OOD 3/6 6/6
All accepted / all cases 47/54 52/54

The threshold 0.4 met the empirical validation precision target of at least 0.9. Test accepted precision was 43/52, about 0.827, and the threshold rejected none of the six new OOD prompts. Validation precision was not a guarantee for a different set of requests. The test was retained as observed; no threshold or feature settings were retuned after reading it.

The known-class majority baseline, selected using training counts and fixed class-order ties, predicts weather and is correct on 16/48 known test cases. This baseline does not address OOD rejection. The reference's English known-route test accuracy was 20/24, Chinese 24/24; each language has only three OOD test examples, all accepted. These tiny synthetic counts do not support population or cross-language superiority claims.

For example, the held-out OOD text Suggest a soup recipe. was accepted as notes, with maximum softmax probability about 0.486. The model's probability is not calibrated confidence.

Reproduce and use

Tested with CPython 3.12.14 and NumPy 2.5.4:

python -m pip install -r requirements.txt
python -m unittest test_router -v
python verify_results.py .
python router.py model.json "Suggest a soup recipe."
python train.py --output fresh-run
python verify_results.py fresh-run

The output directory must be new. train.py runs the single frozen configuration, chooses a threshold on validation, serializes the model and reloads it before saving validation/test predictions and reports. Repeating the command is a replay of the same configuration, not a search over test scores. Final floating-point digits and hashes can vary by NumPy/BLAS/platform; preserve the recorded reference and compare counts and numerical tolerances. There is no cloud-training claim.

router.py accepts a local JSON model and a text argument, limited to 4,096 characters. It emits the top route, all three probabilities, threshold, acceptance and reason. An abstention has route: null. JSON weights contain no executable pickle or remote code. Only NumPy is required for inference; it makes no network calls and does not execute the selected route.

Frozen protocol and data

The 204 records are entirely author-constructed. There are eight bilingual template families per known class, with four slot variants per language. Families 0–3 train, 4–5 validate, and 6–7 test. English/Chinese siblings and slot variants stay in one split. Six separate bilingual OOD families are divided between validation and test; none are used for training.

Split Known OOD Total
Train 96 0 96
Validation 48 6 54
Test 48 6 54

data/FROZEN.json and PREFIT-DATA.json record the data hashes fixed before fitting. Exact text and group overlap checks passed. This limits direct fixture reuse; it does not remove shared vocabulary or prove the template families are semantically independent. The author wrote and could inspect all synthetic templates, so this is not a human-blind benchmark.

Features are NFC then casefold, character 1–3-gram counts with boundary markers, stable BLAKE2b hashing into 4,096 bins and L2 normalization. Hash collisions are possible. There is no vocabulary or IDF fitting on test data. The linear three-class model uses zero initialization, 300 full-batch gradient-descent epochs, learning rate 0.8 and weight L2 coefficient 0.01. This is one frozen configuration.

Threshold candidates are 0.0, 0.1, …, 1.0 plus 1.000001 for always abstaining. Among candidates with at least one accepted validation row and accepted precision at least 0.9, select the most correct accepted known rows; ties prefer higher precision, then higher threshold. Accepted OOD is always incorrect for this criterion. With no feasible threshold, the policy always abstains. Threshold selection is empirical validation selection, not probability calibration.

Evidence and limitations

model.json, RUN.json, training loss, threshold table, full validation/test row predictions and per-language reports are retained. verify_results.py independently recounts metrics from saved rows and original labels, checks the chosen threshold and replays the loaded model. Tests also cover canonical feature behavior, finite/normalized probabilities, invalid model structure, empty denominators and always-abstain selection.

The model sees weather/calendar/notes only and has no semantic model of requests. Six OOD examples per evaluation split cannot characterize open-world rejection. Slot vocabulary and task words are simple; templates are artificial and may be unusually easy. No adversarial prompt, user deployment, latency benchmark, general language assessment or real adoption was tested. A positive known-route score cannot erase the observed OOD failure.

License

Apache-2.0, copyright 2026 Renchao Wu. Data, code and weights are original synthetic work. NumPy is installed separately; its own distribution license terms apply and its source is not vendored. MANIFEST.json hashes all release files except itself. This is a separate experiment and does not revise the older LoRA or JSON-repair evaluation results.

中文说明

这是从零进行CPU训练的字符n-gram线性三分类器,只识别天气、日历和笔记请求,并按验证集阈值决定是否拒答。验证集选出的0.4阈值未能拒绝测试集的六条域外请求,六条全部误放行;完整负结果已保留,没有看到测试结果后调参。已知类别测试44/48正确,也不能抵消域外失败。全部204条数据为手写模板生成的合成样例,不代表真实用户分布、通用双语能力或生产安全性。

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support