Simon Says for a group of friends outdoors, refereed live by open models: a Qwen3.5-4B referee fine-tuned on Tinker, or Liquid's d1 entirely on your own laptop.
Prop a phone on a stump. Step back. A voice calls "Simon says stand on one foot", "touch something green", "freeze", and sometimes just "Hands on your head!" with no Simon at all. A few seconds later: "Number 2, Simon didn't say! You're out." Nobody looks at a screen until there's a winner.
Replay demo, not the field test: the phone page at 390×844 playing a game over the AI-generated
sample photos in samples/people, with the real models on a 2-vCPU CPU: facebook/sam3 (560 px) finds the
players, facebook/dinov2-small (the FALLBACK while the DINOv3 gate is pending) tells them apart, and
the Tinker-tuned referee (Qwen/Qwen3.5-4B + LoRA, sampled on Tinker) judges each player's crop. The video is sped up; the decision table shown in the Ref's notes drawer holds the model's actual
probabilities. With sound: demo/simon-sees-demo.mp4 (Simon's voice is Kokoro TTS).
phone (browser) laptop (Python), or the GPU endpoint for the vision half
┌────────────────────┐ JPEG 640px ┌───────────────────────────────────────────────────────┐
│ camera, 3 fps │ ────────────────▶ │ FastAPI + WebSocket · game engine · roster · voice │
│ speaker │ ◀──────────────── │ SAM 3 ──▶ DINOv3 ──▶ match to line-up ──▶ d1 per crop │
│ (screen: setup only)│ Simon's voice │ open TTS (Kokoro / Piper) · open STT · decisions.jsonl │
└────────────────────┘ └───────────────────────────────────────────────────────┘
Every frame batch goes through four steps (simon_sees/pipeline.py):
- Enrolment (the line-up). Everyone stands in a row and you tap Line up. SAM 3 segments every
person in the frame with the text prompt
"person"; each one becomes a photo card, numbered left to right. Type a name on a card or say it (faster-whisper); untouched cards stay Player N and Simon calls them "Number N". Each player's masked crop is embedded with DINOv3: that vector is their identity. Nobody describes what they look like or what they wear; there is no text description of a person anywhere. - Segmentation. Each round, SAM 3 finds every person in the new frame (masks + boxes). Only tiny figures (under 12% of the frame height, i.e. far behind the players) are dropped, so a player can step well back.
- Embedding match. Each person's masked crop is embedded with DINOv3 and compared (cosine) with every player's small gallery: the line-up embedding plus a few confident sightings from the game, so the signature follows lighting and pose. A Hungarian assignment pairs players with people one-to-one, with a small bonus for staying in the same spot. Below the threshold (0.45) a player is not seen, unless the detection is where they just were, or the head count settles it: a pose change and a move between two looks can drop the cosine to ~0.25-0.4 (measured with DINOv2-small: one person standing vs. squatting 0.23, while two different people standing scored up to 0.57), so when the frame holds exactly as many people as there are players, the assignment is accepted down to 0.25 if each pairing is mutual-best (players are never swapped on the count alone). Solo, the one person in the picture is the player, wherever they stand; with a passer-by in shot too, the player's best match must clearly beat the runner-up. Every crop the referee sees, and the freeze comparison, comes from where the player is in that frame.
- Judging per crop. Each matched player's own crop (background dimmed by the SAM mask so the
neighbour doesn't count, grass and trees still visible) goes to the referee, the fine-tuned Qwen3.5-4B on
Tinker (see Best Use of Tinker below) or Liquid d1 locally, with one yes/no question:
"Look at the person in this photo. Is that person standing on one foot?" Freeze rounds give d1 the same
player's BEFORE and AFTER crops in one request: "These two photos show the same person a moment apart. Did that
person move?" (A pixel-difference check inside the player's box runs alongside and is logged as a second
opinion; it only decides with a model that can't take two images, and it is labelled
FALLBACKin the log.)
Rules: Simon said it and P(did it) < 0.35 → out. Simon didn't say it and P(did it) > 0.65 → out. In between,
the referee isn't sure and gives the benefit of the doubt. A player who isn't matched in a frame is warned, never
eliminated. If everyone is caught at once, the round doesn't count. Every decision is logged to
logs/decisions.jsonl (models, per-player similarity, per-frame probabilities, latency per stage, verdict).
Credit: the segment-then-embed idea is the visual-search pattern Maziyar Panahi showed with DINOv3 + SAM 3.1 (post): SAM 3.1 isolates each object, DINOv3 ranks the others by visual similarity to a query. Here the query is each player's line-up photo.
| step | preferred | fallback (labelled FALLBACK in logs, /api/info and the demo caption) |
|---|---|---|
| segment | facebook/sam3 via transformers (CPU: 560 px input; GPU: 1008 px); facebook/sam3.1 via Meta's repo on CUDA |
PekingU/rtdetr_r18vd person boxes (ungated, no masks) |
| embed | DINOv3 facebook/dinov3-vitb16-pretrain-lvd1689m (gated) |
facebook/dinov2-small (ungated) |
judge (--decider auto) |
Qwen3.5-4B + LoRA fine-tuned on Tinker, sampled on Tinker (when SIMON_TINKER_MODEL + TINKER_API_KEY are set) → the same weights locally (SIMON_REFEREE_WEIGHTS) |
LiquidAI/d1-3B / LiquidAI/d1-omni-600M on your own machine |
--segmenter auto --embedder auto --decider auto (the default) tries the preferred model and falls back
automatically. Without a Tinker key or local referee weights, the judge is d1, exactly as before; the fallback is
logged. With the Tinker/Qwen referee, freeze rounds are judged by the pixel-difference check (that referee sees one
crop at a time); with d1, by d1's two-image question.
On the 2-vCPU, 8 GB, no-GPU sandbox this was built in, with the 7 sample photos (bench/results-sam3-cpu.md):
SAM 3 at 560 px found the right number of people in 7 of 7 frames, at a median 16.6 s per frame. With
DINOv2-small (DINOv3 was still gated for that account), re-identification was fully correct in 5 of 6 frames after
the line-up, including the frame where all three players swapped places (cosine 0.94–0.96 to the right
person), with no identity swaps; the one miss was a squatting player reported as not seen (cosine 0.30), which the
rules turn into a warning, not an elimination. These are 7 generated photos, not a field test or a benchmark.
In the recorded replay demo on the same box (re-recorded Oct 9 2026 with the Tinker-tuned referee), the one pose
round took SAM 3 56.5 s for the frame, DINOv2 1.0 s, and the fine-tuned Qwen3.5-4B referee on Tinker 9.1 s for one
batch of the three players' crops (sent as concurrent requests); the demo is sped up x3.9 to fit in a minute. In that
round ("Simon says put your hands on your head") the referee gave P = 0.00, 0.995 and 0.00, which matches the
frame: only the middle player had her hands on her head, so she wins after one round. (The earlier recording with
LiquidAI/d1-omni-600M gave 0.12, 0.71 and 0.14 on the same frame.) With SAM 3 on a CPU this is far too slow for live play: use a GPU laptop, a Mac
(mps), or the Modal backend below.
Liquid AI's d1 (released Oct 7 2026) is a decision model: you give it a
state (here: one player's crop) and typed questions, and it returns calibrated probabilities in one forward
pass with zero generated tokens. A referee needs exactly that: a fast yes/no you can threshold, not a
paragraph. bench/benchmark.py measures it against a generative open VLM on the same crops.
model.system_one(None, {
"did": {"type": "noul", "instructions": "Look at the person in this photo. Is that person standing on one foot?",
"criteria": {"true": "one foot clearly lifted off the ground", "false": "both feet on the ground"}},
}, images=[player_crop])
# {"answers": {"did": {"type": "noul", "noul": 0.97}}, "usage": {"input_tokens": ..., "output_tokens": 0}}d1 answers fast, but on this box the d1 that fits (d1-omni-600M) was wrong too often to referee a game: 63% on
our human-checked test set, with 65 false "you did it" calls on 141 people who hadn't. So the referee was
fine-tuned on Tinker: a rank-32 LoRA on Qwen3.5-4B (open weights,
Apache-2.0), asked exactly the game's question for each crop, with P(yes) read from one token's logprobs.
Test set: 222 COCO player crops, every label checked by eye, from 210 photos never used for training
(bench/results-tinker.md):
| referee | accuracy | macro-F1 | false "did it" | missed |
|---|---|---|---|---|
| d1-omni-600M (CPU) | 63.1% | 0.629 | 65 | 17 |
| Qwen3.5-4B, untuned | 87.4% | 0.871 | 27 | 1 |
| Qwen3.6-35B-A3B, untuned | 88.7% | 0.882 | 19 | 6 |
| Qwen3.5-4B + LoRA, first run (v1) | 87.8% | 0.874 | 22 | 5 |
| Qwen3.5-4B + LoRA, v2 (the referee) | 93.2% | 0.928 | 9 | 6 |
- What made the difference: hard negatives. The untuned model's mistakes were false positives on near-miss
poses: a crouched skateboarder or a baseball slide "touching toes", cross-legged sitting "squatting", a cyclist
"on one foot". The first fine-tune (v1) learned the easy cases, hit 100% on an easy val set, and didn't move on
test. v2 added 720 near-miss negatives mined from COCO keypoints (
scripts/mine_hard_negatives.py), halved the learning rate and dropped to 2 epochs: false positives 27 → 9,touch_toes75% → 98%. Against the untuned 4B, exact McNemar p = 0.019; against the untuned 35B, v2 is ahead (18 vs 8 disagreements) but p = 0.08. One caveat: the near-miss categories were chosen after looking at v1's test errors (no test photo was used), so read the gain as somewhat optimistic. - Real game crops (7 AI-generated sample photos, 119 crops, 9 yes): v2 98.3%, untuned 4B 99.2%, d1-omni-600M 77.3%. Too small and easy to separate the Qwen models; it does show d1's false positives on the game's own crops.
- Speed: one batched Tinker call per round, ~2.9 s from this sandbox whether 1, 3 or 6 players (480 ms per crop at 6). d1-omni-600M on the 2-vCPU CPU: 2.7 s per crop, 7.9 s for a round of 3.
- Cost: ~$0.00006 per crop, about $0.004 for a 10-round game with 6 players at Tinker list prices. Both training runs plus all evaluation and the 35B label-cleaning teacher came to about $2 (estimated from logged tokens).
- Data: COCO 2017 keypoints (CC BY 4.0) → explicit geometric pose rules → crops cut exactly as the game cuts a
player → train/val labels cleaned by agreement with the untuned 35B on Tinker; test labelled by hand. Details,
licences and what failed (
hand_mouth/wavedropped, thin rare-pose positives, an easy v1 val set):data/README.md,bench/results-tinker.md.
export TINKER_API_KEY=... SIMON_TINKER_MODEL=tinker://7bfed040-1481-5605-ad1f-a7cdee9763ca:train:0/sampler_weights/step0100
python -m simon_sees serve # --decider auto picks the Tinker referee; unset the vars for local d1A Tinker checkpoint is private to the account that trained it. To run the referee without Tinker,
scripts/export_referee.py downloads the LoRA (done here, 278 MiB) and converts/merges it for transformers + peft
or vLLM (--decider local-qwen, docs/tinker.md). That local path needs ~9 GB for the model and was not tested
here (8 GB RAM, no GPU).
\n## Quickstart\n
git clone https://github.com/nice-bills/simon-sees && cd simon-sees
python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
python -m simon_sees dryrun --replay samples/people --segmenter mock --embedder mock --decider mock --fast
python -m pytest -qThe phone client is tested end to end with e2e by TesterArmy (Playwright
under the hood), on a desktop window and a 390×844 phone viewport. The runner starts the real server in dry-run
mode (replayed frames, mocked segmenter/embedder, seeded mock decider), taps Line up, checks the name chips are
numbered left to right, starts a game, and asserts the rounds, an elimination and the winner with deterministic locators and assertions. An optional agent.act /
agent.assert test runs only when AI_GATEWAY_API_KEY is set (Vercel AI Gateway); otherwise it is skipped.
pip install -r requirements.txt
cd e2e && npm ci && npx e2e-web install chromium
SIMON_PYTHON=python3 npx e2e run # both targets; npx e2e run --target phone for the phone onlyCI (.github/workflows/ci.yml) runs pytest, a terminal dry run, and the e2e suite on every push and PR.
The demo above is recorded the same way: cd e2e && npm run demo && python ../scripts/make_demo.py.
With --segmenter mock --embedder mock --decider mock the "people" are vertical bands and the answers are
random: this shows the game loop, voice lines and logging. It says nothing about accuracy.
pip install -r requirements-model.txt # torch + transformers 5.17 (d1 needs >= 5.14, SAM 3 is in 5.x)
pip install -r requirements-extras.txt # optional: Kokoro TTS, faster-whisper STT, Sentry
export HF_TOKEN=hf_... # accept the gates first: facebook/sam3 (and DINOv3, optional)
python -m simon_sees serve # downloads SAM 3, DINO and LiquidAI/d1-3B (~6 GB) on first run
cloudflared tunnel --url http://localhost:8000 # in a 2nd terminal: gives you an https URLOpen the https://….trycloudflare.com URL on the phone, turn the camera on, have everyone line up facing it,
tap Line up, name the photo cards if you like, tap Start.
Fully offline (phone hotspot + local HTTPS cert) and ngrok options: docs/phone-setup.md.
Desk demo with the real model and replayed frames (or your own video):
python -m simon_sees serve --replay samples/people # open http://localhost:8000, the page plays from the replay
python -m simon_sees dryrun --replay my_clip.mp4 --names Kofi Ama "" # line-up from the first frame; video needs opencv-python-headlessUseful flags: --model LiquidAI/d1-omni-600M (smaller, experimental), --device cuda|mps|cpu,
--dtype bf16, --compile (CUDA graphs on NVIDIA), --indoors (no grass/tree commands),
--save-frames captures (keep judged frames locally to label for the benchmark), --tts browser,
--segmenter sam3|sam3.1|fallback, --embedder dinov3|dinov2, --match-threshold 0.45,
--follow auto|on|off (rings follow the players between judged looks; auto = on locally, off with Modal).
Every game is recorded on the laptop in sessions/ (frames, per-player crops, per-stage timings, identity
scores, d1's answers, verdicts, errors; SIMON_RECORD=0 turns it off). Players tap Wrong call on a verdict
or in Ref's notes, and leave a note at full time. python -m simon_sees upload sends the sessions to a private
Hugging Face dataset (bundle makes a zip instead). Step-by-step for a non-expert: docs/playtest.md,
or just ./run_playtest.sh. On the analysis side: scripts/pull_playtests.py, scripts/playtest_report.py
(accuracy vs corrections, per-command errors, p50/p95 latency per stage, identity failures, crashes, notes) and
scripts/export_labels.py (crop + command + label JSONL, source: model | human_correction, for fine-tuning the referee).
deploy/modal_app.py puts SAM 3.1 + DINOv3 + d1-3B behind one HTTPS endpoint on a Modal GPU; the game, roster,
voice and phone page stay on your laptop. Deploy it from your laptop (modal deploy deploy/modal_app.py), then
export SIMON_VISION_URL=… and python -m simon_sees serve uses it. Setup, auth and cost notes:
docs/modal.md. It has not been deployed or timed yet; no GPU numbers are claimed here.
The camera is the interface. The picture (in a replay, the exact frame the referee judged) fills the phone and
a thin HUD floats on it, in the style of the viral juggling-counter app: a huge condensed numeral for the
players still in, letter-spaced caps labels, one lime accent, scrims instead of panels. Every player gets a
glowing ring drawn from their SAM box with a name chip (2 · AMA); the ring's colour is the verdict (lime safe,
red out with the ring breaking, amber "Simon didn't say", "lost you" when the latest frame doesn't show them).
The rings follow the players: between judged looks the server re-finds everyone about once a second (matching
only, no referee call; skipped right before a judged look, and switched off automatically when one look takes
longer than 2.5 s, e.g. SAM 3 on a CPU, where rings move once per round instead). With the live camera the
whole field of view is shown, so there is no fixed spot to stand in. At the bottom: the call, then the
referee's confidence as a per-round sparkline, and ROUND · CALL TIME · TIME. Models, latencies and the
"Wrong call" controls live in a slide-up "Ref's notes" sheet. Fonts (Barlow Condensed + Inter, SIL OFL) are
self-hosted, so it works offline.
Direction and system: docs/design/DIRECTION.md; every state at 390×844 in docs/design/hud/.
| where | what to expect |
|---|---|
| NVIDIA GPU (RTX 4090) | Liquid reports 17 ms for a 384 px image with d1-3B (model card) |
Apple Silicon (M5 Pro, mps) |
Liquid reports 62 ms for a 384 px image |
| Jetson Orin Nano | Liquid reports 202 ms for a 384 px image |
| CPU only (2 vCPU, measured) | SAM 3 at 560 px: median 16.6 s per frame; DINOv2-small: ~0.3 s for 3 crops (bench/results-sam3-cpu.md); d1-omni-600M: bench/results-sandbox-cpu.md |
d1-3B needs ~6.3 GB in bf16 (12.5 GB in fp32, the card's CPU default). On a 16 GB Mac use mps (bf16).
A pose round is one SAM + DINO pass per frame plus one d1 call per seen player per frame, all batched; Simon gives
players a ~2 s beat anyway, so anything under ~500 ms per batch feels instant. SAM 3 and the d1 numbers above are
separate models: the GPU rows are Liquid's d1 figures only, not the whole pipeline.
- Privacy: it's a camera pointed at your friends. With the laptop setup, no video frame leaves the
devices you own: no cloud vision API, no account, no upload. Frames live in memory and are dropped;
saving them is opt-in. Identities are DINO vectors held in memory for one game, never names-to-faces in a
database. (The optional Modal backend sends frames to your own Modal workspace; see docs/modal.md. The Tinker
referee sends each player's dimmed crop, not the frame, to Tinker for one yes/no answer; for a fully local game
leave
SIMON_TINKER_MODELunset or use the local referee weights.) - Works where games happen: parks and beaches have bad signal. Weights download once; the phone can join the laptop's hotspot with mobile data off.
- Free to run: an afternoon of Simon Says is thousands of vision decisions. Locally that costs a laptop battery, not an API bill.
- Hackable: commands are plain-English questions in
simon_sees/commands.py. Add "touch your nose" in one line; swap models with--model.
simon_sees/
engine.py game loop + rules (pure judge() function), line-up → players
vision.py SAM 3 / SAM 3.1 / FALLBACK detector segmenters, DINOv3 / DINOv2 embedders, mocks, person crops
identity.py enrolment numbering, per-player galleries, Hungarian matching (who is who)
pipeline.py one frame batch: segment → embed → match → d1 per crop; local or remote (Modal) backend
decider.py d1 wrapper (system_one_batch), question builders, referee picking (auto), mock for tests
tinker_referee.py the fine-tuned Qwen3.5-4B referee: prompt, Tinker sampling, local transformers+peft
pose_rules.py COCO keypoints -> pose labels for the referee's training data
commands.py the command list: spoken text + the visual yes/no check
motion.py FALLBACK pixel-difference motion check
server.py FastAPI: phone page, WebSocket frames in / voice out, /api/stt
voice.py Kokoro/Piper TTS, faster-whisper STT, spoken-name parsing
replay.py dry-run camera: a folder of images or a video
tracing.py optional Sentry gen_ai spans
recorder.py playtest session recorder (sessions/<time>-<id>/)
upload.py `upload` (private HF dataset) and `bundle` (zip) for recorded sessions
playtest_data.py reading sessions back (report + label export)
static/ the phone page (vanilla JS, no build step)
bench/benchmark.py d1 vs generative VLM: accuracy + latency → CSV + markdown table
bench/sam3_cpu_reid.py SAM 3 person counts + latency and DINO re-identification on samples/people
bench/referee_*.py referee evaluation, report and latency (results-tinker.md)
data/ referee test set (human-verified) + real-game slice; README: data pipeline + licences
deploy/modal_app.py the GPU endpoint (SAM 3.1 + DINOv3 + d1-3B) for --backend modal
scripts/ sample-frame generator, log summarizer, demo video builder, playtest pull/report/export
tests/ pytest: engine rules, enrolment + re-identification, full mocked games, server WebSocket,
the remote-vision client against a mocked HTTP endpoint
e2e/ e2e (TesterArmy) tests of the phone page: desktop + phone viewport, plus the demo recorder
demo/ the replay demo (MP4 with voice, GIF)
docs/ phone setup, Modal + DigitalOcean GPU deploys, the DEV post draft, design/ (UI direction, renders)
No GPU laptop? docs/deploy-digitalocean.md runs the same Docker image on a
DigitalOcean GPU Droplet. Set SENTRY_DSN to get one gen_ai.invoke_agent transaction per round and one
gen_ai.evaluate span per d1 call (latency, tokens; never images). Both are off by default and the local
run never needs them.
- Code: MIT (see
LICENSE). - Model weights are not in this repo.
LiquidAI/d1-3BandLiquidAI/d1-omni-600Mare published by Liquid AI under the LFM Open License v1.0 (license: other,license_name: lfm1.0on the model cards). In short: free to use, modify and redistribute, including commercially, as long as the commercial user's annual revenue is under US$10 million; above that threshold commercial use is not licensed (qualifying non-profits doing non-commercial/research work are exempt from the threshold). Read the full text in the model repo'sLICENSEbefore you build on it. - SAM 3 / SAM 3.1 (
facebook/sam3,facebook/sam3.1) are gated and released by Meta under the SAM License; DINOv3 is gated under the DINOv3 License; DINOv2 and RT-DETR (PekingU/rtdetr_r18vd) are Apache-2.0. Accept each gate on Hugging Face and read its terms before you use it. - The fine-tuned referee is a LoRA on
Qwen/Qwen3.5-4B(Apache-2.0), trained on COCO annotations (CC BY 4.0) and Flickr images under their own CC licences, many of them non-commercial; the weights are not in this repo. - Fonts (self-hosted in
simon_sees/static/fonts/): Barlow Condensed and Inter, both SIL Open Font License 1.1 (OFL-Barlow.txt,OFL-Inter.txt). - Optional components carry their own licences: Kokoro-82M (Apache-2.0), Piper voices (per voice), Whisper / faster-whisper (MIT).

