This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What We Built

THE UNHEARD is an acoustic-memory game where the machine knows the name of every sound and has never heard one.

You meet ECHO-0, a mind that has read about rain, bells and footsteps ten thousand times and holds no memory of any of them. The game asks you to do the one thing a screen cannot: go outside, record something that is really there, and bring it back. You tell ECHO-0 what you heard. It guesses, gets things wrong, takes your correction, and remembers. By the end, the sounds you collected are an orchestra.

The design principle is "The Model Never Changes. The Memory Does." Nothing in this project trains or fine-tunes anything. A frozen audio model hears; a retrieval memory keeps what you taught it; a local Gemma speaks. Three separate mechanisms, never blurred.

It is a local-first app (a React PWA, a FastAPI server, SQLite, CPU audio embeddings, and Gemma through Ollama) with six scenes:

The ending is not scripted. It unlocks only when a real composition exists, built from at least three recordings you named.

The Hearing Room. CLAP's own ears put "Church bells" at cosine 0.367. The name the player taught, "Bells", has a memory similarity of 0.844 from two remembered examples, so memory ranks it first. These are similarity scores, not probabilities.

How it gets people off the screen

The prompt says: make the screen the shortest part. We took it literally, and then we felt how hard it is.

A game about sound should not be played on a screen. The screen is only where the machine keeps its notes. The part that matters happens when you stand in a street at an hour you have never listened to, hold up a phone, and realise you could not have told anyone what that street sounds like. Teaching ECHO-0 to name a sound turned out to be the fastest way to notice how many we had never heard ourselves.

That is why the interface is calm and sparse, why the hearing room shows honest similarity numbers instead of confident percentages, and why the machine asks you instead of pretending. It is a game that wants you to close the laptop.

Before you play

Be ready for a laptop-first experience: capture works offline on the phone, but hearing, memory, composition and Gemma all run on your laptop. This is not an on-device AI app, and we would rather say so than let you find out. For the phone to use its microphone it needs trusted HTTPS, so the repo includes a one-command local certificate setup and a server flag that serves the app itself over TLS. We have written the Android field test up, but we have not yet performed it on a physical phone.

Taking it outside — current status

This is where we owe you the most honesty. The game is built around leaving the screen, and we did not complete that outing for this submission. The Android field test is written up (docs/ANDROID-FIELD-TEST.md) and its results table is empty. Our automated browser runs used desktop Chrome with a simulated microphone and prove nothing about phones. We would rather hand you a clear gap than a made-up trail story.

Demo

Fifty seconds, narrated and scored. Everything on screen is the real app running on the generated set: real CLAP inference, real taught names, a real Composed render.

About the demo audio

Here is the part we owe you plainly. The demo sounds were generated with the ElevenLabs sound-effects API. They are not field recordings. The narration in the video is an ElevenLabs voice, and the music is the app's own Composed render of those generated sounds.

Why: the only audio we had for testing was a research dataset that is licensed for non-commercial use and cannot appear in a public demo. We first synthesised a publishable set from scratch in NumPy (it is in the repo, deterministic, free to reproduce), then regenerated it with ElevenLabs so the demo would sound like something. The project still does exactly what it claims with real recordings, but we are not going to dress generated rain up as a hike.

Code

MIT licensed. Model weights are downloaded locally and are never in the repository.

git clone https://github.com/miclaldogan/the-unheard
cd the-unheard/server
uv sync --extra ml
uv run unheard fetch-model          # one-time CLAP download
cd ../web && npm ci && npm run build
cd ../server
UNHEARD_NARRATIVE_MODEL=gemma3:4b uv run unheard serve   # http://127.0.0.1:8000

How We Built It

ECHO-0 is three things that never pretend to be each other.

• The part that hears: LAION-CLAP. laion/clap-htsat-unfused turns audio into a 512-dimensional embedding and scores it against text labels, on CPU. It is frozen. Nothing here trains it.
• The part that learns: retrieval memory, not training. When you name a sound, its embedding is stored as an exemplar of your label. Later predictions fuse CLAP's scores with nearest-neighbour similarity to your exemplars, and the interface shows which one is speaking. On the demo set, we taught two "Roof gutter" clips. On a third clip it had never heard, that name was absent from the top three before teaching (Rain, Bicycle, Lawn mower) and first afterwards. The model did not change. The memory did.

• A bug worth admitting. If you named a sound before its analysis finished, the name was saved but never became an exemplar: it looked taught and was not. The fix attaches the stored name to the embedding when inference completes, exactly once, and the interface says "memory has not changed yet" until that is true. A real-CLAP test reaches the same similarity (0.837) whether you teach before or after.

• The part that composes: no music model. An offline NumPy/SciPy renderer cuts motifs out of your recordings, works out which are pitched or percussive, builds phrases and sections, and lets one source answer another. Every event in the output traces back to a window of one of your recordings; the piece in the video has 47 of them.

The part that speaks: Gemma, on a leash

ECHO-0's voice is a local gemma3:4b through Ollama (4.3B parameters, Q4_K_M). We did not want a chatbot bolted onto a game. Gemma has three narrow jobs: interpret memory changes, reason about missions, and comment on a composition. The backend finds real gaps in your sound memory, and Gemma can only choose among them and word them. It cannot invent one, and it cannot decide whether you completed it. Plain code does that.

Gemma never sees ids, audio or your notes. It sees short fact lines and must answer in a JSON schema, citing the facts it used. Then a deliberately suspicious validator checks the answer:

• every number must appear in a cited fact,
• every quoted name must be one the player gave,
• no sound source may be mentioned that is not in the evidence,
• no claim of having seen, visited or been retrained,
• no markup, no ids.

If anything fails, a deterministic template says the line instead and the reason is recorded. The game shows the template line first and swaps in Gemma's when it arrives, so nothing ever waits on the model.

The numbers, stated exactly. Over 19 moments of a real journey, three seeds each (57 calls), 84.2% of Gemma's first attempts passed the validator (48 of 57), and the figure reproduced on a second run. Median warm latency is about 1.3 seconds. One rejection class was our own bug: the validator read the apostrophe in "the ears' strongest guess" as an opening quote, and fixing it moved the rate from 71.9% to 84.2%. We also tried telling the model not to write internal handles like "F1", and it wrote more of them, so we reverted.

What we have not measured: the end-to-end fallback rate after the single retry, and whether people prefer Gemma's lines to the templates. We built a blind rating sheet for that. Nobody has filled it in yet, so we make no claim that Gemma's wording is better. In the final demo run Gemma was called twice and one answer was rejected, which is far too few calls to call a rate.

Current limitations and validation

• Human review is missing. The musical quality has not been rated by people. Automatic checks say the refined composer fixes measurable problems such as repeated bars; they do not say it is beautiful. A blind listening page is ready.
• Novelty detection is a heuristic. CLAP's top guess for a quiet or mixed sound is often wrong. That is the point of the game, and why it asks you.
• Generated sounds are easier than real ones. Variants from one generator are more alike than real field recordings, so the demo shows the mechanism, not field performance.
• Recording can stop when a phone locks. We did not claim background capture.
• Tests: 283 backend tests and 78 frontend tests pass, plus gated real-CLAP and real-Gemma tests.

Why Does Open Innovation Matter?

• Your recordings never leave your machine. A sound from your kitchen or your street is exactly what you do not want on someone else's server. CLAP, the memory, the composer and Gemma all run locally. The phone only captures.
• We could put a leash on the model. Because Gemma runs locally behind Ollama, we could force a JSON schema, measure its failures honestly and put a validator between it and the player. That is why we know the number is 84.2% and not "it seemed fine."
• Parts are swappable. The narrator is a setting. The listening model sits behind a small interface and can be replaced without touching the game.
• It costs nothing to run once the weights are downloaded.

Our Agent Session

We built this with AI coding assistants, so the post is disclosed as AI-assisted: Claude Code for the backend, narrative engine, release engineering and the video tooling, and Codex for the interface.

No public DevRelay agent-session link is included in this draft.

Prize Categories

We are entering three: Overall, Best Use of Gemma and Best Use of ElevenLabs. Here is exactly what each one did in this project, including where it stops.

Best Use of Gemma

Gemma is ECHO-0's voice. Not a chatbot, not a skin over a template: a local gemma3:4b (4.3B parameters, Q4_K_M) that runs through Ollama on a 15 GB laptop, and whose job is to turn what actually happened into words.

Three narrow jobs, each testable on its own:

1. Interpret memory. You taught a sound a name, or corrected the ears. Gemma is handed a few fact lines and says what that means in one or two sentences.
2. Reason about missions. The backend finds real gaps in your sound memory: a name with one example, a sound nobody could name, nothing pitched yet. Gemma picks among them and words them. It cannot invent a mission, and it cannot decide whether you completed one. Plain code does that.
3. Comment on the composition. Using the real source roles and the real plan of the piece it just rendered.

How it is kept honest. Gemma never sees ids, audio, or your notes. Every fact reaches it as a short line with a handle (F1, F2…), and it must answer in a JSON schema, citing the facts it used. Then a deliberately suspicious validator reads the answer before the player does. Every number must appear in a cited fact. Every quoted name must be one you gave. No sound source may be mentioned that the evidence does not contain. No "I walked outside," no "I was retrained," no markup, no ids. If a single check fails, a deterministic template says the line instead, and the reason (invalid_json, for instance) is recorded next to it. Open the "What this voice is grounded in" panel and you can see which facts any line was built from.

What we measured, and what we did not.

Two things we are glad we found. One rejection class was our own bug: the validator read the apostrophe in "the ears' strongest guess" as an opening quote, and fixing it moved the rate from 71.9% to 84.2%. And when we tried to tell the model not to write handles like F1 into prose, it wrote more of them, so we reverted the instruction. The honest fix was a validator, not a plea.

The game shows its template line first and swaps in Gemma's when it arrives, so nothing ever waits on the model, and if Gemma is unavailable the game plays on with the templates.

What we do not claim. We did not measure the end-to-end fallback rate after the single retry, and in the final demo run Gemma was called only twice (one answer valid, one rejected as invalid_json), far too few calls to be a rate. A blind rating sheet that compares Gemma's wording with the templates exists, but nobody has filled it in, so we do not claim Gemma is better than the templates. Gemma was not fine-tuned, it does not hear audio (CLAP does), and it owns no game state. Open weights mattered here for one concrete reason: running it ourselves is what let us force the schema, measure its failures, and put a validator in front of the player.

Best Use of ElevenLabs

ElevenLabs is why the demo has sound at all, and we want to be exact about where it sits: it is demo tooling, not part of the running game.

• The Sound Effects API generated the demo sounds. We could not publish the research audio we tested with, because it is licensed for non-commercial use. So a committed script, scripts/m8_elevenlabs_sounds.py, asks the Sound Effects API (eleven_text_to_sound_v2) for 21 clips: six that drive the full journey (rain in a metal roof gutter, tolling bells, three takes each) and a fifteen-clip pack of wind, footsteps, birds, water and machines. Each is a ten-second text prompt with no music. The script reads the API key from the environment only, converts each clip to 48 kHz mono, levels it so it clears the app's own ingest checks, and writes a manifest with the prompt, model, settings and checksum of every clip. The audio is not committed, but anyone with their own key can regenerate the set.
• Then the game does what it does with real recordings. CLAP hears the generated sounds, a name we teach becomes retrieval memory, and the Composed orchestra in the video is built only from those clips.
• A voice narrates the 50-second video. The narrator is the CRITIC voice from our earlier project, ECHOLALIA, generated with ElevenLabs text-to-speech, seven lines mixed under the app's own music.

The sounds, the narration and the music in the demo are generated or composed, not field recordings, and the video says so on screen. We would rather you trust the project than be impressed by a fake hike.

Team submission — built and designed by İclal Doğan (@iclaldogan) and Melike Zaman (@melike_nesrinzaman_713c0). Melike wrote the audio ingestion, the hearing and memory layer and the first versions of the acoustic atlas; İclal built the narrative engine, the composer integration, the release engineering and the demo.

Go outside. Listen closely. And if ECHO-0 asks what you heard, tell it the truth.