FieldIssue: take a walk, report a problem, come back with evidence

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

Most "report a pothole" apps stop at the photo. FieldIssue is built around the second walk.

Snap a broken bench, an overflowing bin or a cracked pavement, add a short note, and an open-weight Gemma model writes up what it sees. Walk past again later, take another photo, and FieldIssue compares the two: which conditions were added, which were removed, which haven't changed. Over time, a report becomes a dated trail of evidence instead of a single complaint.

A real walk: I photographed this cut tree and litter in Lucknow on 9 October and reported it from my phone. Gemma listed three conditions: cut or broken branches, piled dried branches and leaves, and plastic waste.

On the same walk I reported a second spot about 50 metres away, FI-000006, where fallen branches block a dirt path. Both reports carry my phone's GPS and capture time.

The next morning I went back to FI-000007. My first revisit photo was the wrong tree, and it exposed a real bug: FieldIssue compared my second, correct photo against that wrong one, and the saved result said "not comparable" while still listing two added conditions at 0.9 confidence. I fixed it the same morning. Owners can now exclude a wrong photo without deleting it, comparisons skip excluded evidence, and a "not comparable" result must have empty change lists and zero confidence. Old results stay in the history, marked superseded.

With the wrong photo excluded, Gemma compared the 9 October original with the 10 October photo: Unchanged, citing the same severed trunk and the same pile of branches and litter. The report stays open, because a person decides. One honest gap: both revisit photos were saved with the report's coordinates rather than fresh GPS, so new revisits now record their own location. See the full evidence walkthrough.

Two rules keep it honest:

• The model recommends, a person decides. Gemma suggests a category and severity for each photo, and a status when it compares two photos, but only a person changes a report's status, and a manual resolution is labelled as one in the history.
• Identical file, no new evidence. Upload the same image twice and you get "Repeated photo: no new evidence." It's an exact-byte check: it stops lazy re-uploads, not a determined faker.

The safeguard in action, tested with a sample photo and placeholder coordinates.

Beyond single reports, you can plan a walk past nearby issues, keep private saved walks with an optional account, and run a private community that assigns reports and needs fresh compared evidence plus two member approvals to close one.

Demo

Open FieldIssue · Explore reports · Model Lab

A 2-minute tour (no sign-up needed; the free Render instance may take about a minute to wake):

1. Open the evidence walkthrough from my real walk, then a sample report that shows the repeated-photo safeguard.
2. Play its ElevenLabs audio briefing, then open the Lab's recorded Tinker examples: all 234 test notes with the base, five-example and fine-tuned answers side by side, mistakes included.
3. Create your own report, come back with a new photo, and compare.
4. In Walk, pick a starting point, add nearby open issues and open the route in Google Maps.

It's an installable web app: if you lose signal mid-walk, captures are queued on your phone and uploaded only after you review them.

Code

GitHub repository · MIT license · Setup and demo guide

Started on 6 October 2026. A labelled fixture mode runs locally with no API keys; the hosted demo uses the real integrations.

How I Built It

React, Vite, Tailwind, shadcn/ui and Leaflet on the front. A TypeScript Hono API runs typed Mastra workflows for observations and revisits. One free Render container runs the Node API and a private FastAPI process, with Render PostgreSQL/PostGIS as the source of truth. Gemma runs through Google's hosted API, and Tiger Data keeps a separate pgvector index for semantic search.

Each model has one job: Gemma reads photos, Tinker reads the reporter's words, Backboard compares two open models side by side. None of them can close an issue on its own. Model output is schema-validated before it's saved, and offline retries reuse an idempotency key so a dropped connection never creates a duplicate report.

What the fine-tune actually bought

With Tinker I fine-tuned Qwen3-8B (LoRA rank 16) on 408 synthetic field notes in English, Hindi and Hinglish. The severity rubric and both test sets were frozen before any model ran, and epochs and learning rate were picked on 66 separate validation notes. Then I tested it against two baselines: the base model as is, and the base model given five worked examples, which is the comparison a skeptic would ask for.

A second training seed landed at 179/204, so it isn't one lucky run. On the synthetic set the gain over five examples is +9.3 points (95% interval +3.4 to +15.2). The 311 reports are genuine public text written by strangers, and they're where the five-example prompt stopped helping. For "Needles left behind" the five-example prompt said LOW; the fine-tune said HIGH. The whole run cost about $0.94 in credits.

It isn't a clean win, and the full evaluation says so:

• On the 311 reports, category accuracy fell from 25/30 to 20/30.
• The fine-tune leans toward MEDIUM. Always guessing MEDIUM would score 17/30 on the 311 slice, so 23/30 beats that, but by less than the headline suggests.
• An AI assistant labelled the 311 reports against my rubric; they aren't independent human judgments, and none are in Hindi.
• Its first live call on my real walk note, "Tree has been cut down", came back as a HIGH obstruction. Nothing in the note says it blocks anything. That answer stays on the report and didn't change the issue's status.

All 234 test answers, mistakes included, are browsable in the Lab, with filters by language and source. The two earlier, smaller runs stay published too.

In the app, Interpret with trained model turns an owner’s note into a saved category, severity, condition and evidence summary after consent. It sends no photo or coordinates and does not overwrite the report’s classification or resolve it.

I also tried TabPFN for ranking which issues to walk past. The first test was too small to mean anything: 16/24 against a 17/24 "always guess the common answer" baseline. So I reran it properly on synthetic data, at training sizes from 32 to 1,024 rows with five repeats each, against logistic regression, gradient boosting and the formula that generated the data. I wrote the pass rule down first: TabPFN had to beat logistic regression at 128 rows or fewer. It didn't (AUC 0.706 vs 0.722 at 128 rows) and only edged ahead at 1,024. That's fair, since the data came from a logistic formula. It stays in the Lab and out of real recommendations, and the learning curve publishes every repeat.

What broke

• Saved walks failed in the browser even though the API tests passed: the location picker was sending display metadata to an endpoint that expected bare coordinates. Lesson: test the whole form, not just a clean payload.
• My ElevenLabs key expired mid-week, and the cache hid it. Saved briefings kept playing; only a fresh generation returned 401. I now check fresh calls separately from cached ones.
• Sentry separated a soft failure from a hard one. A SerpApi lookup for a test report returned no verified place context, but the Gemma report still saved, and Sentry logged it as PLACE_CONTEXT_UNAVAILABLE without leaking notes or photos (event, training trace).

The verified build passes 292 API and Python tests plus 16 browser tests, plus hosted checks for account isolation, saved walks, membership revocation and audio playback (acceptance ledger).

GitHub Actions: checking the shipped container

The Backend CI workflow runs on pull requests and pushes to main. Alongside 199 API tests, 93 Python tests and 16 browser tests, it verifies PostGIS and pgvector, exercises the development and compiled APIs over HTTP, and checks the Render container under a 512 MiB memory limit. It checks that missing production secrets stop startup, FastAPI listens only on loopback, and the container exits if its Python service dies. All jobs passed in run 38125992350. CI uses labelled fixtures; live provider checks are recorded separately.

Why Does Open Innovation Matter?

If an app is going to tell a city "this got worse", people should be able to check how it decided. Anyone can read FieldIssue's prompts, check the output schema, rerun the evaluations and see exactly where the models failed. Open weights also mean the vision model is swappable, and a narrow task like note reading can be fine-tuned and measured against its base. The app is MIT-licensed and was built on free tiers and sponsor credits.

Limits

• Most model evaluation here uses synthetic data; the 30 real 311 reports were labelled by an AI assistant, not by people. Next: human-labelled notes and a change history from real revisits.
• Free tiers run out: the Tinker checkpoint expires on 6 November 2026, the Render database on 7 November and the ElevenLabs key on 8 November. Gemma, nearby search, speech and Backboard share a cap of 60 provider units a day. Tinker and TabPFN have separate daily and lifetime limits. Saved results stay viewable.
• Sentry traces are sampled, so not every request has a complete trace.

My Agent Session

Codex built the backend and ran integration and browser checks. Claude Opus built the frontend and reviewed the later changes. The Entire CLI captured the frontend session; a reviewed excerpt of nine real messages is public. A public Codex coding checkpoint, ID 01M4MS0B5PNB39JGA84HZTC0GH, contains reviewed coding records and the committed synchronization verifier. Its session and Changes loaded in signed-out Safari Private Browsing. The landing illustration is AI-generated and labelled as such. This write-up was prepared with AI assistance.

Prize Categories

• Best Use of Gemma: reads every photo and compares selected photos, such as the original against a revisit, listing what was removed, added and unchanged.
• Best Use of Tinker: Qwen3-8B fine-tuned on 408 English, Hindi and Hinglish notes; severity 178/204 vs 159/204 for a five-example prompt, and 23/30 vs 15/30 on real Boston 311 reports, live in the app.
• Best Use of Render: one container runs the web app, Node API and private FastAPI process, with Render PostgreSQL/PostGIS as the database.
• Best Use of TabPFN: a learning curve against logistic regression, gradient boosting and the true formula, with the pass rule written first; it failed at small sizes, so I published that and kept it out of recommendations.
• Best Use of Mastra: typed workflows run every new observation and every revisit comparison.
• Best Use of Sentry Agent Tracing: sanitized spans on each workflow step, with soft provider failures like PLACE_CONTEXT_UNAVAILABLE recorded.
• Best Use of SerpApi: nearby place context when a place is found; a report still saves without one.
• Best Use of ElevenLabs: cached spoken briefings for each report.
• Best Use of Tiger Data: hybrid keyword and pgvector search over reports (all-MiniLM-L6-v2), with a retry queue so indexing never blocks a report.
• Best Use of Backboard: Gemma 3 and Qwen 2.5 read the same report side by side, so a person can see where they disagree.
• Best Use of Entire: public checkpoint 01M4MS0B5PNB39JGA84HZTC0GH with reviewed coding records and a code diff, verified signed out.
• Best Use of GitHub Copilot (via GitHub Actions): automated API, database, HTTP and Render container checks, with the workflow and passing run linked above. The category explicitly includes GitHub Actions automation.