You Need Evals to Ship AI Features

When you build a traditional app, you write tests and sleep easy (well, easy-ish). But turn an LLM loose in your product and suddenly, you’re not just squashing bugs. You’re wrangling a chaos engine.

So, how do you keep AI on the rails and measure its success?

We’ve spent a whole bunch of time engineering and improving agentic AI features in Fusion, and for us, a good chunk of our success comes down to evals.

The real problem with building (and maintaining) AI features

AI features don’t just break. They get weird.

Ship a new component, change a prompt, or swap model versions, and there’s no telling what you’ll actually get. Yesterday’s golden output becomes today’s “why did it just sell a car for one dollar??” moment.

With AI, regression isn’t an edge case; it’s the status quo. Every tweak in the LLM supply chain risks breaking flows that already worked: sneaky performance drops, subtle COPPA violations, or Grok calling itself “MechaHitler.” And when nobody’s looking, these regressions slip by, all while your CI stays smugly green.

But the real stake isn’t just stability. It’s progress. When a new model version drops, or your team hacks on a “better” onboarding flow, can you actually prove that users are happier? Are thumbs-ups climbing, or did new “magic” quietly nuke the old wins? Classic tests can’t answer that.

Long story short, if you care about not waking up to Slack fires (and about steadily making your product better), you need a way to measure both safety and progress. That’s where evals come in.

What are evals and why do they matter?

An eval is a bit like an automated code review for your AI’s behavior.

Classic tests are great for checking “Does 2 + 2 still equal 4?” Evals pick up where those assert statements give up, like, “Is this chatbot actually being helpful, or is it hallucinating a pizza tracker?” They’re automated, repeatable scenarios that pin down the wildest corners of LLM behavior and ask: Are we still good here? Did we get better, or just weird in new ways?

Evals don’t replace your test suite. They live inside it:

Evals are about making measurable promises: No matter how you change your prompts or which shiny new model you’re rolling out this week, core product flows don’t regress. Or, at the very least, if regression happens, you spot it before your users do (and before you’re in the next “lol dumb AI did what” thread on X).

More than just catching safety issues, evals let you experiment—try new models, prompts, agent flows—and show, with hard numbers, whether the changes actually made users happier, faster, or less likely to faceplant. If green-test dopamine is your thing, evals add a new flavor: the confidence to ship real AI stuff without just praying it works in prod.

Evals in action: How we maintain LLM stability and progress in Fusion

Let’s get specific about how this actually works in code, and not just on tech Twitter. Throughout the development of our AI products like Fusion we consistently use evals in development and production.

Dev-time evals are the “bug zapper” for regressions

When someone at Builder lands a PR with a new AI feature, it doesn’t just run unit tests and hope for the best. Our CI pipeline also spins up a fresh set of evals: Does the bot still answer basic onboarding questions? Does it accidentally slip back into that multi-paragraph-answer energy we worked so hard to squelch a few months back?

The idea isn’t to catch all weirdness (unfortunately impossible), but to make regression a thing that gets face-time with the dev, not a user. You want to be the first to know when your LLM starts roasting product managers again.

Every user-reported issue becomes a new eval, making the AI better and better over time. When CI says green, we know that means “Fusion still works, isn’t more annoying, and probably got better,” and not just, “Nothing crashed.”

Evals tied to A/B tests let us see we’re getting provably better

Model upgrades aren’t a flex. They’re a risk.

When we moved from Claude 3.7 to 4, we didn’t just let optimism steer the rollout. Instead, we ran evals: did the new model improve answer helpfulness? Was it less “stuck in the prompt” when users asked wildly off-topic stuff? Did user thumbs-up rates in actual, anonymized A/B tests show genuine progress, or did we just get fancier errors?

Sometimes, new models can tank key metrics. For instance, Sonnet 3.7 was so much more verbose than 3.5 that before being able to roll it out to prod, we had to tweak the system prompt a bunch just to get it to shut up. Evals helped us not launch the model blindly, saving us money and keeping users happy.

Evals don’t just measure; they defend. Tie your “ship-it” button to a set of minimum eval thresholds and you’ll have far fewer consensus-shipped features that are, in fact, regressions.

Evals in prod act as real-time guardrails for users

In Fusion, evals aren’t just a one-time hurdle to pass on deploy. They’re also woven into the live user experience as real-time guardrails. Some folks might not call these evals, but it’s the same idea—just in production.

How does this look in practice?

Design systems aren’t a suggestion. They’re enforced.

The Fusion agent has a real map of your design system and components. When a user or teammate prompts a design change, guardrail evals step in to verify: Did the output use actual Button, Card, or Grid components with the correct tokens? Is spacing, color, and hierarchy enforced?

If the LLM sneaks in its own rogue styles or one-off colors, the eval can block or even fix it before the AI generation completes. This is part of why our Figma-to-code conversions are so reliable; not only do they grab all relevant tokens and measurements, but they also, through forced LLM iteration, leave almost no room for AI hallucination.

Live context means no more AI hallucinations about your UI.

Unlike agents that just take a guess based only on code, Fusion’s checks also run on the rendered DOM and live app state.

When the agent “fixes” a bug or implements a change, we can trigger screenshot diffs, inspect the DOM structure, and check the new output against the code and design system. Any drift or accidental breakage? The system flags it, surfaces it to the LLM, and kicks off a correction.

Here’s an example where I had an odd bug with visual alignment on a notification component, when there were zero items in the cart: