Software Development

The Economics of Flaky Tests: What They Really Cost a Team

A flaky test rarely breaks production. Instead, it quietly taxes every pull request your team opens.

The Hidden Tax

Every team knows the moment. A build turns red, someone shrugs, and the pipeline is simply run again. It passes, and everyone moves on. That small shrug is where the money leaks away, because each rerun costs minutes of waiting, a context switch, and a little more distrust in the suite.

This article puts numbers on that tax. First, we look at what published research says. Then we use simple math to show why even tiny flake rates matter at scale.

What the Research Actually Says

Google’s Numbers

In a May 2016 post on the Google Testing Blog, Google reported that about 1.5% of all its test runs show a flaky result, meaning the same code both passed and failed. It also found that almost 16% of its tests have some level of flakiness. In the post’s own example, a flaky rate of 1.5% means that about 15 tests will likely fail in a batch of 1,000, and each failure needs investigation by a build cop or developer.

One caution is worth making here. You may have seen the “16%” quoted as the share of developer time lost to flaky tests. That reading is wrong. The 16% describes the share of tests with some flakiness, not time. Please check any article that repeats it as a time figure.

What Causes Them

Next, why do tests flake? A well-known study by Luo, Hariri, Eloussi and Marinov, An Empirical Analysis of Flaky Tests (FSE 2014), examined 201 fixes in open-source projects. Its top three categories are shown below. As co-author Lamyaa Eloussi summarised on the Salesforce engineering blog, they cover most of the cases.

CauseShare of fixesTypical remedy found in the study
Async wait45%54% were fixed by waiting for the result instead of sleeping
Concurrency20%Data races, atomicity bugs and deadlocks
Test order dependency12%74% were fixed by cleaning shared state
Everything else23% (remainder)Network, time, unordered collections, and more

In short, most flakes are bugs in the tests themselves. Consequently, they are fixable engineering work, not bad luck.

Why Small Rates Add Up Fast

Here is the part that surprises people. A suite passes only if every test passes. So if each test flakes with probability p, a suite of n tests is green with probability (1 – p) to the power n, when the code is correct. A quick check: a 0.05% flake rate across 100 tests gives a green run about 95% of the time.

Computed from the formula above. It assumes independent flakes, which is a simplification.

Look at the orange line. With a seemingly harmless 0.05% rate per test, a 2,000-test suite is green only about 37% of the time. Therefore, the bigger the suite, the more a tiny rate hurts.

A Simple Cost Model

Now let us turn those probabilities into hours. The assumptions below are mine, not measured data, so please replace them with your own numbers. I assumed 20 developers, 4 CI runs per developer per day, 230 working days, a 2,000-test suite, and 30 minutes lost for each false red build. I also assumed no automatic retry.

Illustrative model, not measured data. Hours lost = runs per year x chance of a false red x 0.5 hours.
Per-test flake rateFalse red buildsHours lost per yearShare of team capacity
0.001%2% of runs1820.5%
0.005%9.5% of runs8762.4%
0.01%18.1% of runs1,6684.5%
0.02%33.0% of runs3,0338.2%

Capacity here means 20 developers times 230 days times 8 hours, or 36,800 hours. Even the modest 0.01% row costs about 1,668 hours, or roughly 208 working days, every year.

The model also leaves out two costs that are harder to count. One is the cost of ignored failures, because people who expect a flake may dismiss a real bug. The other is infrastructure. A research paper from George Mason University, Root Causing Flaky Tests in a Large-Scale Industrial Setting, reports Google spending roughly 2 to 16% of its testing budget on reruns. It cites another source for that range, and I could not trace the original, so treat it as approximate.

What to Do About It

Google’s post describes the tools it uses: rerunning only failed tests, automatic reruns, and a tool that quarantines a test when its flakiness grows too high. A test can also be marked flaky, so that it must fail three times in a row before a failure is reported. These are good first aid. However, reruns hide the cost instead of removing it, so the real fix is to repair the cause.

Quick checklist

  • Measure first: record every rerun and tag each failure as real, flaky or unknown.
  • Quarantine proven flakes quickly, and file a ticket for each one.
  • Replace sleeps with waits for a condition.
  • Make every test create and clean its own data.
  • Run tests in random order now and then to expose hidden dependencies.

What We Have Learned

Flaky tests are a recurring tax, not a rare nuisance. Google reports about 1.5% of test runs as flaky and almost 16% of tests as touched by flakiness, and the 16% is a share of tests, not of time. Because a suite is green only when every test passes, a tiny per-test rate compounds quickly: at 0.05% per test, a 2,000-test suite is green about 37% of the time. In my illustrative model, a 0.01% rate costs a 20-person team around 1,668 hours a year. Since the best-known study traces most flakes to async waits, concurrency and test order, the cure is mostly better tests. So measure your own reruns, quarantine fast, and fix the cause.

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button