Core Java

Property-Based Testing and Fuzzing for Java: Finding the Bugs Your Example Tests Miss

Every Java team knows this quiet problem. The suite is green, the coverage report looks healthy, and yet a bug still reaches production. The reason is simple. Example-based tests only check the inputs that someone thought of, while the bugs that hurt are usually the ones nobody thought of.

Why example tests keep missing the same bugs

Consider the famous midpoint bug. In 2006, Joshua Bloch explained that the binary search in the JDK’s own java.util.Arrays averaged two indexes with (low + high) / 2, which overflows for very large arrays. According to his account, the bug sat unnoticed for roughly nine years, even though the algorithm had been proven correct on paper (Bloch’s post). Careful authors and reviewers missed it. Therefore, a handful of hand-picked examples has little chance.

To make this concrete, I planted three small bugs in a plain-Java experiment. Each one passes the examples most of us would write first.

Planted bugTypical example inputsWhat really breaks it
A. Bucket index: Math.abs(hash) % 105, -7, 100Integer.MIN_VALUE, because its absolute value is still negative
B. Midpoint: (a + b) / 2(2, 4), (10, 20)Two large ints whose sum overflows
C. Leap year without the 400-year rule2024, 2023, 1900Any multiple of 400, such as 2000

Two tools, two kinds of search

jqwik: you state the rule, it generates the inputs

jqwik is a test engine for the JUnit platform. You write a property, which is a rule that must hold for all valid inputs, and you mark each parameter with @ForAll. By default, jqwik runs 1,000 tries per property, and it mixes in edge cases along the way. When a property fails, it also shrinks the failing sample to a simpler one and reports the random seed, so you can reproduce the run (see the jqwik user guide).

Jazzer: coverage feedback finds the hidden doors

Jazzer is a coverage-guided, in-process fuzzer for the JVM, developed by Code Intelligence and based on libFuzzer (project page). Instead of generating data blindly, it observes which branches each input reaches and keeps the inputs that reach something new. As a result, it can walk through nested checks that random data would almost never pass. In JUnit, you use the @FuzzTest annotation.

jqwikJazzer
QuestionDoes my rule hold for all valid inputs?Can any input crash or break this code?
InputTyped values with constraintsRaw bytes, or a FuzzedDataProvider
You writeThe rule (the oracle)A target that calls the code; exceptions and detected security issues are the signal
Typical useEvery buildQuick regression runs in CI, longer sessions on a schedule

A small demo you can run

You need a recent JDK and Maven. First, add the three test dependencies. The versions below are the ones I confirmed in the projects’ documentation and release history; newer ones should work as well.

<dependencies>
  <dependency>
    <groupId>net.jqwik</groupId>
    <artifactId>jqwik</artifactId>
    <version>1.10.2</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>com.code-intelligence</groupId>
    <artifactId>jazzer-junit</artifactId>
    <version>0.30.0</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.14.4</version>
    <scope>test</scope>
  </dependency>
</dependencies>

A property for the leap-year bug

Next, we compare our method with the JDK’s trusted Year.isLeap. The class name ends in Tests so that Maven picks it up by default.

package demo;

import java.time.Year;
import net.jqwik.api.ForAll;
import net.jqwik.api.Property;
import net.jqwik.api.constraints.IntRange;

class LeapYearTests {

    // code under test, with a planted bug: no 400-year rule
    static boolean isLeap(int year) {
        return year % 4 == 0 && year % 100 != 0;
    }

    @Property
    boolean agreesWithTheJdk(
            @ForAll @IntRange(min = 1, max = 3000) int year) {
        return isLeap(year) == Year.isLeap(year);
    }
}

The property fails only for the seven multiples of 400 between 1 and 3,000, namely 400, 800 and so on up to 2800. jqwik should report one of them as the failing sample. Meanwhile, three typical examples (2024, 2023, 1900) all pass.

A fuzz test for a gated input

Now the fuzzing side. The planted bug hides behind a four-byte header, which is exactly the kind of gate that blind data rarely passes.

package demo;

import com.code_intelligence.jazzer.junit.FuzzTest;

class DecoderFuzzTest {

    // code under test, with a planted bug behind a 4-byte header
    static int decode(byte[] in) {
        if (in.length >= 4 && in[0] == 'F' && in[1] == 'U'
                && in[2] == 'Z' && in[3] == 'Z') {
            throw new IllegalStateException("planted bug reached");
        }
        return in.length;
    }

    @FuzzTest
    void decodeNeverThrows(byte[] data) {
        decode(data);
    }
}

Finally, run both. Without the environment variable, Jazzer only replays stored inputs, which keeps normal builds fast. With JAZZER_FUZZ=1, it actively searches (the command below is for Linux and macOS).

# jqwik runs like any other test
mvn test -Dtest=LeapYearTests

# Jazzer fuzzes only when JAZZER_FUZZ is set
JAZZER_FUZZ=1 mvn test -Dtest=DecoderFuzzTest

One caution before adopting Jazzer: its licence terms have changed over time, so read the licence of the exact release you choose on the project page.

What the numbers say

To measure the difference, I wrote a library-free Java harness and ran each setting 2,000 times with different random seeds. It is a toy model of the two ideas, not jqwik or Jazzer themselves. First, the three bugs with a budget of 1,000 tries, which is jqwik’s default.

self-run plain-Java experiment, 2,000 seeds per setting, 1,000 tries per run. The “boundary mix” generator is a simplified stand-in (5% chance per parameter of picking 0, 1, -1, MIN_VALUE, MAX_VALUE or a range limit), not jqwik’s internal algorithm.

The examples caught none of the three bugs. Plain random data found bug B almost instantly (median of 2 tries) and bug C in about nine runs out of ten. However, it never found bug A, because MIN_VALUE is one value out of roughly 4.3 billion. Mixing in boundary values fixed that: all 2,000 runs found it, with a median of 51 tries in my toy. This is why edge-case injection matters so much. Budget matters too. For bug C, random search had found it in about 21% of runs after 100 tries, 51% after 300, and 91% after 1,000.

Next, the gated input. A blind generator has a one in 2564 chance per try of producing the right four bytes, which is about one in 4.3 billion. I compared it with a tiny feedback-guided loop that keeps any input reaching a new branch.

self-run, 30 runs per approach. The blind runs each stopped at 20,000,000 executions without a crash (0 of 30 found it); the guided runs found it in all 30, with a median of 58,848 executions (range 13,012 to 141,323). The guided loop uses branch coverage only; real Jazzer adds richer instrumentation.

The guided loop needed tens of thousands of executions, while the blind one found nothing in twenty million. In other words, feedback turns one impossible jump into four easy steps.

Quick checklist

  • Write properties for rules such as round trips, ordering, and agreement with a trusted implementation.
  • Point fuzzers at parsers and decoders that handle untrusted bytes.
  • Keep every input a tool finds as a permanent regression test.
  • Run jqwik on every build, and schedule longer fuzz sessions separately.
  • Never turn off boundary values, because they find the bugs random data cannot.

What We Learned

Example tests verify what we already imagined, so they keep missing overflow, boundary and rare-rule bugs. Property-based testing with jqwik fixes this by checking a rule against many generated inputs, mixing in edge cases, and shrinking failures. Fuzzing with Jazzer goes one step further for byte-level code, because coverage feedback lets it pass nested checks that blind data cannot. In my small experiment, three example-passing bugs were found by random or boundary-aware generation, and a four-byte gate that defeated twenty million blind tries fell to feedback within about 59,000 executions. Therefore, use both tools: properties for rules, fuzzing for hostile input, and examples for documentation.

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button