Software Development

What Research Says About Code Review: Which Practices Actually Catch Defects

Almost every team does code review, and almost every team believes it catches bugs. But is that true? Over the last fifteen years, researchers at Microsoft, Cisco, and several open-source projects have measured what review really does. The answers are more interesting than the folklore.

The Surprise: Reviews Rarely Fix Bugs

Start with what reviewers actually produce. In 2013, Alberto Bacchelli and Christian Bird studied review at Microsoft. They observed developers, interviewed them, surveyed managers, and classified hundreds of comments. Their ICSE paper found that finding defects is still the main motivation, yet reviews turned out to be less about defects than people expected. Instead, they delivered knowledge transfer, team awareness, and alternative solutions.

Open source tells the same story with harder numbers. Moritz Beller and colleagues manually classified more than 1,400 review-triggered changes in two projects. In their MSR 2014 study, the ratio of maintainability fixes to functional fixes was about 75 to 25.

 Beller et al. (2014), two open-source projects, over 1,400 changes. The 75:25 split is the authors’ reported ratio, and they note it matches earlier industrial findings.

Moreover, the same study found that 7 to 35 percent of review comments were discarded, and 10 to 22 percent of changes were not triggered by any explicit comment. In other words, much of what happens in review is not a reviewer spotting a bug.

Where Review Does Correlate With Quality

Even so, review is not useless for defects. Shane McIntosh and colleagues examined the Qt, VTK, and ITK projects. Their MSR 2014 paper linked post-release defects to two review habits: how much code is reviewed (coverage) and how much reviewers engage (participation).

McIntosh et al. (2014). Values are the maximum additional post-release defects per component that the authors’ models estimate for lax review. These are upper estimates from regression models, not averages.

The details matter. Coverage alone was not enough: it added significant explanatory power in only two of the four studied releases. Participation, however, mattered more. The follow-up journal version added reviewer expertise, and found that components with many changes lacking a subject-matter expert were more defect-prone. Note that these are associations in observational data, so they do not prove cause and effect.

The Cisco Numbers on Size and Speed

Next, consider how to review. The most cited data comes from a ten-month case study of Cisco’s MeetingPlace product: 2,500 reviews, 3.2 million lines of code, and about 50 developers. SmartBear, a code review tool vendor, ran and published it, so treat it as a useful field study rather than a peer-reviewed trial. The summary is in The Largest Case Study of Code Reviews, Ever.

PracticeWhat the data showed
Review under 200 to 400 lines at a timeDefects found per line dropped sharply above 200 lines
Stay below about 400 to 500 lines per hourBeyond about 450 lines per hour, density was below average in 87% of cases
Limit sessions to 60 to 90 minutesThe authors report detection rates fall after that
Authors annotate their own changesAuthor-prepared reviews varied far less in results

Sources: the Cisco study summary above. The 87 percent figure comes from Mike Conley’s summary of the full study, which I did not read directly.

One more detail deserves attention. In the same study, 61 percent of reviews found no defects at all, and the average density was 32 defects per thousand lines. That is a reminder that a clean review does not mean clean code, as the Conley summary notes.

Practices That Hold Up, and Their Limits

Small Changes, Reviewed Slowly

This is the most consistent advice, and it is supported by one large industrial study. Small changes also reduce the understanding burden that Bacchelli and Bird identified as the key challenge.

Real Participation, Not Rubber Stamps

McIntosh’s data suggests that a review with real discussion is worth more than a review that merely happened. Therefore, count meaningful comments and questions, not just approvals.

The Right Reviewer

Expertise appears in the journal study as a factor. If a change touches an area where nobody with experience has looked, treat it as higher risk.

Do Not Expect Review to Replace Testing

Because most review fixes are about maintainability, review should sit beside tests, static analysis, and CI. It is a strong tool for readability and shared knowledge, and a moderate one for functional bugs.

Checklist: a review that can catch defects

  • The change is small, ideally a few hundred lines at most.
  • The author left notes explaining the intent and the risky parts.
  • The reviewer set aside real time, not a quick skim.
  • Someone who knows the area took part.
  • The discussion produced questions or changes, not only an approval.

What We Have Learned

Studies at Microsoft and in open source show that most code review fixes concern maintainability, with roughly three quarters of changes in one analysis, while functional defects are the smaller share. Review still matters for quality: in Qt, VTK and ITK, weak participation and missing expertise were linked to more post-release defects. The Cisco field study suggests small changes and slow, time-boxed reviews find the most. However, the evidence is observational and partly vendor-run, so use these numbers as guidance, keep testing alongside review, and measure your own results.

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button