What Research Says About Code Review: Which Practices Actually Catch Defects
Almost every team does code review, and almost every team believes it catches bugs. But is that true? Over the last fifteen years, researchers at Microsoft, Cisco, and several open-source projects have measured what review really does. The answers are more interesting than the folklore.
The Surprise: Reviews Rarely Fix Bugs
Start with what reviewers actually produce. In 2013, Alberto Bacchelli and Christian Bird studied review at Microsoft. They observed developers, interviewed them, surveyed managers, and classified hundreds of comments. Their ICSE paper found that finding defects is still the main motivation, yet reviews turned out to be less about defects than people expected. Instead, they delivered knowledge transfer, team awareness, and alternative solutions.
Open source tells the same story with harder numbers. Moritz Beller and colleagues manually classified more than 1,400 review-triggered changes in two projects. In their MSR 2014 study, the ratio of maintainability fixes to functional fixes was about 75 to 25.

Moreover, the same study found that 7 to 35 percent of review comments were discarded, and 10 to 22 percent of changes were not triggered by any explicit comment. In other words, much of what happens in review is not a reviewer spotting a bug.
Where Review Does Correlate With Quality
Even so, review is not useless for defects. Shane McIntosh and colleagues examined the Qt, VTK, and ITK projects. Their MSR 2014 paper linked post-release defects to two review habits: how much code is reviewed (coverage) and how much reviewers engage (participation).

The details matter. Coverage alone was not enough: it added significant explanatory power in only two of the four studied releases. Participation, however, mattered more. The follow-up journal version added reviewer expertise, and found that components with many changes lacking a subject-matter expert were more defect-prone. Note that these are associations in observational data, so they do not prove cause and effect.
The Cisco Numbers on Size and Speed
Next, consider how to review. The most cited data comes from a ten-month case study of Cisco’s MeetingPlace product: 2,500 reviews, 3.2 million lines of code, and about 50 developers. SmartBear, a code review tool vendor, ran and published it, so treat it as a useful field study rather than a peer-reviewed trial. The summary is in The Largest Case Study of Code Reviews, Ever.
| Practice | What the data showed |
|---|---|
| Review under 200 to 400 lines at a time | Defects found per line dropped sharply above 200 lines |
| Stay below about 400 to 500 lines per hour | Beyond about 450 lines per hour, density was below average in 87% of cases |
| Limit sessions to 60 to 90 minutes | The authors report detection rates fall after that |
| Authors annotate their own changes | Author-prepared reviews varied far less in results |
Sources: the Cisco study summary above. The 87 percent figure comes from Mike Conley’s summary of the full study, which I did not read directly.
One more detail deserves attention. In the same study, 61 percent of reviews found no defects at all, and the average density was 32 defects per thousand lines. That is a reminder that a clean review does not mean clean code, as the Conley summary notes.
Practices That Hold Up, and Their Limits
Small Changes, Reviewed Slowly
This is the most consistent advice, and it is supported by one large industrial study. Small changes also reduce the understanding burden that Bacchelli and Bird identified as the key challenge.
Real Participation, Not Rubber Stamps
McIntosh’s data suggests that a review with real discussion is worth more than a review that merely happened. Therefore, count meaningful comments and questions, not just approvals.
The Right Reviewer
Expertise appears in the journal study as a factor. If a change touches an area where nobody with experience has looked, treat it as higher risk.
Do Not Expect Review to Replace Testing
Because most review fixes are about maintainability, review should sit beside tests, static analysis, and CI. It is a strong tool for readability and shared knowledge, and a moderate one for functional bugs.
Checklist: a review that can catch defects
- The change is small, ideally a few hundred lines at most.
- The author left notes explaining the intent and the risky parts.
- The reviewer set aside real time, not a quick skim.
- Someone who knows the area took part.
- The discussion produced questions or changes, not only an approval.
What We Have Learned
Studies at Microsoft and in open source show that most code review fixes concern maintainability, with roughly three quarters of changes in one analysis, while functional defects are the smaller share. Review still matters for quality: in Qt, VTK and ITK, weak participation and missing expertise were linked to more post-release defects. The Cisco field study suggests small changes and slow, time-boxed reviews find the most. However, the evidence is observational and partly vendor-run, so use these numbers as guidance, keep testing alongside review, and measure your own results.


