Fix a bug every 8.7 minutes

1 reader likes this.
annoying bugs

With almost 30 years of active development under its belt, BRL-CAD is believed to be the second oldest open source codebase in the world that’s still under active development (VistA, the EHR of the Veterans Administration being the oldest). It has also been the primary tri-service solid modeling CAD system used by the U.S. military to model weapons systems for vulnerability and lethality analyses.

BRL-CAD is a powerful, cross-platform, open source combinatorial Constructive Solid Geometry (CSG) solid modeling system that includes interactive 3D solid geometry editing, high-performance ray-tracing support for rendering and geometric analysis, network-distributed framebuffer support, image and signal-processing tools, path-tracing and photon mapping support for realistic image synthesis, a system performance analysis benchmark suite, an embedded scripting interface, and libraries for robust high-performance geometric representation and analysis.

BRL-CAD is written in a combination of C/C++ code, is just over 1 million lines of code, and includes a developer community of approximately 12 core developers and 20 committers to the source code repository.

Due to the critical nature of how and where the code is used, BRL-CAD has a long-standing commitment to code quality and implements strict compliance in the code. The BRL-CAD team has spent a lot of time maintaining and cleaning the codebase, including frequent refactoring, and has implemented a variety of quality measures over the course of development, including a Coverity Scan.

The solid modeling system is frequently used in a wide range of military, academic, and industrial applications including in the design and analysis of vehicles, mechanical parts, and architecture. The package has also been used in radiation dose planning, medical visualization, computer graphics education, CSG concepts and modeling education, and system performance benchmark testing among other purposes.

In April 2011, the BRL-CAD source code was put through Coverity Static Analysis as part of its involvement in Coverity Scan, evaluating approximately 840,000 lines of code (1.2 million actual codebase size with comments and whitespace). Over 1,800 defects were flagged by Coverity Scan. According to a project leader, Coverity showed BRL-CAD "a different caliber or class of issue" than other testing and quality measures they have adopted in the past. The project leaders were "quite ecstatic" with the results provided via Coverity Scan, but then needed a way to address these newly found defects.

In November 2011, Christopher Sean Morrison, a lead contributor to BRL-CAD, posted a call for participation to the developer community for a physical get-together to share knowledge about defects and help resolve the issues detected. The call for participation can be found here:

Last week, we were finally able to get a FULL valid Coverity scan of the BRL-CAD source code! If you're interested in helping resolve the  issues detected, please join in our development discussion on the brlcad-devel mailing list or IRC channel and introduce yourself.

The outcome of this effort was published in the Coverity Scan: Open Source Integrity Report of 2011:

The BRL-CAD project community off-site was held in January 2012, just outside Baltimore. Half of the core development team participated in person and two contributed remotely for eight hours a day over a five-day span. The BRL-CAD development team first outlined the entire defect management workflow. 

The goal of the off-site was to fix all outstanding defects instead of a multi-pass "prioritize then fix" approach. Specifically:

  • Inspect and fix all issues discovered by Coverity
  • Peer review, verify, and validate all changes
  • Categorize and document all changes
  • Educate on common defect patterns

So how did the team do?

  • Inspect and fix all issues discovered by Coverity – 90%
  • Peer review, verify, and validate all changes – 10%
  • Categorize and document all changes – 100%
  • Educate on common defect patterns – 100%

The group was able to decrease outstanding defects:

  • From 1,840 to 186 after five days, a beginning defect density of .0022 which decreased to .00022 (defects per thousand lines of code)
  • This equated to approximately 330 defect fixes per day
    • 42 fixes per hour
    • A fix every 87 seconds across the team
  • Per individual developer this equated to approximately
    • 55 defect fixes per day
    • 6.9 fixes per hour
    • A fix every 8.7 minutes

This process illustrates the unparalled efficiency of open source agile practices.

Event though only 10% of the fixes started the code review proces, The feat of being able to address 90% of the defects in a project (about 1,700 defects in this case), by a focused effort of one week, is something that traditional software developement methodologies can only dream of.

In open source, this can be just "another day at the office."

Congratulations to the BRL-CAD team for this impressive achievement. More details about the process are available in the Coverity Report.

User profile image.
Luis Ibáñez works as Senior Software Engineer at Google Inc in Chicago.

Comments are closed.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.