How Johns Hopkins Enhanced Software Reliability and Developer Productivity With Klocwork

Klocwork Helps Johns Hopkins…

  • Accelerate Development Times

  • Prevent, Detect, and Eliminate Defects

  • Track and Manage Project Development

Project Background

The Defense Advanced Research Projects Agency (DARPA) Revolutionizing Prosthetics program (RP2009) created a fully functional (motor and sensory) upper limb that responds to direct neural control. The technology combines neuroscience, robotics, sensors, power systems, actuation, and complex embedded software to deliver a prosthetic that is far more advanced than any device currently available. The results of this program allowed upper limb amputees to have as normal a life as possible despite their severe injuries.

Why Static Analysis Was Necessary

The Johns Hopkins University (JHU) Applied Physics Laboratory (APL) software team was tasked with building the embedded software component for RP2009. The software was designed with a distributed microprocessor and microcontroller architecture with two OMAP (Open Multimedia Application Platform) Command & Control (C&C) processor nodes. One of which provides coordinated and top-level hierarchical control for the Modular Prosthetic Limb (MPL), and the other processes patient signals and acts as the gateway for patient interface devices.

These complex software systems must accept user signals for translation into limb motion and provide sensory feedback (tactile and temperature) to the patient. Those complex requirements along with the following realities made the adoption of a static analysis tool an important priority:

  • Tight project timeline and the need to maximize productivity.
  • Need for increased software quality given the safety-critical and real-time requirements.
  • U.S. Food and Drug Administration (FDA) guidance for medical devices strongly recommends the use of static analysis.
  • Multiple team members had positive prior experiences with modern static analysis tools.

Why Klocwork was the Ideal Tool

Based on their base experience with static analysis tools, the APL C&C software team knew that these tools can significantly reduce costs by discovering defects earlier in the software development life-cycle. As an automated technology, they require very limited human effort when compared against manual code reviews and traditional execution-based testing.

Of course, they do not remove the need for code reviews and execution-based testing, but they offer a very effective supplement in the battle to remove software defects.

Since the benefits were well understood, the team focused its energy on ensuring they selected a static analysis tool that would easily fit into their environment, which consisted of:

  • C source code (C89)
  • QNX Neutrino 4.6.1 RTOS
  • GCC 4.3.3 compiler
  • Texas Instruments OMAP 3503 processor
  • Simulink Real-Time Workshop and Embedded Coder
  • CruiseControl for continuous integration

Based on a recommendation by another technology vendor, the APL C&C software team initiated an on-line demonstration and a 30-day evaluation from Klocwork, with very positive results. The team did not require technical support during the evaluation due to the excellent documentation and intuitive user interfaces. All these factors, combined with a simple licensing model, led the team to procure Klocwork Insight for use on the RP2009 project.

“We had a very tight schedule and without Klocwork Insight, we would have had difficulty meeting our objectives on time. Even with a relatively small development team, we estimate 900 person hours saved by using Klocwork.”

— Justin Thomas, Software Team Lead, APL C&C

Tight Integration with Agile Build Process

The APL C&C software team had a pre-existing Agile build process known as continuous integration. The technology used is the open-source CruiseControl tool which builds the embedded software every time a change is made to the source code CM repository. This modern approach to building software ensures that the team is always delivering working code, so it only made sense to integrate Klocwork’s static analysis with this paradigm. Using Klocwork Insight’s feature-rich command line interface enabled complete automation of the static analysis process on every code check-in. Using this approach, hundreds of analyses were performed without expending any human effort.

The process steps are as follows:

  1. Developer commits (checks in) new source code.
  2. Build server detects the modification and triggers a build.
  3. Build server builds the software.
  4. Build server executes any defined unit tests.
  5. Build server performs a Klocwork static analysis.
  6. Build server aggregates all build results into an email.
  7. Developers receive email notification about the status of the build and any discovered defects less than 10 minutes after their code check-in.

Summary of Klocwork Benefits

Impressive analysis capabilities

The APL C&C software team was impressed by Klocwork Insight’s defect detection capabilities and extremely low false positive rate.

Since the software is an embedded real-time system, defects such as array bounds violations, use of uninitialized data, NULL pointer references, and thread synchronization errors are all examples of potentially critical issues that were found by the tool. Many of these issues are very difficult and challenging to manifest and isolate during testing, and could potentially lead to unexpected behavior and software failure in the field.

Of note, the tool also found potential security vulnerabilities where the software wasn’t validating integer use, a vulnerability type that could allow malicious users to access the system’s communication protocol to cause system failure or inject malicious code. While medical devices aren’t high targets for hackers, the APL team takes the security and robustness of its software extremely seriously, so all potential situations like this are removed.

Automated Build-Level Reporting

With hundreds of analyses run by the tool and a high-pace of development, the ability to monitor key metrics each and every build was a high priority. APL used the rich, customizable reporting capabilities of Klocwork Insight to track these key metrics each and every build:

  • Project summary and dashboard that shows key metrics and fix activity.
  • List of top defects and their severity to prioritize investigation based on criticality.
  • Source code size and complexity to monitor key components whose complexity can impact run-time performance.

Lessons Learned From Using Klocwork

The experience with Klocwork led to many positive benefits for the project as well as some lessons learned that will be applied to future projects.

  • Start performing analyses right away. Klocwork Insight analysis was introduced approximately one month after development had started. As a result, the team had a larger than desired defect list which had to be reviewed after the fact which is a less productive fix process and can adversely impact productivity if developers are forced to debug any side-effect problems that are caused.
  • Don’t underestimate the tool’s defect detection ability. There was more than one case of an identified defect that was analyzed and thought to be a non-issue, when in fact a true defect existed.
  • Integrate into a continuous integration system. During faced-paced development, it is difficult for users to always remember to run an analysis. By integrating Klocwork Insight with the CruiseControl continuous integration system, analyses were performed many times a day without users having to take action.