Compiler validation is simply highly controlled, repeatable and reproducible testing of a compiler using a recognized set of test programs commonly known as a validation suite. The purpose of such testing is to provide a reliable indication of how well a compiler complies with the standard for the language that it implements.
Typically compiler validation is sought in connection with the development of safety-?critical, security-?critical or mission-?critical systems. Until recently, compiler validation has been recommended but rarely formally required for the development of safety-?related software complying with safety standards such as IEC 61508 Part 3, ISO 26262 Part 8 Section 11, and/or comparable provisions in similar industry-?specific standards. Most validations to date have been for large-?scale system developments. This is beginning to change as a result of technical advances in microcontroller design.
For example, various microcontroller manufacturers now offer safety-?rated microcontrollers. An increasingly popular form is the dual-?core lockstep microcontroller for which regulatory bodies are beginning to issue safety certifications covering the hardware. These microcontrollers offer very high safe failure fractions through their physical and logical design. One weakness, however, is that both cores run the same object-?code image. For software to match the safe failure fraction achieved by the hardware, it is now becoming more important for the critical parts of development tool chains to be proved fit for purpose.
To date most compilers used in embedded system development have been accepted for use based on their history of reliable operation. Developments in hardware safety are causing a shift towards the use of formally validated compilers where previously validation would not have been required. Clients can now expect regulatory bodies to require compiler validation more and more often for critical system developments. It is expected that this will be seen initially in projects based on new safety-?rated microcontroller designs.
Most compiler suppliers make extensive use of recognized test suites, such as SuperTest™, in their product development. This however, can really only establish that a compiler is qualified for a of limited range of critical uses. It does not establish that a compiler is fit for purpose in any specific use. For C compilers used in (safety-?)critical projects it is important to understand why this is.
The C language standard makes clear that an implementation is a specific configuration of hardware platform and compiler software used under particular compiler invocation options. However thoroughly a compiler supplier may test his compiler, it is likely that:
- The testing will have been done on-host (using a simulator or emulator rather than real target hardware) because on-?target testing may not be flexible enough or can take too much time,
- Testing has not been done with exactly the same compiler options that are used in a specific embedded development project.
Consequently the compiler developer's testing is not actually testing the same implementation that the embedded developer will be using. This is not just an issue of definition: in practice, a small change in the compiler options can have a huge impact on the generated object code. Compiler validation services fill this gap by performing tests under the implementation options used for a particular development project and as far as possible on real target hardware.
