Mutation-Based Qualification of Module Verification Environments
Rahkonen, Samuli (2016)
2016
Master's Degree Programme in Information Technology
Tieto- ja sähkötekniikan tiedekunta - Faculty of Computing and Electrical Engineering
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2016-01-13
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tty-201512111824
https://urn.fi/URN:NBN:fi:tty-201512111824
Tiivistelmä
The increasing complexity of integrated circuits has put more emphasis on the reusability of both the designs and their verification environments. Verification is often based on assumptions of the quality of the created test cases rather than objective measurements. Currently used coverage metrics fail to measure how well the design functionality is verified. Mutation-based qualification is a way to tell how well the verification environment can detect (artificial) bugs in the design.
The aim of this study is to examine if an existing ASIC verification flow benefits from the use of a mutation-based qualification software, Synopsys Certitude. This is done by examining how the methodology reduces the number of flaws in the verification environments and reduces the spent time in verification. Other criteria are: Scalability (small IP blocks and large subsystems), feasibility to qualifying existing verification environments, and usability. Finally the achieved gains are compared to spent effort.
Certitude injects faults to the design and uses the verification environment to check if it correctly activates bugs, if the bugs propagated to a point where they could be detected, and finally if the bugs are detected. The test cases are SPI, CR and coprocessor IPs and subsystems.
The results suggest that Certitude can point out many flaws in the existing verification environment, but it may not fit in the existing verification flow. The compilation and simulation flow has to be done within two separate Certitude scripts and the results must be separable from each other. This proved to be difficult in the SPI case, where approx. 85 % of the bugs were not qualified. The tool is not easy to use for beginners and the analyzation of the results requires much knowledge of the design, the design code, the verification environment and the complex working principle of the tool. Certitude is more laborious than traditional coverage methods with very complex designs (CR). Analyzing an issue took about twice the time of traditional coverage methods. Because of the difficulties, the user may not find any flaws that wouldn't have been discovered with just coverage. However, high reliability designs would benefit from the extra qualification.
The tool proved to be worth of the spent effort if the verification environment could be set up to Certitude's configuration model. If the tool configuration is lacking the separation of tests, operation speed and configuration portability, its usage becomes too time consuming. Too much time was spent for configuration (30 days) in the SPI case. The CR and coprocessor cases confirmed that the same person should do both the verification environment development and qualification with Certitude.
Certitude scaled well from small modules to a larger subsystem module, because it supports grid computing platforms. The qualification run time of the coprocessor decreased from 24 hours to three hours with the grid.
Having the methodology in wider use would require quite significant mind-set change for the verification engineers. Today, tests are made with assumptions on the tests' functional coverage, even though the reality of the checkers' completeness can be something totally different. The tool should be used incrementally by checking the detection outcome every time one thinks the test is good enough.
The aim of this study is to examine if an existing ASIC verification flow benefits from the use of a mutation-based qualification software, Synopsys Certitude. This is done by examining how the methodology reduces the number of flaws in the verification environments and reduces the spent time in verification. Other criteria are: Scalability (small IP blocks and large subsystems), feasibility to qualifying existing verification environments, and usability. Finally the achieved gains are compared to spent effort.
Certitude injects faults to the design and uses the verification environment to check if it correctly activates bugs, if the bugs propagated to a point where they could be detected, and finally if the bugs are detected. The test cases are SPI, CR and coprocessor IPs and subsystems.
The results suggest that Certitude can point out many flaws in the existing verification environment, but it may not fit in the existing verification flow. The compilation and simulation flow has to be done within two separate Certitude scripts and the results must be separable from each other. This proved to be difficult in the SPI case, where approx. 85 % of the bugs were not qualified. The tool is not easy to use for beginners and the analyzation of the results requires much knowledge of the design, the design code, the verification environment and the complex working principle of the tool. Certitude is more laborious than traditional coverage methods with very complex designs (CR). Analyzing an issue took about twice the time of traditional coverage methods. Because of the difficulties, the user may not find any flaws that wouldn't have been discovered with just coverage. However, high reliability designs would benefit from the extra qualification.
The tool proved to be worth of the spent effort if the verification environment could be set up to Certitude's configuration model. If the tool configuration is lacking the separation of tests, operation speed and configuration portability, its usage becomes too time consuming. Too much time was spent for configuration (30 days) in the SPI case. The CR and coprocessor cases confirmed that the same person should do both the verification environment development and qualification with Certitude.
Certitude scaled well from small modules to a larger subsystem module, because it supports grid computing platforms. The qualification run time of the coprocessor decreased from 24 hours to three hours with the grid.
Having the methodology in wider use would require quite significant mind-set change for the verification engineers. Today, tests are made with assumptions on the tests' functional coverage, even though the reality of the checkers' completeness can be something totally different. The tool should be used incrementally by checking the detection outcome every time one thinks the test is good enough.