MANASWINI is Associate Vice President, ADAS and Autonomous Vehicle Group at KPIT
In the February 2017 edition of Auto Tech Review, we talked about the importance of system engineering in autonomous drive. The system engineering rigor brings discipline to the overall processes and ensures that the requirements and safety concepts are in place.
Today, in the journey towards automated driving, verification and validation coverage is one of the most keenly discussed topics. With numerous reported incidents of malfunctions during autonomous drives, there are concerns about whether the industry is ready to take self-driving vehicles mainstream. Auto manufacturers must first focus on mitigating the cause of the failures, which occur mainly due to the absence or lack of verification and validation coverage.
The verification and validation cycle, also known as the ‘V-cycle’, is an integral part of system engineering. It covers the following — the depth and scale of test cases at all levels including unit, functional, system and integration testing; testing the behaviour of the autonomous features against dynamic environment scenarios in the laboratory before they get incorporated into the vehicle; test coverage to depth at each component level; and covering the test cases against safety cases to identify malfunctions. (1) describes the testing involved in autonomous features in ‘V-cycle’ format.
COVERAGE OF VERIFICATION AND VALIDATION
The coverage of verification and validation includes the following major aspects, as outlined earlier — Independent verification – test coverage in unit testing, functional testing and system testing; Coverage for dynamic environment – bench, vehicle, test track, road; Levels – component, sub-system, system, vehicle; and “Illity” coverage – safety, reliability, recoverability, controllability, maintainability.
A single view about the phases of the vehicle life cycle and the role of verification and validation therein is provided in (2).
HOW VERIFICATION AND VALIDATION COVERAGE IS COMPROMISED TODAY
It is pertinent to consider each level of verification, validation and safety coverage. At present, the depth of coverage is missing for system-level testing in the bench, as the focus is mostly on establishing the coverage through road testing. While road testing may ensure the durability of the system, it may not include testing of feature behaviour against dynamic scenarios, as it is difficult to create them. The autonomous feature of the vehicle is critical from a safety point of view. It needs test coverage for the dynamic environment scenarios as well as a methodology to conduct scenario-driven, simulation-based testing in the bench.
The second aspect is the lack of coverage in verification. There is a tendency to consider validation and verification as the same. In reality though, there is a thin line of difference between the two, in terms of requirement coverage. Verification covers functional requirements to a great extent, whereas validation establishes coverage for non-functional requirements. In order to assure coverage in both verification and validation, the system demands organised requirement coverage. Today, coverage is limited to functional requirements but the need for test coverage would subsequently drive the need for requirement coverage.
As per the industry standards, the percentage of effort required for verification and validation coverage of safety critical systems ranges between 45 to 55 %, depending on the criticality levels. This includes both bench and field validation. Since significant effort and attention goes behind testing in test track and road mileage coverage, bench validation tends to get compromised. This becomes one of the biggest causes of failures in assuring coverage.
SIMULATION BASED VALIDATION ASSURES GOOD COVERAGE
Performance, safety and reliability of the autonomous features need to be validated in bench. One of the ways to achieve higher coverage is through validation of feature performance, ensuring reliability and safety against critical scenarios and utilising use cases through simulation. Scenarios or use cases are major components in assuring higher coverage in validation.
The probability of validation coverage for critical use cases and scenarios in hundreds of thousands of mileages testing is extremely low. As per NHTSA (National Highway Traffic Safety Administration) requirements, more than a million miles of coverage has been mandated as a proof of validation for feature performance. But this mileage coverage may not assure coverage of all the critical use cases and scenarios. It may potentially put an autonomous system in danger due to lack of coverage of all the performance, reliability and safety related test coverage during mileage validation.
This makes simulation-based validation extremely important. Thousands of use cases and scenarios are created and modelled through simulation environment. This creates a virtual environment, in which the performance of the autonomous features can be validated in terms of safety, reliability, controllability, and response time. Millions of test cases get generated out of the library of use cases and tested in the virtual simulation environment. A closed loop test bench on the other end assures the closed loop validation of the features.
Coverage of verification and validation is the only way to assure performance of autonomous vehicles on road. The coverage needs to be assured through both bench and road validation. A simulation-based validation methodology results in a higher level of test coverage, thereby reducing the road testing cost to more than 40 %, along with increasing the reliability and robustness of the system.