Need For Machine Learning Based Simulator Approach To ADAS Validation

Guest Commentary Machine Learning Based Simulator Approach ADAS Validation Sasken Technologies
Need For Machine Learning Based Simulator Approach To ADAS Validation

RAM RAMASESHAN is Senior VP and Global Head, Automotive and Industrial Business Units at Sasken Technologies Ltd



The global ADAS market is growing at a rapid pace and is expected to reach $ 60 bn by 2020 at a CAGR of 23 %, according to a report by Allied Market Research [1] with Europe and North America being the key geographies driving this growth. In order to make way for this rapid growth, investments are being made by various constituents of the value chain such as silicon vendors, OEMs, and sensor manufacturers along others to capture a pie of this market. This is resulting in innovations in chip design, artificial intelligence (AI) and deep learning, radar, and computer vision-based solutions, which are key to realising vehicle autonomy.

As ADAS features become more mainstream, one of the major challenges industry faces is achieving economies of scale so that the technology moves to mid-tier segments, where the volumes are present. Another challenge is the lack of standardisation for exchange of information between various sensors, vision-based systems, radars, et cetera, in a car’s network.

Increasing system complexity due to multiple sub-systems being integrated in autonomous cars and the need to process the output data in real-time is also becoming a challenge. As standalone ADAS features transition to autonomy, cars will be making decisions based on AI and deep learning, and would hence require lengthy and expensive validation cycles. With regulations for autonomous cars still evolving, it becomes necessary for technology providers to ensure that the systems are compliant with increasing regulatory requirements. The issue of liabilities in case of mishaps is a complex question that does not have a simple answer. This therefore requires a validation strategy that is aligned with this changing and evolving landscape.

As per a study conducted by Rand Corporation [2], autonomous vehicles have to be driven for 275 mn miles to demonstrate with 95 % confidence that their failure rate is almost 1.09 fatalities per 100 mn miles. This is a huge ask and gives a perspective of the validation task that lies ahead for the industry. This is primarily due to the fundamental departure from state flow-based design, where co-relation between input and output can be established to machine learning systems, and where the decision taken is not so apparent since it is based on probabilistic models and also on what the machine has learnt over a period of time.


Let’s take a closer look at the evolution of validation in the world of ADAS. Testing with rigged vehicles in real roads involves high cost and risk. The scenarios are also limited by the distance and terrain being test driven. Next, testing the vehicle on a designated test track involves lower cost and risk but also increases the scenario covered since the test track is designed for a variety of boundary cases that may not be encountered on normal roads.

Testing offline using pre-recorded drive data allows the various electronic systems to be tested using the data that has been collected already. This provides savings in terms of expensive drive tests and also increases the coverage. Finally, testing offline using simulated data provides the advantage of simulation of various scenarios that could not be realised in real life, yet at a fraction of the costs involved in real life testing and high coverage scenario and low risk.

A simulator based approach to ADAS validation provides flexibility to perform ADAS tests in the safety of a lab environment. Computer-based models simulate the sensor output signals close to what would be generated in a real-life driving scenario. Such an approach would require:

:: Ultra-powerful GPUs enabling photorealistic renditions of driving scenarios;

:: Advanced gaming engines enabling detailed scenario construction and rendition;

:: Advanced modelling tools like MATLAB or Simulink
enabling accurate simulation of sensor characteristics and behaviour; and

:: Powerful CPUs enabling real-time simulation of vehicle dynamics using physics-based models.

This type of solution can be used at all stages of the system’s development – Model in Loop, Software in Loop, and Hardware in Loop testing. This is especially useful in the early stages of algorithm development to iron out the issues before porting it out on to the hardware. Here are a few advantages of simulation based testing:

:: Eliminates the risk to human life and property of on-road testing, especially during the initial stages of development;

:: Enables testing of the algorithms behaviour under extreme situations. For example, testing the algorithms response in an unavoidable crash scenario;

:: Significant reduction in validation costs by eliminating the need of traditional test methods for a large part of the development cycle. For example, drive test costs – fuel, manpower, vehicle rigging, etc;

:: Reduction in costs involved for frame-by-frame manual annotation of thousands of hours of drive recording videos;

:: Reduction in costs associated with risk to human life
and damage to property;

:: Reduction in costs of building or renting specialised
test tracks;

:: Accelerates time-to-market;

:: Tests can be run in parallel to quickly accumulate drive-test miles under various conditions;

:: Eliminates the need to wait for the target environmental conditions to occur naturally. For example, to test under snowy conditions one need not wait until winter;

:: Avoid delays in testing due to clearances from regulatory authorities;

:: Real-time validation of algorithms performance against
the ground truth, combined with failure pattern analysis,
and step-by-step scenario replay helps accelerate debugging of issues;

:: Provides the only cost-effective approach for safe testing of autonomous systems;

:: Drive recording-based approaches cannot dynamically respond to control inputs from the self-driving algorithms; and

:: Specialised test tracks may not provide sufficient scenario coverage.


While one may argue that validation based on synthetic images may not be adequate, but the advances in GPU are certainly making the images photo-realistic, allowing for such simulator-based validation to advance and mature. Furthermore, simulator-based validation is going to become a necessity to aid in the transition from ADAS to autonomy as it would be practically impossible to validate all possible scenarios otherwise.