Imagine that you’re the driver of a four-door family sedan approaching a stop sign. When you reach the stop sign, you notice a bicyclist trying to cross the road. Through eye contact, facial expression and body language cues, the bicyclist negotiates their right of way with you. As a result, you decide to let the bicyclist cross the road first, before you proceed to cautiously enter the intersection.
In the autonomous driving world today, there would be no way to “tag” or categorize such an event, said Cognata CEO Danny Atsmon. Current methods allow you to visually identify the bicyclist, but training systems to recognize and understand complex negotiations on the road remain a challenge for the $10.3 trillion autonomous driving industry.
In fact, autonomous driving represents “the single hardest computing problem the world has ever encountered,” as NVIDIA CEO Jensen Huang admitted when he unveiled some of the world’s most powerful graphics processors during the GTC 2018 keynote in San Jose, California.
Bridging the Real and the Virtual
“The world drives 10 trillion miles per year,” Huang said in a pointed presentation — but Atsmon pointed out that self-driving cars only covered three million miles of roads last year. For self-driving vehicles to drive better, they must learn more, and that’s fundamentally the largest challenge faced by the industry. To train an autonomous driving system to have the competence of a human driver, computers would need to drive roughly 11 billion miles, Atsmon told us.
It’s the single hardest computing problem the world has ever encountered.
That figure is calculated based on the 1.09 fatalities per 100 million miles driven in 2015. “So, to say a machine could have as safe a performance as a human being with 95 percent of confidence, you would need to validate for 11 billion miles,” said Atsmon.
Aside from the time needed to reach that goal, there’s also the expense to consider. Right now, the cost per mile for operating an autonomous car is in the hundreds of dollars — accounting for engineering time, data collection and tagging, insurance costs, and the time of a driver to sit in the cockpit of a car. Multiply that by the 11-billion-mile benchmark, and the massive expensive associated with training autonomous cars becomes clear.
Validation is key, and recent accidents involving autonomous vehicles show that incomplete data tests and training scenarios can prove fatal. In one less extreme example, a self-driving shuttle in Las Vegas was navigating at around 0.6 miles per hour, but it crashed into a truck (Jeff Zurschmeide, a freelancer contributor to Digital Trends, was there when it happened). No one was injured, but the puzzling scenario happened because the truck was pulling forward, then backing up as it tried to park. The cause for the crash, according to Atsmon, is that the shuttle wasn’t validated for this type of situation, and it didn’t know what to do — so it proceeded forward slowly and crashed.
Better Simulation for Deeper Learning
The industry’s current solution to bridging the 11-billion-mile gap for autonomous systems to reach human driving competency is to develop simulations to allow cars to learn faster by combining deep learning with a virtual environment.
“Simulation is the path to billions of miles,” Huang said at GTC. Late last year, Alphabet-owned Waymo unveiled Carcraft, its approach to learning by simulation.
Cognata is using the latest advancements in graphics and sensor hardware to create more life-like and realistic models of the world for autonomous cars to learn from. For the computing brains of a self-driving car, it’s like entering a video game modeled on the real world, and that could lead to more realistic driving scenarios to test and validate car driving data. The company has recently mapped out select cities, like San Francisco, using data from GIS — high definition cameras and sophisticated computer algorithms that runs over satellite and street view imagery, resulting in a photo-realistic scene.
Simulation is the path to billions of miles.
To further improve simulations, Nvidia, and some of its partners, are using data from the sensors of autonomous vehicles to build higher definition maps. When autonomous vehicles hit the road, these machines will not only rely on the data that is available through training, but also contribute to data collection by sharing the data that it has captured from its LIDAR, IR, radar, and camera arrays.
When this newly captured data is combined through deep learning with existing low-quality data sets, it will make streets and roads look more photo-realistic. Cognata claims that its algorithms can process the data in a way to bring out details in shadows and highlights, much like an HDR photo from your smartphone’s camera, to create a high-quality scene.
While simulation is an excellent tool, Atsmon noted it has its own flaws. It’s too simple, and for autonomous driving to be realistic, it must learn from edge cases. Cognata claims that it only takes a few clicks to program in an edge case to validate autonomous vehicles for more unusual driving scenarios. Companies building autonomous vehicles will have to be diligent in their search for edge cases that can trick self-driving cars, and creative in crafting solutions for them.
When Self-Driving Fails
Safety is so paramount to autonomous vehicles that Nvidia considers it the single most important thing for the industry. When things fail, fatalities can and do occur, as was recently proven when an autonomous Uber struck and killed a pedestrian in Arizona.
“I can assure you that [Uber is] equally crushed at what happened.”
When questioned in a press meeting about the Uber crash — Uber is a partner of Nvidia — Huang deferred to the ride-sharing company for comments, saying that “we should give Uber a chance to understand what has happened and to explain what has happened.”
“I can assure you that [Uber is] equally crushed at what happened,” Huang added.
Because Nvidia develops an end-to-end solution for autonomous driving, different partners — from Uber to Toyota and Mercedes Benz — may utilize all or some parts of the system. “There are some 370 companies around the world who are using our technologies in some way.” At the show, Nvidia also announced Orin, the next-generation computer of its DRIVE platform.
Humans as a Backup
While self-driving cars are getting smarter over time, Huang still believes that there should always be a human backup, even in instances where a car is designed without a driver seat. To achieve this, Nvidia showcased its Holodeck during this year’s GTC keynote, allowing a remote driver to control a physical car in real-time through virtual reality.
“It’s teleportation,” Huang said, highlighting that this is possible through Nvidia’s early investments in virtual reality.
During the demo, Tim, the driver, was located in a remote location. When he put on a pair of virtual reality glasses, he will felt like he’s in a physical car, enabling him to feel the car, and see the car’s controls and instrument panel. From this remote location and with the aid of his VR headset, he could take control of an autonomous vehicle, allowing him to drive the vehicle and park it.
It’s like what the military has been doing for a while — allowing drone operators to fly unmanned drones from remote location. But in Nvidia’s case, with the power of VR, the driver will feel like he is physically present in the cockpit. The company believes that simulation powered by its GPUs will eventually make autonomous cars nigh-infallible but, until them, the Holodeck can help humans watch over self-driving fleets.
