How to train your self-driving car

This article is an installment of Future Explored, a weekly guide to world-changing technology. You can get stories like this one straight to your inbox every Thursday morning by subscribing here.

When you’re driving, your senses collect information about your environment and send it to your brain for processing. Your brain then tells your body what it needs to do to get you to your destination safely (ease up on the gas, turn the wheel left, etc.).

Self-driving cars need their own “senses” and “brains,” and different developers have their own ideas about what those should be and how they should work together. 

Here’s how three of the most promising companies are approaching the challenge of building cars that can drive themselves — and potentially save lives in the process.

Waymo: Sensors — sensors everywhere

Waymo started as Google’s Self-Driving Project, but it is now its own company, nestled under the Alphabet umbrella. 

In 2020, it became the first company in the world to begin operating a paid robotaxi service without human backup drivers behind the wheel. That service is now available in Phoenix and San Francisco, and will soon expand to include parts of LA and Austin.

The sensors: Normal cameras “see” objects by detecting the visible light reflected by them, and because they’re cheap and small, most autonomous car developers rely heavily on them. Waymo is no exception — its cars have 29 cameras that can record objects up to 500 meters away in any direction.

Waymo’s vehicles are also equipped with LiDAR. These sensors fire laser pulses and then record the time it takes the pulses to return after bouncing off objects (this is how a lot of police speed detectors work). This information can be used to create a 3D map of the sensor’s surroundings, day or night.

Waymo also has six radar sensors on its vehicles, which use radio waves to help track objects around the car. They can reveal an object’s distance from the car, as well as its speed and direction, even in bad weather conditions, such as rain or fog.

Just in case all of that isn’t enough information, Waymo also creates its own static, high definition maps of every location where it plans to deploy its cars. 

The brain: Like most developers, Waymo relies heavily on computer simulations of driving to train its autonomous car “brain,” which it calls the Waymo Driver. 

In these simulations, Waymo can challenge its AI with countless driving scenarios, see how it reacts, and then modify the system so that the car does what Waymo wants it to do out in the real world. Because simulations don’t have to move at the pace of the “real world,” the AI can gain 100 year’s worth of driving experience in a single day.

Waymo will validate the training in a vehicle on a closed course before deploying it on public roads. New scenarios encountered on the road in the real world can then be recreated in simulations, and the cycle repeats. 

Tesla: Power of the fleet

Unlike Waymo’s vehicles, which are all owned and operated by the company, Teslas are sold to private owners and mostly driven like regular cars. However, they do come equipped with “Autopilot,” a system that can automate certain driving tasks under certain conditions.

Drivers must be ready to take control of the vehicle at any time, though, which means keeping their hands on the wheel, eyes on the road, and staying alert, even when the car is driving itself. CEO Elon Musk has been promising for years that full self-driving is just on the horizon, but it’s unclear when it’ll actually arrive.

The sensors: Tesla has taken a much more stripped-down approach to sensors than Waymo — its Autopilot AI relies solely on input from nine cameras (the company used to feed the AI data from radar and ultrasonic sensors, but those have been removed).

“Humans drive with eyes & biological neural nets, so [it] makes sense that cameras & silicon neural nets are [the] only way to achieve [a] generalized solution to self-driving,” Musk tweeted in 2021.

Tesla vehicles have never included LiDAR, and based on how vehemently Musk has criticized the tech, they likely never will: “Lidar is a fool’s errand. Anyone relying on lidar is doomed. Doomed! [They are] expensive sensors that are unnecessary … It’s ridiculous, you’ll see.”

The brain: The Waymo Driver might have the benefit of more input data than Tesla’s Autopilot, but Autopilot has a major advantage when it comes to training data: numbers.

While Waymo’s fleet is in the hundreds, there are millions of Teslas on the road, and when Autopilot isn’t engaged, it still operates in “Shadow Mode,” essentially imagining what it would do if it was in control of the vehicle.

If its choices aren’t the same as the driver’s, information about the scenario, such as camera data and the car’s speed, is sent to Tesla. 

This can reveal limitations of the system — an inability to see signs that are obscured in a certain way, for example — that can be corrected with software updates, or it can provide new scenarios that can be replicated in simulation for training purposes.

As is the case with Waymo, the step after simulation is driving on closed courses followed by public roads.

Wayve: The intelligent car

Rather than manufacturing its own autonomous cars, London-based startup Wayve is developing an autonomous driving system — AI Driver — that it wants to sell to other companies for use in their vehicles. 

The Microsoft-backed startup isn’t as far along in development as Tesla or Waymo — it still uses safety drivers in its vehicles — but by the time it’s done, it expects to have a system that’s far cheaper than its competitors’.

The sensors: Like Tesla, Wayve believes that cameras alone might reasonably be enough to enable fully autonomous driving. However, it’s developing its system to also include radar to address the limitations of cameras, such as problems seeing in inclement weather.

It’s not clear exactly how many sensors Wayve uses, but it describes the suite as “lean” and “optimized for affordability, safety, and scale.” It’s designed to work on any type of electric vehicle and without the need for high definition maps.

The brain: This is what really sets Wayve apart from Waymo or Tesla — instead of training its system to take specific actions in specific scenarios, it’s teaching its AI the general concept of driving. 

This generalized approach would mean that Wayve’s AI Driver can learn to drive in one city and then navigate another, or train on a passenger car and then control a van, just like a human driver.

“It’s interpreting its environment, understanding the context, and making the safest decisions to drive through there, and the beauty of that is that it can learn things that are more complex than you can hand engineer,” CEO Alex Kendall told Bill Gates during a demo.

The company relies heavily on a concept called “reinforcement learning” for training, which involves letting an AI learn through trial and error and “rewarding” it when it does something right.

This training can take place in simulations or directly in the real world — in 2018, Wayve demonstrated how it could teach an AI to keep a real car in its own lane by rewarding it for going as long as possible without a human safety driver having to take the wheel. 

The AI relied on input from just a few cameras, and the process took just 15-20 minutes. 

The bottom line

Autonomous cars have the potential to make our streets safer while also freeing us from the stress and tedium of the daily commute, but only if the tech is safe, affordable, and robust. It’s not yet clear whether Waymo, Tesla, or Wayve — or all of them, or none of them — will meet those requirements.

As of 2022, Waymo’s LiDAR package had an estimated cost of $35,000 to $100,000, which is far too expensive to translate to personally owned vehicles. Tesla’s reliance on visible light cameras, meanwhile, could mean Autopilot will never be safe enough to work under all conditions, even with a mountain of training data. Wayve’s AI-focused approach is so new, it’s been dubbed “Autonomy 2.0,” and because it’s so new, it hasn’t had a chance to prove itself yet.

Still, these are just three of many developers each taking their own approach to training autonomous cars, and while none has managed to develop a fully autonomous car, yet, we are seeing a surge in vehicles with some self-driving features on the road.

This suggests we’re on the right path to making autonomous cars a major mode of transportation in the future — the journey might just take longer than we had hoped.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].