California’s Department of Motor Vehicles (DMV) yesterday released its annual Disengagement Report, which breaks down the number of times a human driver takes over control of an autonomous driving system being tested on the state’s public roads. The DMV released reports for 29 companies for the period from December 2019 to November 2020.
Waymo (29,944.69) and Cruise (28,520.34) are the leaders in the clubhouse when it comes to the number of miles driven between disengagements. China’s AutoX (20,367) is a distant third. This is the sixth straight year in which Waymo increased its number of miles driven between disengagements.
Cruise, which is owned by General Motors, drove more than 770,000 miles last year to overtake Waymo, which has historically owned the top spot. Waymo drove just under 629,000 miles in 2020 after logging more than 1.4 million miles in California in 2019. Both companies drove more miles in 2019, but many self-driving car operations were shut down during part of 2020 due to the COVID-19 pandemic.
Zoox, which was acquired by Amazon in 2020 for $1.2 billion, drove 102,521 miles and had a disengagement every 1,627.32 miles. Aurora, which acquired Uber ATG in December 2020, drove just 12,208 miles in California and had a disengagement every 329.93 miles.
The chart below compiles data from each company’s performance in 2020 and their respective miles per disengagement in 2019. It is ordered highest to lowest based on the number of miles driven between disengagements in 2020. We have more information about the 2019 data here.
2020 Calif. Self-Driving Car Disengagements
|Company||2020 Miles Driven||2020 Disengagements||2020 Miles/Disengagement||2019 Miles/Disengagement|
Argo AI, Atlas Robotics, DeepRoute, and QCraft reported numbers for a full year for the first time. Interestingly, 34 companies licensed to test self-driving cars on California’s public roads reported no testing took place. Tesla is one of those companies. Under California’s rules, there is no need to report data on systems below SAE Level 3. Tesla’s “Full Self-Driving” system is considered a SAE Level 2 driver assist system and not autonomous driving.
Seven companies (Bosch, Changan, Faraday Future, SAIC, Samsung, Torc Robotics, TuSimple) didn’t even renew their license. The license for Kaizr was suspended as of Feb. 9, 2021 as the company failed to submit a disengagement report despite conducting on-road testing.
Most autonomous vehicle companies, including Cruise and Waymo, have said disengagement reports are misleading for a variety of reasons. One major reason is the numbers are self-reported, so accuracy is always a question. But without a standard way to evaluate a self-driving car’s performance, this is the best information available. And for the first time, the California DMV shared who initiated the disengagement – the safety driver or the system – and where the disengagement occurred.
“For all of 2020, we averaged only one reportable disengagement per 28,520 miles,” Cruise wrote in a blog on Tuesday. “For the second half of 2020, we improved to more than 60,000 miles between reportable disengagement. And in the final three months of the year, we had zero reportable disengagements, a 10x improvement over the same period a year before.”
Waymo wrote on Twitter that while it appreciates what the DMV is trying to do with the report, “the metrics provide limited value in assessing the capabilities of the Waymo Driver, or in distinguishing its performance from other A/V companies.
“Further, our testing program spans different vehicles, software platforms, testing environments and objectives, so the disengagement rate in any given year simply can’t be predictive of real world performance.”
AutoX, which recently launched a robotaxi service for the public in China, drove only 40,734 miles. It experienced a disengagement every 20,637 miles, which is nearly double its performance last year.