Justin P. Schorr, Ph.D., Principal Collision Reconstruction Engineer
Waymo internally develops, funds, conducts, publishes, and peer-reviews its own internal study which shockingly reaffirms their long and deeply held belief that their technology is safer than human drivers.
Waymo, LLC. published a March 8, 2021 paper titled Waymo Simulated Driving Behavior in Reconstructed Fatal Crashes within an Autonomous Vehicle Operating Domain. The paper was authored by six employees of Waymo, LLC. As a company, Waymo has boasted about their safety record for years and, more recently, has been developing creative ways to showcase what they believe to be dominance of their self-driving systems over us humans.
The paper is pointless. The study is designed such that achievement of the expected outcome was obvious and inevitable. Briefly, the study collected police data for 117 fatal collisions occurring over a 10-year period in Arizona. Admittedly, limited data was available to reconstruct the details of how these collisions occurred. Of these 117 collisions, only 72 were included in the data set used for analysis. Researchers identified 91 “vehicle actors” in the 72 collisions – 52 initiators and 39 responders (initiators being when the assumed actions of a human driver initiated the collision sequence and responders being the assumed actions, or inactions, of a human driver in responding to the actions of an initiator). The human driver was then “replaced” with the Waymo self-driving technology in a simulation framework and the scenario was “re-run” with the tech in control. The outcomes were recorded and voila Waymo proved their technology is better than human drivers!
I refer the reader to the actual paper (cited above) to confirm that my brief description is not an oversimplification of what was done. Moreover, the procedure I’ve described may strike some readers as being a reasonable methodology for quantifying increased or decreased safety of the self-driving system when compared to human operators. Consider first that in all 52 instances where the self-driving technology replaced the initiator, the collision was avoided. Had you replaced the human operator with a different human operator who didn’t make a mistake, the same result would have been achieved. Even more confounding, the basis for the comparison is fundamentally incorrect. Proper analysis would, at a minimum, consider failures from both the human operators and the self-driving technology such that the replacement could have been made in both directions. Here, we know that every time the self-driving technology “disengages” and the human safety driver successfully takes control this would constitute a reasonably corollary to the study and would have rounded out the analysis.
While the remaining 39 vehicle actors also lack a reasonable basis for comparison, some insight can be gleaned from these results. Results of these 39 simulations were 82% complete avoidance (32/39), 10% “reduced injury risk” (4/39), and 8% no change (3/39). If we consider the no change simulations and half of the “reduced injury risk” simulations, this represents about a 7.5x improvement over humans, AT (the very, very) BEST.
Waymo vehicles, tested on California roadways in 2020, had either a system failure or a crash once in every just over 20,000 miles and were involved in one collision every just under 63,000 miles (21 disengagements and 10 crashes in about 628,781 miles). Conversely, human drivers are involved in collisions once in close to 500,000 miles. This represents about an 8 – 25x improvement over self-driving systems. While the analysis here is basic, it is sufficient in scale to demonstrate that the technology is nowhere close. Year over year, improvements remain linear across all companies testing Autonomous Vehicles (AVs) in California and, absent eternal factors, the timeline for driverless implementation remains indefinite.
Examining the industry as a whole, California has released last year’s set of “Disengagement Reports” from the companies which are currently testing partially Autonomous Vehicles in their state. The 2020 reports are the 6th set of yearly reports to be released since testing began in 2015. Here are some highlights from my analysis of this data:
- There are over 25 different companies currently testing on California roadways. In 2020, the total miles driven by these vehicles was 1,955,195. This is a dramatic reduction from the 2,866,363 test miles driven in 2019. It should be noted that included in these milage figures are periods of time when the test driver is in complete control of the vehicle.
- In total, test vehicles were involved in 47 collisions in 2020 – 1 collision every 41,600 miles (approximately). This marks the best year over year performance (in terms of the number of collisions) measured to date. In 2019, for instance, there was a collision every 28,400 miles (approximately).
- Test vehicles suffered 3,700 disengagements in 2020 – 1 every 530 miles. Once again, this is a marked improvement over previous years where, for instance, there was 1 disengagement every 310 miles in 2019.
While these figures may seem like they indicate definite improvement in self-driving technologies (and this will undoubtedly be the story which is told in the media) – this is not necessarily the case. Estimates of the total miles driven in California by humans in 2020 are over 15% below the total miles driven in 2019. This reduction is even more dramatic when considering local roadways in urban settings – the exact setting in which the test vehicles primarily operate. As such, with reduced usage, there is a subsequent reduction in conflicts – and this is likely a major factor in what may be a superficial improvement in autonomous vehicle safety. Parallel to the reduction in congestion, there is a necessary improvement in the artificial intelligence which controls these vehicles that comes along with an additional year’s worth of trips on the same routes as this repetition works to “train” the software brain of the vehicles.
There is no question that, in terms of safety, 2020 was a banner year for California’s autonomous test vehicles. Still, when evaluating the expected improvements of the artificial intelligence software against the backdrop of a transportation landscape which was (hopefully) unique to 2020, consumers should be left with a healthy bit of skepticism as to whether this data indicates progress or the same stagnant growth we have seen since testing began 6 years ago. What’s certain is that the initially promised exponential improvements in safety have not manifested in the least. Moreover, the “events” of 2020 have introduced a new concern: in the promised utopia of connected, autonomous, shared, electric (CASE – as it is termed by the US DOT) vehicles, they who control the cars control the lockdowns.
Justin P. Schorr, Ph.D., Principal Collision Reconstruction Engineer with DJS Associates, Inc., can be reached via email at experts@forensicDJS.com or via phone at 215-659-2010.
Have A Question About This Article or Want to Contact the Expert?