When Waymo went live without a driver in Phoenix, it wasn't a test, it was the real deal.
When they run simulations, often they're running simulations of specific intersections or stretches of road. Using simulation dramatically condenses development timelines, but simulation is still anchored to physical testing on real roads.
Testing in multiple cities is necessary to avoid overfitting for a specific region.
When I compare Waymo and Cruise, I use every publicly available piece of data I can get my hands on. There actually wasn't much to know about Cruise's progress before the end of 2016, when the disengagement reports came out.
Waymo reported 5000 miles between unplanned disengagements on average. Cruise reported 180 miles of testing per disengagement out of much less overall miles. None of the other 2 dozen or so companies registered to test with the California DMV even mattered. Going 180 miles on busy downtown streets between disengagements was hard to believe coming from a non-entity like Cruise, but in spring of this year Cruise released a string of videos demoing their driving in SF, and sure enough, they were doing it. At that point it was time to regard Cruise as a contender. Mentioning them as a contender on forums like this one were generally met with derision.
On the 28th of this month Cruise did it's first driving demos for the press, and so we finally had people to independently verify the vehicle's performance. Out of a dozen articles written by various journalists, one reported an unplanned disengagement when one of Cruise's Bolt's was confused by a cherry picker. Most journalists commented that the ride was 'herky-jerky', and that Cruise's vehicles would often stop, sometimes suddenly and think for a long time before making a decision. One thing is clear, Waymo has put more effort into polish and ride feel. Cruise contends that ride feel is not a priority for them at the moment.
Cruise makes a big deal about focusing their testing efforts in downtown SF, and notes they encounter 40x the number of interesting situations per mile as in the more suburban Phoenix. They intend to launch in NYC next year, and to roll out 1000s of Bolts, with the aim of scaling up their testing to 1 million miles/month in prep for an initial commercial deployment sometime in 2019.
Waymo supposedly intends to begin their initial pilot commercial deployment early next year. Very exciting stuff, it will mark a major milestone, they'll be the first company to go from zero to one with fully autonomous vehicles.
Waymo doesn't have a vehicle manufacturing partnership, and until this is addressed it's a constraint on their ability to scale their deployment. Their current strategy of retrofitting Pacificas is not an efficient or long term solution.
One of the first things Krafcik did in his freshly minted role as Waymo CEO was blow off a promising manufacturing partnership with Ford. This left Ford in the lurch, and ultimately led to the replacement of Mark Fields with Jim Hackett. Then the Waymo team was given a series of massive bonuses that were promised them in exchange for meeting certain benchmarks. Then most of Waymo's key people left. Levandowski went to Uber. He was sharply criticized by Chris Urmson, who himself left to start Aurora.ai with Sterling Anderson (their website went live today). Brian Salesky, also a Google vet, left to start Argo.ai with Peter Rander (they had their first delivery of testing vehicles today). The only main guy left at Waymo is Dmitri Dolgov. Waymo's engineers are more loyal to Dolgov than Krafcik. There are reports of leadership conflicts between Krafcik and Dolgov.
Kyle Vogt and Justin Kahn, before founding Cruise started and scaled Justin.tv, which rapidly scaled and became Twitch. They sold it to Amazon and used their wealth to self-fund Cruise. Their initial plan was to develop a retrofit kit for the Audi A8, and they spent a few years working on that, growing their team to 90 people before being acquired by GM and showing an impressive ability to solve hard problems that many other established and well-capitalized development programs are struggling with. Vogt is legendary for locking himself in a room with a problem and not coming out until he has a solution, sometimes for days. For what it's worth, he's also an asshole with a giant ego, and still very young. He participated in the DARPA Grand challenge in 2004 as a 19 year old MIT junior, he's been into robotics since he was a kid.