Airbnb Eval-driven development: Lessons from evaluating GenAI at scale

Sidetracked by this blog being on Medium, I feel like I jumped back in time 5 years.

Why should we care what Airbnb has to say? I get they're a YC darling, a poster child for making money, but how much of a role does software play in this? Do propery owners and travelers really care that much about the app? The Platonic ideal is you spend 5 minutes making sure an opening exists in a location you need to be at, at a price you can afford, with pictures and reviews good enough to make you believe it's tolerable to sleep there between doing the things outside of the house you actually care about doing while traveling. Hopefully that 5 minutes once or twice a year is all the time you spend in the app ever. The vast bulk of the value is provided by the properties and the locations, which were already there well before LLMs ever existed, and software can't build houses or cities.

Same with Uber, Netflix, Facebook. I don't doubt they've done some great engineering in figuring out how to deal with scale and data analytics, but the value to the users isn't the software. It's the cars, the content catalog, the network of other people. Waymo is an actual tech company. AWS is an actual tech company. Uber is a taxi provider. Netflix is a media distribution company. Facebook is a virtualized third space. I don't mean to denigrate these companies. Malls provide value to both customers and stores. Everyone benefits in terms of discoverability and ease of use having all the stores and all the customers in the same place at the same time. But malls don't write blog posts telling store operators how to run a retail business. Malls are property owners, a fundamentally different business. A quasi-legal hotel operator is a fundamentally different business from a business that actually sells software.