> In my social circles, many people have read James Scott's Seeing Like a State, which is subtitled How Certain Schemes to Improve the Human World Have Failed. A key concept from the book is "legibility", what a state can see, and how this distorts what states do. One could easily write a highly analogous book, Seeing like a Tech Company about what's illegible to companies that scale up, at least as companies are run today. A simple example of this is that, in many video games, including ones made by game studios that are part of a $3T company, it's easy to get someone suspended or banned by having a bunch of people report the account for bad behavior. What's legible to the game company is the rate of reports and what's not legible is the player's actual behavior (it could be legible, but the company chooses not to have enough people or skilled enough people examine actual behavior); and many people have reported similar bannings with social media companies. When it comes to things like anti-fraud systems, what's legible to the company tends to be fairly illegible to humans, even humans working on the anti-fraud systems themselves.

I think legibility is one aspect of it, but I think another could very well be empathy. If you want to support a user who has a problem, then you have to want to help them and follow through. If you want to make a customer whole after they lost money on your platform, then you want to pay them back and follow through. If you want to stop your platform to be a tool used to disseminate hate, then you want that outcome and move heaven and earth to make it happen. You have to want to see the problems and you have to want to follow through and fix them.