Big Tech Sees Like a State (2020)
thediff.co
thediff.co
I work for one of those Big Tech companies, and internally, it is anything but legible. Very consequential decisions are made by a small team that happens to have responsibility for the codebase in question, oftentimes very far down in the org. Culturally at least there is a respect for data and a sincere effort to do the right thing, but when your decisions impact 3 billion people there is no way a human can fully evaluate them. And ironically, a recurring problem for the last 20 years is that the CEO has no idea what's going on within the company. Most records about what's being worked on, why, how far along they are, who's involved, etc. are firmly in the realm of metis, tribal knowledge that's passed down in 1:1s and watercooler conversations.
The more I encounter other powerful government bureaucracies (like building codes, local government, zoning, legislation), the more I suspect they work the same way. Somebody at the time it was formalized decided "this is how it should work" on the basis of the best information available at the time, and then once it's formalized, it's virtually impossible to change.
I'm interested in crypto not for the "get rich quick" or even the "currency" aspect, but for the prospect of replacing corporations and executives. Crypto's big innovation is in creating a shared reality everybody accepts without a centralized authority to enforce that reality. This is exactly the same efficiency advantage that having a single executive gives. Could you use blockchains to essentially create an "artificial executive", a plan of record that everybody gets behind that doesn't fall prey to human fallibilities?
Yes but it will either be a rigid bureaucracy, or subject to manipulation by interested parties, or an inscrutable AI, or some combination of those.
It's not like we have good results when trying to write flawless code, that never needs big fixes, and that is so future proof as to never need updates.
I like to think of AI not as "artificial" intelligence but as "aggregate" intelligence. Computers are really good at aggregating the opinions of millions of individual humans. That's what search algorithms like PageRank do, it's what recommendation systems do, it's what voting systems do, it's what markets do, it's what statistical quant trading systems that try to predict the movements of markets do. For these purposes, a computer seems to be reliably better than humans. So why not use computers for what they're good at, aggregating the behavior of millions of individual humans, while still letting humans give their own input.
PageRank was great, until SEO was invented by someone who realised there was an opportunity if they gamed it for free advertising, then it became a Red Queen race between SEO teams and Google.
Markets are mostly pretty good, but the USA has a special law about onion futures because someone realised they could game the free market: https://en.wikipedia.org/wiki/Onion_Futures_Act
Etc.
It doesn't have to be perfect, it just has to be better than what we have now.
It's the "more easily" that I'd dispute. A computer can be examined in detail, its weaknesses probed ruthlessly and systematically until a vulnerability is found.
Any one human will have traits in common with others, and differences from those others. For example, your own question:
> Haven't you ever told an exec what they wanted to hear, or refrained from telling a manager something that might make you or your project look bad?
this would be a dangerous similarity, one that can be manipulated — but not universally, not all of us, and not all the time: there's people who react very negatively to such sycophancy or boasting.
> The whole lobbying apparatus is about altering the perceptions of lawmakers so that you get more favorable laws. People usually operate on fewer pieces of data than computers, and so one falsehood can alter their decision-making more than it would a computers.
Yup, absolutely. But if the software developer misses something that an outsider can learn about — this can be something that just didn't even exist at the time the developer made their choices, or just a bug they didn't spot — it can be abused in the same way.
When it comes to accidents, the USA almost started WW3 because someone forgot to account for the fact that the moon rises in the same part of the sky that a Soviet strike group would come from, and also doesn't respond to IFF pings. The Soviet union made their own mistake with sun glare.
When it comes to things outsiders learn and the original developer didn't think of, other than all bugs and exploits, consider also things like this: https://arxiv.org/pdf/1707.07397
It depends. Knowing some government organizations in Germany this is quite spot on especially when "CEO" level positions are set by political policies based on regular elections and not based on normal carrer-based government servants. However there are exceptions but they tend to be rarer the bigger an organization becomes.
> Somebody at the time it was formalized decided "this is how it should work" on the basis of the best information available at the time, and then once it's formalized, it's virtually impossible to change.
This is indeed also accurate but depends on precedent and the number of relevant dependent entites. If you have precendet of changing a set of rules (somewhat) regularly then dependants can get used to adapting it, e.g., tarifs, tax code and parts of building code.
This however gets a lot harder when more people depend on it because you will then have to make sure (information of/about) your rule changes get propagated appropriately, e.g., updates to driving safety legislation that affect everyone with a car.
Taken to the extreme, you have the social security administration, which has a manual which essentially prescribes a procedure for any of thousands of scenarios.
Zoning or code compliance requires some level of professional judgement, but the standards and procedures are driven by legislation or policy.
Pushing that stuff to the end user just pushes the complexity of the business rules to the end user.
Because there’s usually multiple political parties or interests and a level of public accountability, governments are good & efficient at delivering what they are mandated to deliver. (What they are asked to deliver is usually the “problem”.) When was the last time that benefits checks weren’t issued?
Commercial entities with similar qualities are also very bureaucratic (ie banks, airlines). Tech companies tend to be pretty bad at internal controls, and engage in a lot of hold my beer.
I have no regrets on robbing the corporate media and user hostile advertising companies of their revenues, while keeping my browsing experience clean and thwarting most trackers.
I go a step further and run GrapheneOS instead of normal Android, so not even Google themselves can track me. Google Play Services is running in a sandbox that makes it behave.
What were dealing with today is much worse than the old banner ads, these things track your every move online and even track the GPS on your phone.
> Lives at a particular location, and has an exact address.
A person in the UK may live at any natural number of places, including zero. The NHS has specific provisions for providing healthcare to the homeless. At the other end of the spectrum, landowners with property in different parliamentary constituencies were entitled to vote in all of those constituencies until the passage of the Representation of the People Act 1948.
> Has a specific name.
In the UK, a person has no specific legal name. A person may use any name they wish, including multiple names, without the need for a deed poll or any other legal instrument.
> Earns money in a currency which the government understands, and pays taxes in it.
The UK has a thriving (although regulated) cryptocurrency sector. Tax policy is a little lax, and it's very easy for someone to live in the UK unbanked and untaxed, which is probably a contributing factor to so many irregular immigrants moving specifically to the UK, rather than remaining in the first 'safe country'.
> Was born on a particular date, and can thus be called up for jury duty or conscription.
The UK is good at tracking when and where people were born, and to whom, but someone can slip through the cracks of the system if they put a little effort in to avoid service to the state.
> Doesn’t steal other people’s goods, trespass on their land, injure them, or kill them. (Except as a result of the aforementioned conscription situation.)
Due to reductions in policing budgets, shoplifting is currently rife and, when the value is low, not addressed, but this is a temporary state of affairs. Trespass is generally a civil matter, and is only criminal in specific situations, such as when the trespass is into a home.
> Speaks a language intelligible to the government employees who are responsible for checking all of the above.
The UK's only official language is Welsh, and Welsh only has that status in Wales. Some legislation is written in Norman French.
On the other hand, I grew up in Australia, where that list very much applies, because Australia is a nation that learnt civilisation from prison guards. It's not a bad list - the UK is just a very peculiar place.
Remember though, how often in the history of states, the temporary cannily usurps permanence. It's hard to tell what's a brief deviation from the mean, rather than an early glimpse into the new normal.
I have no idea what I'm talking about, either in terms of understanding society nor the UK specifically, but your phrasing tickled my paranoia about that phrase a sort of famous last words for civilizations, lol.
The British state is administered on the principle of maximum effect for least effort and least cost. The Empire was administered in the same way, although, as Lord North discovered, that principle has limits. The surveillance state is very visible because its primary purpose is deterrence, not solving crime at scale. The population isn't made legible; it's made to think that it's legible.
A few years ago, there was a lot of media coverage about 'super recognisers'[0]. This is because CCTV footage has to be reviewed by humans. There's a reality TV series that shows how much human work has to go into turning the information from the surveillance state into actual outcomes on the ground.[1] It's not an everyday thing.
The much simpler explanation is that both governments and big companies are large groups of people. You need to make simplifying assumptions to make large groups of people work together at scale.
That's true of any system at scale. Just consider how people give their servers unique names at small scale and numbers at big scale. Its just how scaling works.
Whether that is a good or bad thing is an entirely different question.
The article is using the vocabulary of sociology, where "legibility" in reference to the state is a well-known concept. The article specifically references Seeing like a state and James Scott's other writings, but he's not the only one to think in terms of such concepts. Foucault's idea of the Panopticon is very similar - it's about the state maintaining control through the citizenry always knowing that they're watched - while Habermas's idea of a "lifeworld" ("metis" in this article) which is "colonized by expert systems" is the same phenomena.
A group of people want to run their enterprise under rule of law and this requires a government. this legal entity we call a corporation. it may be a for profit endeavor. it might be a public good physical area monopoly(a town, county, state, country). It might be a for profit physical area monopoly(these were some of the first licensed corporations, but you don't see it much now days). anyway there are many types of corporations.
Usually to achieve the desired legal protection a license is acquired from a parent corporation that lets you form the government needed to run the corporation. it's corporations all the way down. At the top level they are peers(nominally) and must guarantee their protection the hard way(armed force). Organized crime is an example of an unlicensed cooperation, there is still a government, but the endeavor is run unlicensed, in rebellion to whatever larger entity claims jurisdiction over the area.
This lens is increasingly the norm in the PoliSci and Management spaces, with Institutionalism and research into administrative capacity being extremely vogue rn.
Economists like Daron Acemoglu, Yasheng Huang, Rakesh Khurana, etc have dug into this.
Most corporations have owners, most governments don’t.
One person grows their operation organically from nothing, they then are the owner and get to be the king/dictator.
A group pools their resources to finance the organization, as such they are the owners to each a share according to their amount.
A group decides that all members of the group are the owners without regard to how much they financed it. this is a democracy.
The point being every group of people has an owner.
If policy is designed locally or at least allowed to be adapted to the local context, it could be more effective but with the possibility of inconsistency; Otherwise the policy is implemented flawlessly but with the cost of effectiveness because of lacking local context.
Any attempt of increasing legibility means a policy being implemented consistently across different contexts so everyone can assume the same thing regardless of the context, and, as expected, not able to incorporate the complexity of local social orders.
Big Tech Sees Like a State - https://news.ycombinator.com/item?id=25009130 - Nov 2020 (14 comments)