How We Institutionalized Incompetence
charleshughsmith.blogspot.com
charleshughsmith.blogspot.com
Our space program is finally running at a more reasonable cost thanks to some private competition. What services are costing more now outside of healthcare? Most things are getting commoditized/cheaper. What enormous investments in infrastructure education etc? The problem right now is a lack of investment into fixing bridges- not that the money we're pouring in is wasted. And - the new laguardia terminal seems pretty great. Isn't that an example of an infra success? Are all the highly paid academics clueless and delusional? In the past few years we got CRISPR, an explosion in AI, the Higgs boson, gravitational waves, etc etc etc. All from academics. If every decision is made to maximize self interest - what about the near eradication of polio and Guinea worm disease by the gates and Carter foundations?
Personally, I'm not that compelled by the "everything sucks because you know it just feels that way" arguments. You can make someone feel whatever way you want by picking your examples properly. Isnt America doing so great now? Spacex and CRISPR and a booming tech sector so on? And this is pretty clearly just meant to get you into the mood the author things will make you most likely to buy his book at the end.
Infrastructure. Our (USA) costs are COMPLETETLY OUTRAGIOUS compared to the global standard. We're talking multiples.
Things like the gateway project in NJ/NYC cost 4x what they should for a crappier outcome.
NY / SF, even germany for a bit with Berlin's airport. I look at what they did in the past (huge railroad bridges in 100 days) and even now with private money (Tesla in germany is going up fast and sports stadiums go up like weeds despite their scale) the contrasts are remarkable.
Space Launch System is $20B with not a single flight. SpaceX comes along, get's no $1B/year base, get's paid a fraction of the "big players" - and has done 80+ F9 launches.
Also - success academies in NY seems to have some pretty amazing results relative to local schools. I realize some confounding factors, but if I was a parent in NY - that would be the choice if I lived in an area they served.
You might be conflating Falcon 9 and Falcon Heavy. Falcon Heavy is a more applicable comparison to SLS. Falcon Heavy had one launch in '19 and two in '20.
When it comes to infrastructure, I think there are a few confounding factors. 1) Our level of risk is lower. We no longer find it an acceptable risk to lose workers at the rates that were acceptable 100 years ago and 2) the systems are inherently more complex. Even so, we tend to oversimplify them when planning [1]
> The answer, I believe, has to do with American incuriosity. Incuriosity is not merely ignorance. Ignorance is a universal trait, people just differ in what they are ignorant about. But Americans are unique in not caring to learn from other countries even when those countries do things better.
I think there's a lot of truth in that; look at how almost all the discussions about healthcare ignore the successful systems in other countries. It goes beyond that, though. We shouldn't have to look at other countries to consider the possibility that we could be doing things better. But whenever it's suggested that we could make things better, there seems to be a lot of hostility to the idea and push back against it.
[1] https://pedestrianobservations.com/2019/03/03/why-american-c... [2] https://data.oecd.org/eduresource/education-spending.htm
Is the one thing that seems to be enough to shut every discussion down for a certain political group.
This just isn't correct. Many majority non-white school districts are very well funded -- cities like Chicago or Washington or LA. It's simply not an issue of lack of funds.
Simply pumping more money into poorly performing school districts isn't a solution, but we continue to do so. Some of that money might be better spent on social services to help ensure stable households and early education for the kids in those areas. Or higher minimum wages to help ensure the parents don't need to work 12 hours/day to make ends meet. Etc.
And neither side of the political aisle can come to a consensus on what to change, so here we are. Le Sigh.
Vouchers might allow a few exceptional students to leave but strands the average students. And those exceptional students still need to find transport to the school across town. It also assumes a better school actually exists.
Society would be better off ensuring equal opportunity at all ages, stable households, good healthcare (including food) instead of blowing up the system to the advantage of a small number of people.
This is not an isolated incident, just forgotten. Well funded districts are often not spending on every child.
Apparently they forecast future prison populations by...3rd grade standardized test scores :(
From my observations of Canada, the reason is that it "creates jobs". It it's the classic "not-invented-here" syndrome. Noone wants to borrow the solutions of another country, or even a province or city within Canada!
Always finding some rationale to do it their own way, or do more local (inferior) studies to tick a box. Then hire a bunch of inexperienced engineers to build something that they have never done, and get x10 expensive, and 10x worse results.
All under a flag that it will create local jobs.
Imagine that's how tech would work? We want to be build a complex micro-service system, that we have never built, and we are only going to hire talent from within the distance of 10 km, to "create jobs". How would that work out?
It has been law in many states starting in the 1960's. The original intention was to prevent too much competition which could skew incentives.
* From the perspective of an average consumer. It's a brilliant system if you're in a position to profit off it. And it has resulted in the best health care in the world (for those who can afford it).
Is '[too much] competition' not supposedly the bedrock of our Wonderful System of Market Capitalism?
Infrastructure buildout in AZ is done in an incredibly efficient and cost effective manner.
Dallas is similar.
Utah’s buildout for the Winter Olympics was amazing and inexpensive.
Cali/NY/MA? Not so much.
I looked up Utah’s Olympics build. It cost $2.1 billion, and was 24% over budget. I don’t really know whether that’s efficient or not?
It's great that our universities are capable of turning out transformative technologies like CRISPR, but it's arguably slowing down, and it's happening at a time when our basic assumptions about what education opportunities should be available to our children. And make no mistake, they're narrowing.
Space travel may retrospectively look in a thousand years like an absolute necessity. But that will be from a perspective of inequality here on earth having exploded. And fewer people will be involved than our collectivity would like.
That is not an argument AGAINST these things. It may be a valid trade-off that we all accept. Or it may be a power grab by some department or group. It is tough to know generally speaking.
That is true in all areas where costs are rising wildly - medicine, education, cost of building real estate in pre-COVID bay area, cost of infrastructure. Whether you think the trade-offs are justified will depend on your mood affiliation as much as anything else.
So, what makes your examples better than the author's?
And you can also be deliberately avoidant of the things you don't want to think about. Yes, the space programs are doing okay, but the city streets are decaying and filled with crime. Yes, we know about the Higgs Boson, but we also have no savings, and an opioid epidemic.
> If every decision is made to maximize self interest - what about the near eradication of polio and Guinea worm disease by the gates and Carter foundations?
Good things can come from self-interested acts, nobody denies that, but you're thinking about this from within the paradigm of liberalism and so you're missing the critique, like someone who can't see the forest for the trees. In fact, that adage has never been more apt, because the issue with liberalism and with self-interest in general is that it boils down to a cult of the individual.
Within the decaying liberal system, the individual is above all else. Only the experiences of individuals matter. "Communities" don't exist, except as categories; and the categories themselves only exist as ways of aggregating individuals for marketing purposes. We are all atomized, all recluses even when we are being social.
> In the past few years we got CRISPR, an explosion in AI, the Higgs boson, gravitational waves, etc etc etc. All from academics.
Each of those academics is motivated by many factors, some of them selfless, some of them pure curiosity, but ultimately in the case of every successful participant there is the selfish interest in being the person to publish the paper, getting that credit, getting a bit of career advancement, maybe even a bit of fame. Individuals.
The result here is that the academics who do get some notoriety are those who are working on sexy projects, like the ones you named. CRISPR, very cool; funded with a lot of tax money; never likely to do a goddamned thing for 99.999% of the population who fronted the money for it, nor for their children, but hey, maybe 100 years from now we live in some gene-edited hellscape. Yay.
Although there's been an uptick in recent years, violent crime is way down compared to the 70's. As for nonviolent crime, I'd like to see numbers.
When I worked south of market I'd see drunk art school students walking down seedy alleys at 2am. Past drug dealers, homeless men, and crazy people. Nothing ever happened to them. But leave a pack on the passenger seat of your car at two in the afternoon.
At least in the Bay Area I've noticed that a lot of property both private and public have been fixed up over the last 30 years. Used to drive through Oakland in the 90's and most of the houses were run down and now they aren't.
some of the dysfunction is just baffling. the rent for the apartment I live in increases faster than my salary. this is in a city where the population has been decreasing YoY for at least a decade and in a neighborhood where I can see two large apartment complexes under construction out my window. how is this possible?
Housing and Higher education are obvious standouts. Relative to other countries networking and cellphone service are absurdly overpriced, but so is tax prep, highway construction, and other government adjacent industries.
This is a known effect in economics: https://en.wikipedia.org/wiki/Baumol%27s_cost_disease
"... the rise of salaries in jobs that have experienced no or low increase of labor productivity, in response to rising salaries in other jobs that have experienced higher labor productivity growth"
FAFSA is basically a perfect price discrimination system for colleges: why wouldn't they attempt to extract the maximum amount possible from each student?
An additional factor to consider is that higher education has increased the number of roles in their org-chart where Baumol's original proposition applies: the number of administrators in higher education just keeps rising. I'm not super familiar with the economics of orchestras, but I would be very surprised if the San Francisco Symphony is staffed with 4x the number of administrators that it was in 1978, even though the number of chairs in the orchestra remains the same.
Essentially the most expensive schools are priced as feeders into the elite collages of the North East. Exclusivity is a benefit when students are competing with each other students for placement into top schools. Other parts of the country have a much stronger focus on institutions with a religious rather than purely academic focus.
Which gets back to my original point. You get inefficiency when that’s what’s rewarded. It’s not that schools can’t operate cheaply, rather people actively want a different model.
Also covered in economics! https://en.wikipedia.org/wiki/Signalling_%28economics%29
Being able to get into and complete several years of college signals to a potential employer that you are a better candidate. The more prestigious the institution, more difficult to get in etc, the better for the candidates that manage to. That increases demand, but the (roughly) fixed supply at a college results in increased prices and other things that reinforce the college (eg better sports and more successful sports teams).
How about the price of food. Only Ramen and maybe fresh produce have stayed low. Also any kind of internet connectivity, housing.
The problem with the position that "everything is fine" is that it doesn't resonate at all with the observable reality experienced by 99% of people (outside of HN). Maybe it's hard for you to believe but for most people, the premise of the article could not be more obvious. It's like trying to convince homeless people who are sleeping outside the gates of a mansion that they are actually wealthy because they own a sleeping bag...
I can't really agree with this. If you look closely, ridiculous budgets for infrastructure disappearing into black holes are rampant in the US. A great example is the southern California High Speed Rail project. As other commenters have also stated, US infrastructure development costs are out of proportion with the rest of the developing world and seem to be absurdly expensive simply because they can be [1].
[1] https://www.brookings.edu/wp-content/uploads/2019/07/2019-07...
A quick google shows China's existing high speed rail system at 23,600 mi, in comparison to the total of 800 miles planned for CA.
I don't disagree that it was a moon shot, but it was not a moon shot for reasons of technical difficulty or novelty, but for the reason that it is now so systemically difficult to competently implement any large scale infrastructure in America, and California particularly. I won't comment on what the reasons for that might be.
My interpretation is that this kind of decay in capability is that the author is driving at.
Do you really want to use them as an argument for "Just make rail 4Head?"
Ultimately the irony here is that the article itself is a pretty good example of self interest - its a bunch of the type of arguments they go after academics for making, and then uses it to try and sell their book.
Well said. This needs to be carved in granite somewhere
All services when the buyer is the government.
On the other side it's worse, health care, education loan, minimum wage not a living wage, affordable housing homeless etc. Things are are important for not everyone not just a few at top.
1. The lack of a real "competitor" in a geopolitical sense for the West. I don't think it is a coincidence that the US arguably had the most competence and scientific development circa ±1940 - 1990, when it had a direct competitor to measure itself against. Incompetence against the Nazis or Soviets got you killed, whereas today, nothing really happens (immediately.) The ancient Greeks (and subsequently Nietzsche) recognized the usefulness of competition toward keeping yourself sharp and called it agon. Ironically the rise of China can be seen as a good thing from this perspective.
https://en.wikipedia.org/wiki/Agon
http://www.nietzschecircle.com/v3/2017/03/17/book-review-ago...
2. The loss of a civilization-wide teleological narrative. For the past ±1,500 years, that's been Christianity, in various forms. These days, there is no real "purpose" to the West, beyond some vague notions of individualism, choice and comfort. Institutions and standards become more and more difficult to maintain as a sense of apathy and the lack of a perceived future grows. Younger people especially seem unconvinced that there is anything waiting for them in 50 years.
That might work for some people? I don't know?
But I do know that the Charleston church shooter sat down with a bunch of little old ladies for the duration of a bible study engaging in, what I can only assume was, a calm in-person discussion. Then he blew their heads off.
Point is that, for some reason, the internet has a much stronger hold over people than do in person connections. People deeply believe they are more connected to people in their virtual spaces than they are to their neighbors or even physical friends. That sense of connection is real to those people. To some of those people, it's real enough to kill for.
are there some people for whom virtual spaces are more real than their neighbors and IRL friends? sure, but most people don't seem to be like that. as far as I can tell, most people use social media to interact with an extended network of IRL friends/family. some people do try to get tons of followers that they don't know on instagram, but I think even this is best understood as jockeying for status among IRL peers.
note: IRL is a somewhat antiquated term this days, but I use it as a shorthand for "relationships that originate in in-person interactions". so if I had a friend A who moved away and befriended B there, but we all play video games together, B is still an IRL friend to me.
I expect that there will be a strong reaction on this forum to the word 'socialist', but the communities you describe definitely still exist.
I agree that they do exist. But they don't seem to be mainstream as they once were, and I'd very much like to change that.
But also, historically, these sorts of communities have been targeted by governments. Police literally went to war with unions to break up community power for coal miners. While communism is different than socialism, there's a huge public smear campaign trying to couple those two together. (And, in fact, historically there were major pushes to blacklist folks who were connected to communism.)
I try to look at the long arc of human history as basically the story of how we grow the size of the "in-group" and shrink the size of the "out-group". It isn't easy, but we have made significant progress over the centuries.
As someone who grew up in a small, rural Christian community, this is a nice sentiment, but far from true in my experience. People instead become highly punitive and disparaging towards anyone who does not think and behave exactly like them. Whatever community exists is built around a hateful exclusion of “the other.”
That being said, there was definitely a “fear of the other “ with people being extremely suspicious of the few handful of outsiders who moved in and didn’t have any connection like relatives, existing connections, or children going to school there.
I get that the World, and Europe in particular, got badly burnt by Utopias in the 20th century. But still seems like something is needed - especially given the looming geopolitical changes.
It seems true to me that most individuals (myself included) are not up to the task of inventing a reason to exist. You can moan about how the stories are made up til Kingdom come, the point of it is practicing the belief. Otherwise you get what we're talking about in this thread, billions of people growing up believing nothing, having no rational reason to believe they are valuable, no reason to think the future warrants their contribution.
this is a rather ethnocentric view of religion based on what is clearly a limited historical understanding evidently informed mostly by judeo-christianity. just to pick one counter-example: the ancient greeks sought virtue and purpose from philosophical inquiry, not from their religious practices, which were much more transactional.
The modern Unitarian Universalist (UU) church is very welcoming to people of all faiths, religions, and creeds. You do not need to be atheist to join, nor do you need to believe in anything. Most congregations are very welcoming of new members.
Four US Presidents, eleven Nobel prize winners, and Sir Tim Berners-Lee are Unitarians. There are many Unitarians out there, though they do not tend to proselytize much, which is maybe why you haven't heard of them.
As always, finding the right congregation is very important. The UU church has a good site here to help:
With the current Covid-19 crisis, local churches near you may be closed or only doing virtual services. Please check with the minister.
People were united by a monopoly on education and political power. It's too bad that we're losing that ubiquitous societal fabric, but in the face of the alternative, seriously, good riddance.
If spiritualism is your thing though, do drop by at a Unitarian Church close to you. It's an interesting experience. Not for me, but still warmly recommended.
Maybe that is why the devil and the hell were introduced at some point ;-)
You're essentially describing (secular) Humanism, e.g.
https://en.wikipedia.org/wiki/Secular_humanism
(Just found the author of the study/studies: Sosis[3]
[1]: https://www.amazon.com/Righteous-Mind-Divided-Politics-Relig...
[2]: A commune being defined as a group of people not sharing kinship deciding to live and work together.
[3]: https://anthropology.uconn.edu/wp-content/uploads/sites/944/...
Though honestly, the difference between the US can hardly be called drastically significant. Both are doing horid.
The real question is whether this trend can continue forward. My bet is yes for the foreseeable future, but no power system lasts forever. So while we're seeing the gradual (seeming) decline of the West, we'll probably be repeating the pattern in a few decades for East Asia.
https://www.medrxiv.org/content/10.1101/2020.09.21.20198796v...
2) You have almost no idea how incompetent or wasteful most private organizations are, unless they are so bad that they go bankrupt and dissolve.
Your atom bomb mention is interesting considering Oppenheimer's Hinduism reference.
"Now I am become Death, the destroyer of worlds."
https://www.amazon.com/True-Only-Heaven-Progress-Critics/dp/...
"In the old days it was better", "golden age" - no it was not better. There were incompetent people in 1960's, there were scummy and incompetent clerks in 1500's. History lessons are about great people.
Maybe competition model is not that good, because instead of fighting for survival some people could start working on inventing their own things. Just like now billionaires are playing with rockets and having results. I would say there can be a lot of creativity when people are left alone and can build things, without having to think about threat of war. Going to the moon was necessity caused by knife on the throat, not some noble achievement because people wanted go to the moon. That is why no one went to the moon since then again.
Forgive me as I try to articulate what I'm thinking.
Being the sole super power somehow turned out to be bad thing. We (the USA) needs to an opponent, a competitor to better define and understand ourselves.
During the Cold War, aspirational preaching about human rights and democracy and capitalism was a way for us to differentiate ourselves from our opponents. A bit like Scott Galloway's thesis about Apple using privacy as brand laddering technique to differentiate itself from Google, Samsung, etc. Privacy is not central to Apple. But it makes for great ad copy.
I'm not saying USSR, China, whoever were (are) better or worse than the USA. I have no opinion and I really don't care. (Well, I do have an opinion. People in glass houses...)
What I am saying is the USA needs the idea of an opponent. So that domestically it (we) can set aside our differences to meet the external threat. And to have some kind of narrative to pitch why aligning with the USA is the better choice.
I don't think the USA is unique this way. Us vs them is base human psychology.
I do hope that The Powers That Be, on both sides, under stand the realpolitik is just kabuki and it matters don't degenerate into another Cold War. We don't need another arm's race.
While we did have to fight the Cold War, which led to much progress on a technological front, the US also had the advantage of reduced competition from previous world powers (UK, Germany, France, Japan, etc) which had been destroyed by WWII.
The Communists gave us focus. The end state of WWII provided the economic ability to put that focus to good use.
Or something like that. But, I do agree that the lack of a single unifying enemy is a problem when it comes to motivating society as a unit. It feels like we're more likely to be destroyed (as a nation) from internal factors than external, and that's not good.
Expect the law to be followed shouldn't count a "high expectations". Sigh.
This has never worked! And widespread online inventory systems date back, what, 10 years? This is a weird place to begin an essay about how good the good old days were.
He is, of course, totally wrong - back then they couldn't even make a reliable car, much less a reliable global supply chain management system. Computers are orders of magnitude more complicated than anything mankind had built as of 1970 so really it's more amazing that they work at all than that there are bugs.
The real difference may be one of intelligibility - in the 50's your car wasn't very reliable, but when it did break, you could understand and quite possibly fix what had gone wrong. Now, the world is way too complicated and nobody can possibly understand more than a small fraction of how anything works.
The "everything is getting worse" narrative ignores substantial improvements in life expectancy, literacy, health, yes, and even homelessness. Homelessness is down over the past decade [0]. So yes, many of these things actually are getting better.
[0] https://en.wikipedia.org/wiki/Homelessness_in_the_United_Sta...
To reparaphrase a popular quote: those who forget about the past are doomed to complain about the present.
I would also argue that in America specifically there ARE numerous counter-examples, and I'd vouch to say that HN is probably a statistically significant outlier of such. A lot of startups we discuss or celebrate (and even lots of startups that we bash;) work on very different principles and incentives than those listed here.
Do public infrastructures and services cost more in inflation adjusted dollars and achieve less? That's a pretty significant claim which needs serious evidence.
The contention that we are in some uniquely incompetent time is also dubious. American history and all history for that matter is loaded with stories of the rise and fall of fakes, cranks and cons.
Daniel Sickles, The origin of the Chase Manhattan company and John Brinkley just off the top of my head.
https://en.m.wikipedia.org/wiki/Daniel_Sickles https://en.m.wikipedia.org/wiki/JPMorgan_Chase https://en.m.wikipedia.org/wiki/John_R._Brinkley
Yeah, it would have been nicer if claims like this within the article had links to supporting discussions of some kind. On this topic, at least, I've seen numerous discussions on HN, so I googled a little bit and found some articles that look familiar:
https://fee.org/articles/the-unbearable-truth-about-infrastr...
https://news.ycombinator.com/item?id=22122460
EDIT: Replaced the second link with an HN discussion about it from earlier this year. Couldn't find one for the other article.
This rings home as I was recently talking to someone at an engineering firm that often bids for government contracts. And they were explaining that you build your timeline back from the deadline in the proposal even if you know it isn't feasible. A lot of companies will then bid under-cost and hope that there was a flaw in the design that requires adjustments. Then since they are already on the site with a bunch of workers they will charge extortionary process for the modification to make money on the whole project. So that $10M dollar bid jumps to $15M because they needed to reroute a gas line before starting the building (not a real example, but representative)
Of course a lot of the problem is that the city has strong processes and policies to avoid corruption, however the end result is that it is very difficult to hire the people you know do good work, instead you are stuck with the lowest bidder. Of course there is a process for banning companies however it is long, complicated and very bureaucratic, so even if you could get the ban in place it is easier for any project manager not to bother.
I guess it all comes down to lack of trust, and this is why we can't have nice things.
I'm not sure that the result is axiomatically over budget for major projects. That said, I'm less worried about budget than I am worried about who benefits (e.g. I'm generally ok with transit going over because it provides such a valuable benefit for the city.)
I'm open to the idea that we spent more than we should have (even if it was under budget), but I'd want to see more data about the project and where the costs were unreasonably high.
The saddest part is honest companies want nothing to do with these huge gov contracts. The only way to navigate the minefield of regulations is to lie which honest people dont want to do. This results in nefarious companies that specialize in ripping of the government.
And just to get my mandated downvotes. Trump has been beating the drum of their being too many regulations since he got in office. I'm guessing HN won't want to hear that though.
You're right, but he is also the type of person that make the corruption-prevention regulations look like a good idea from my point of view.
But he was elected and I'm not from the US so maybe my point of view isn't the important one here.
The people paying these companies outrageous sums are the ones taking campaign donations from those companies. The people that decide that the high speed rail which needs to stop at every town along the route (and that the route needs to include all their towns) are -- again -- the politicians.
At every turn politicians stick their noses into the workings of infrastructure projects. Quality and cost finish well behind kickbacks to the politician and largesse to their supporters.
Then, when seeking bids, you could factor in the scores of those involved in the organizations bidding - giving it a weight or requiring a threshold (for average or median or P95 or something) among the participants.
But I think this is on the right track. As a simple approach you can have final-cost ratio. "Unknown" bidders would use the average score across all projects (or all first-time projects?) and known-good bidders would be able to use their own score (which may be close to 1). Known-bad bidders may have 1.5-2 multipliers which "corrects" for their underbid.
Of course this too simple. You want to:
- Account for time.
- Account for how easy they were to work with.
- You need to factor in unexpected changes. (which do have a cost, but how do you determine the true cost?)
- Using the average for unknowns may make it too difficult for new companies to enter. I way I see it you want some new entries to keep your pool in check. (Although the majority of projects, and quite possibly the largest projects should be going to known-good companies).
I think this is part of the problem. There are so many factors that it is difficult to rigidly define them, and if they aren't rigidly defined than you have chances for corruption.
The TL;DR is that unless you just let a human make the decision it is a very hard problem, and right now we don't think that humans are trustworthy enough.
A new company can (should!) hire some people with solid local experience, so can have a score (we still need defaulting rules for individuals, of course).
Individuals risk reputational damage if people elsewhere in the project behave badly, which should increase cooperation with controls. Similarly, well managed companies will find it easier to hire, while poorly managed companies will find it more difficult.
Companies that have a lot of poorly scored individuals because of poor historical practice will be extra motivated to bring on people with a history of working on well managed projects, which would help knowledge flow about how to keep projects well managed.
For physical infrastructure that's often the case but with software you get 2-3x cost overruns simply because the project is effectively re-done 3 times. Because often what the buyer is asking for is simply not what he needs. Of course, bidding laws makes it illegal to contact the bid writer and if you try to anticipate anything not explicitly written, you'll end up with a more expensive proposition than your competitor.
The IBM Phenix payroll system is a prime example of that, but there are many more.
The author assumes these statements are obvious to the reader, but they are definitely not obvious to me.
Are things getting worse? The examples he cites, with the possible exception of infrastructure spending, seem to me about the way they have always been.
Outside of that though I'm not so sure. I wouldn't like to go back to the seventies.
I was just reflecting on The Organization Man last night and received from the get-go a nice little gem from a man giving a Yale commencement address in 1908:
"Here merit is the sole test. Birth is nothing...Here only a natural order of nobility is recognized, and its motto, without a coat of arms or boat of heraldry, is 'Intelligence and integrity'." (p16 2002 UofPenn Press)
And then the author, Walter H. Whyte, remarks:
"But the very industrial revolution which this highly serviceable ethic begot in time began to confound it...by the 1880's the corporation had already shown the eventual bureaucratic direction it was going to take...the big organization became a standing taunt to this dream of individual success. Quite obvious to anyone who worked in a big organization, those who survived best were not necessarily the fittest but, in more cases than not, those who by birth and personal connections had the breaks." (pp16-17 ibid)
The argument being, America escaped the moral inefficacies of the old world, yet it slowly crept back in once it became more profitable - and above all convenient - to be ambitious on a career ladder than ambitious before the nakedness of the sun and stars. Which means it is common to have to lie and conspire in order to get ahead - and how many people are willing to for a nicer car to drive around the neighborhood!
The Visible Hand by Alfred D. Chandler, Jr. is a great read corroborating the growth of agglomerated industry, and Joseph Schumpeter's Capitalism, Socialism, and Democracy also has some good passages relating to this phenomenon of the homestead getting saturated.
But I'm an eternal optimist and see this as the price of progress :)
Another explanation: those who hyperoptimise for efficiency get what they pay for.
This is buried toward the end of the article but I think it's a big statement with massive consequences for many practical issues.
We live in a society where a heck of a lot of value is created, but the value accrues to people who don't seem to be the ones creating it.
If you look at economics, the "who creates value" question has been neatly sidestepped. If you can get someone to pay you, GDP goes up by that amount. There's not a whole lot of quibbling about whether the value paid was correct, although the GDP accounting system has had a number of changes throughout the years (vast simplification). At present if you just don't do something illegal, the dollar you earned is as good as the dollar anyone else earned.
This system has the fault that it doesn't seem to connect with ordinary ideas about who contributed. If whoever ends up with the trophy is the person who deserves the trophy, what is the trophy for? There's a kind of nihilism about what work is for if the purpose of climbing the greasy pole is to reach the top of the greasy pole. Why should I bother to write good code if the thing that really has value is to sit in meetings and talk myself up to people who have no idea what good code actually is? Why should I not simply create a compliance tax for my colleagues, and become the troll who extracts payment?
I can start with things most people agree with: good code should have unit tests and documentation. And we can move on and add code reviews, from at least 2 other devs, before any new merge is allowed. And we can't have the same two guys that are your buddies, let's rotate it a bit to make sure there's no corruption. And make sure the rotation system is documented. And signed. By an independent auditor. With an industry accreditation. From an organization that is overseen by government committee. A government committee that confers with other government committees to ensure best practice.
All those people need to be paid. And they are valuable, you can't deny that documentation is important. So all these people are creating value and need some of the pie.
The cost per km of the first London Underground lines, which were dug largely by hand from the road downwards and built with a lot of manual labour was about 1/10th of the cost of building a similar tunnel in the 21st century with automated machinary and far fewer "expensive" people.
How do you dig by hand? Simple, get a load of people with no training and give them a spade and get on with it. How do you do that with a $100M tunnel boring machine? Send people on expensive training courses, pay a load of cash in specialist repairs and maintenance and when it breaks down or you get any other kind of problem off the beaten track, just stop work until you can sort it out or pay someone else even more money to get you out of your hole.
Same with software. How easy is it to keep bank records in a physical ledger? Really easy and very quick. Introduce some software and you now have a multi-million dollar complicated recording machine with loads of semi-skilled operatives using it, breaking it (accidentally), and only very slow and expensive teams can investigate and fix. We have accepted that sometimes we need to ask 10 times to get an address changed because "computer says no".
Of course, technology brings you benefits that didn't exist before like centralising data, like making some hard physical processes easier to automate and to backup data that could otherwise be lost in flood or fire but are these worth the complexity and cost? In many cases that's a hard "no"!
In my world, compliance is about offsetting risks, sure there are plenty of risks involving incompetence, but that is only one of the issues you need to address. Putting the spotlight on incompetence is going to cause you to miss other significant issues in your organization.
Ultimately, lists and checking boxes are a long term solution to the classic "How do I do xyz?" "I dunno ask Bob he usually does that".
I think what happens is that everybody needs to have a job and source of income and people try to do the best for themselves. But, of course, they don't get more intelligent.
As the technology progresses, more and more jobs are high skill or abstract tasks and both require intelligence.
What this means is decreasing availability of intelligent people for the jobs and necessary incompetence of people who perform them (or the positions would not be filled).
This was the red flag line to me.
I haven't heard anyone ever propose "let's do more of what's failed". What I do hear is people saying "we got that wrong, and here's the minor tweak we need to fix it".
You wrote some code:
for (int i = 0; i < len; ++i)
copy (src[i], dst[i]);
for (int i = 0; i <= len; ++i)
copy (src[i], dst[i]);
This isn't "let's do more of what's failed spectacularly". This is "let's make what we're trying actually work". Sure, someone could come along and suggest that we use Algol or brainfuck to solve the issue instead, and that's not a priori insane. But a clear minded reading of the situation - we've got a lot of stuff right, but made some mistakes - would favor trying to fix what we've done wrong, not stop doing what we're trying to do.
The cynicism of this analysis mostly ruined the rest of the piece for me.
We are closing in to the bottom where the a rather radical solution has obvious and major benefits that it will finally be taken. And then the change will be swift.
Note that in previous iterations the solution has often been a large scale military conflict.
Edit: spelling.
> the late David Graeber
He died on Sept 2 2020, aged 59, unexpectedly while vacationing. RIP
Jared Diamond called this the "Anna Karenina Principle", from the opening line of Tolstoy's novel of that same name:
All happy families are alike; each unhappy family is unhappy in its own way.
The idea being that a number of things all need to be correct to be successful; if any one of those things is missing, things will fail.
So often when I talk to programmers they excuse their code with "Its just a toy implementation", "Our customers dont need all the features", "its fast enough", "we can re-implement it later". Rule one if you avoid failure: Aspire to be good.
Why are we fetishizing mediocracy? Why are so few people aiming to implement things on the beading edge? Why are we trying to avoid anything that is hard?
It's cheaper to buy hardware than acquire talent to write good software. By several orders of magnitudes.
It's not. It just cost a lot.
What a joke.
Everything he complains about are services, or labor-heavy products. All prices are relative. As manufactured goods become cheaper, services and labor become more expensive.
That is exactly what I thought was going on. Most big companies definitely give off that vibe and the longer you work there, the more you get promoted, the more dirt you uncover, the more you get paid, and the least value you create...
What are the odds that the author of this article came to the exact same (very specific) conclusion as I did? Clearly it's a matter of time before this is going to blow wide open.
I think humanity's actually doing quite well these days. Not perfect, of course, so let's keep working on making things better. But I'd rather be alive now than pretty much any time in the past.
But I understand what you mean though. Violence used to be the way of life and inseparable from it whereas nowadays some parts of the world are doing relatively much better. But it might be short term victory.
People will get away with do less for higher pay so long as the system allows. That is not a violation of ethics. It’s an absence of ethics in the first place.
Now I'm just left equally as irritated as I was when I started reading.
This is Moloch https://www.youtube.com/watch?v=aDaOgu2CQtI
Yeah you're free to make an alternative, you might succeed, but if you get too big then the FAANG's will sue you right out of business.
Oh I forgot, or the FAANG's PR team will use the media to accuse you of being a home for trolls, hate, etc.