lkrubner

I've written about the poor style of decision making that often happens at "flat" organizations. Repeating a point that Jo Freeman makes, without a leader or a process for decision making, flat organizations make decisions accidentally. The winners tend to be whoever is the most stubborn. See "The Accidental Democracy of Flat Organizations":

https://respectfulleadership.substack.com/p/the-accidental-d...

Exactly. For those who need to understand what this can look like, I wrote up a detailed case here:

https://respectfulleadership.substack.com/p/the-accidental-d...

This is one of the main topics that I teach entrepreneurs about.

I wrote something reasonably short here:

https://respectfulleadership.substack.com/p/to-grow-you-must...

I wrote something much longer here, but this material is mostly from other CTOs who I surveyed:

https://respectfulleadership.substack.com/p/a-survey-of-ctos...

If you are in New York City, we have 2 different speakers who will speak to 2 different aspects of this problem, at our event on February 27th.

Many graphic designers don't know how to work with the template systems of CMSs that attempt to serve multiple websites from a single app instance. Wordpress is kept alive, in part, by designers who feel it is easier to hack than any other system.

This. If Symfony or Ruby on Rails or Django or Quarkus or NextJS could replace Wordpress, then they would. I myself wrote Symphony for many years (2008 to 2014) and I was certain it could kill Wordpress, but it failed.

The combination of Wordpress's defaults, its ease of use by designers, and the ability for one server to handle hundreds of installments, means that almost nothing can replace it.

The Soviet Union could have survived if a central political power had survived, but by the late 1970s this was becoming impossible. The collapse of the USSR resembles the collapse of many states that are overtaken by oligarchs, and the oligarchs were gaining power long before the USSR collapsed, though they were called Apparatchiks and had somewhat different roles, and less freedom-of-maneuver than the true oligarchs that emerged in the 1990s. But by the late 1970s the largest industrial powers were consolidating into a single bloc that would be able to defy the Politburo, and so Gorbachev had to worry about a coup the entire time he was in charge, and he was never able to push through the reforms that he knew were needed.

I'll post here a long excerpt from the best book on the topic.

The Struggle to Save the Soviet Economy:

Mikhail Gorvachev and the Collapse of the USSR

By Chris Miller

©2016

Page 181-183

The only force that proved strong enough to break the military-industry-agriculture coalition that dominated Soviet politics was the collapse of the Communist Party - which in turn caused the dissolution of the Soviet state. Until mid-1991, the three economic lobbies were bound together by interest, ideology, and inertia. Had Gorbachev been able to divide the coalition partners, playing one interest group against the others, he might have had more success in asserting control over the Communist Party and the Soviet state. But a strategy of divide and rule proved impossible. So long as they dominated the Communist Party, and as long as the party controlled the state, these groups’ shared interests overwhelmed any tactical alliance Gorachev could have conceivably offered. In the years after the collapse of 1991, with the military divided and discredited by the failed coup, and with industry and agriculture writhing under the pain of inflation and depression, Russian president Boris Yeltsin finally managed to split the groups, co-opting much industrial support while slashing farm subsidies and cutting military funding. Even though the Soviet Union by then no longer existed, however, Yeltsin still faced several years of resistance in his attempt to break the lobbies’ stranglehold on the federal budget and on the central bank. Only after Yeltsin shelled parliament in 1993, pushing the country to the brink of civil war, was the military-industry-agriculture coalition finally destroyed.

Was the Soviet Union simply unreformable? China’s experience proved that there is nothing inherent in Marxism-Leninism, in autocratic political systems, or in centrally planned economies, that makes a transition to a market economy impossible. In the USSR, to be sure, decades of wasteful investment left the country a burdensome economic inheritance. Yet the most damaging legacy of the command economy was not economic inefficiency, but political sclerosis. The Soviet system proved unreformable not because its economic problems were insurmountable, but because it entrusted vast political power to groups that had every reason to sabotage efforts to resolve the country’s economic dilemmas. In part, this situation was the result of the USSR’s relative wealth. When Deng took power in China, for example, the country’s farmers were on the brink of starvation. No matter what Deng did, the state of China’s countryside could hardly get worse, so China faced no built-in lobby that opposed change. By contrast, the USSR was stuck in a politically induced middle-income trap: many Soviet citizens, especially among the elite, lived decent lives that were threatened by change. Whereas Chinese farmers embraced decollectivization, Soviet farmers - who had benefited from several decades’ worth of farm subsidies - found that Gorbachev’s agriculture policies offered risks as well as rewards. A similar mechanism obstructed change in Soviet manufacturing and service enterprises.

Economic efficiency was also restrained by the relative leniency of the Communist Party during the postwar period. Under Stalin, the party had few interest groups because the Soviet dictator enforced his writ through purges and mass killings. Enterprise managers dared not miss production targets, on pain of death. The rapid rotation of cadres, facilitated by Stalin’s purges, reduced the influence of patronage networks. Brezhnev’s policy of “stability of the cadres” ended the use of the firing squad to encourage effective management. That made the Soviet system more humane, but it degraded incentives to work efficiently. Bureaucrats and managers now faced few reasons to act effectively: their firms could not go bankrupt, their salaries did not depend on performance, and they received promotions based on political connections. In China, the Cultural Revolution of the 1960s had shaken up the party and the bureaucracy, cutting back the strength of interest groups and paving the way for Deng Xiaoping’s market reforms. In the Soviet Union, by contrast, the 1960s and 1970s saw patronage network and interest groups solidify. Gorbachev inherited a system in which economic lobby groups played a larger role than ever before. Yet his powers as head of the Communist Party were weaker than any Soviet leader since the Bolsheviks took power in 1917.

That mixing of Sapien and Neanderthal genes, which survives in the modern population, occurred in the Middle East, in the Caucus mountains, and in modern day Turkey. But of the Sapien DNA that we have from Europe before 30,000 BC, we cannot find any trace of it in the modern population.

That mixing of Sapien and Neanderthal genes, which survives in the modern population, occurred in the Middle East, in the Caucus mountains, and in modern day Turkey. But of the Sapien DNA that we have from Europe before 30,000 BC, we cannot find any trace of it in the modern population.

Sapiens in Europe went extinct roughly the same time that Neanderthals in Europe went extinct, or shortly thereafter. Careful DNA testing has not been any to detect any DNA signature that carried from 30,000 BC to the present population. All humans, of all species, seemed to have gone extinct at some point. It is known that Europe was colonized by Neolithic farmers who expanded out of Turkey about 17,000 years ago, this group had DNA that is still found in the modern population. But none of the older DNA seems to have survived. Therefore it is not clear that Sapiens eradicated or diluted Neanderthals in Europe. They all died out.

You discover that by writing the test. You don't try to predict that ahead of time. The goal is to get away from trying to predict problems ahead of time, because most of those problems turn out to be phantoms.

When it comes to code reviews, the return on investment faces the Law Of Diminishing Returns. While many of the comments made in code reviews might be interesting, they are not so interesting that they pay for themselves. If you put some dollar value on the time invested, you'll find that the vast majority of this process is simply burning money. And not only money, but also, as this article says, morale.

A curious fact about the politics of modern tech teams is that some of the same people who consider themselves "anti bureaucracy" are strongly in favor of this type of bureaucracy, even when it brings no measurable benefits.

My opinion of code reviews has become more negative over time, mostly due to the fact that I have not seen it achieve reliable positive outcomes.

I'll give specific examples:

https://www.futurestay.com had the worst code that I'd ever seen. When I joined the company I was horrified at the level of tech debt. And yet, for years, they had a rule that at least two engineers had to review every PR. So this terrible code, the worst I'd seen in my 24 years of coding, had been approved by two engineers.

And likewise:

https://openroadmedia.com also had very bad code, and also had a rule that nothing could be pushed to production until at least 2 engineers had reviewed the PR.

In both companies the code review slowed down the deployments, slowed the down the team, and created a culture of internal sniping, while leading to no improvements.

But how can I say the code was objectively bad? Because many bugs were found in production. And what would reduced the number of bugs in production? More tests, especially end-to-end tests. And so I eventually came to this conclusion: it is best to skip code review, and instead have the team invest that same time in writing more tests, especially high level tests. For the most part, if the engineers on a team are bad then the code reviews will also be bad, but if the engineers on the team are good, the the code reviews will not be needed. And in both cases, the path to improvement comes from writing the kinds of tests that ensure no bugs get into production.

As I've grown in my career, and taken on higher level management jobs, I've also realized that code reviews do not scale to large-scale leadership, but tests do. Code review seemed like a good idea when I was leading a team of 3 engineers, but not when I was leading a team of 30. When I was leading a team of 30, the only tool-of-oversight that worked for me was automated testing. And so I've concluded this is where the effort should be made. Sitting at my own computer, I cannot review the work of 30 engineers, but I can still run the tests on the various software, to see if they are passing. In particular, I can look at API interfaces and then change the dummy test data to break the tests, and this lets me see quickly thorough the test coverage is, and how prepared we are for unexpected shocks.

When I am running a team I will assign software developers the task of writing various tests, including high-level end-to-end stress tests. These tests sometimes reveal problems. We then respond to the problems. But we don't waste time responding to problems that have not been proven by a test, which is to say we do not do code reviews. Code reviews are all about predicting what might be a problem. But much of the concerns are phantoms. It is better to respond to real problems that have been revealed by tests.

I've also come to realize that software developers, quite naturally, develop strong opinions about questions of style, and yet these questions have no long-term impact on the health of the tech in the company. Other than enforcing the rules of some automated linter, the time spent on issues of style are 100% wasted. If you allow the team to spend even one minute discussing issues of style, then you are setting money on fire. But I know this opinion is unpopular, because software developers enjoy enforcing their own opinions about style. But it is all bikeshedding, it has no real-world impact.

I have the impression that there is some middle era, in the career of a software developer, where concerns about issues of style and organization tend to peak. The junior developer does not know enough to care about such issues, but somewhere between 4 years of experience and 10 years of experience, these issues feel important. I think you need more than 10 years of experience to see the waste. In particular, you need to run one large project where the team invests a lot of time into code reviews, and you need to notice how much tech debt builds up, despite the code reviews, to realize that code reviews do not offer a path forward.

On recent projects I've told the team there will be no code reviews, but instead we will focus on building tests. I get a surprising amount of pushback on this. Less experienced software developers get angry with me and tell me that I'm being unprofessional. In some sense they are correct, in the sense that "professional" refers to "standard norms that are accepted by a profession" -- I am deviating from those, clearly.

The most reasonable criticism I get is that code reviews could catch not just problems of style but problems of algorithms. What if, they ask, some junior developer introduces code that is ignorant of the implications of Big O Notation? What if they introduce code that runs in polynomial time? But I would ask, how do we know if it is running slowly? We can only know that through tests. So lets build tests that measure time, and stress test with large loads. Big O Notation offers an excellent example of where software developers can worry about the wrong thing: what if a junior level software developer writes code that runs in polynomial time but on data that will only ever have a few hundred records? While the algorithm might be sloppy, the code will run quickly because there are few records? In that case, any time invested to find a different algorithm will be a poor investment. Once I'm running a team of 30, the only thing I care about is whether code is actually slow, and I discover that through end-to-end stress tests, not code reviews.

Kent Beck, when he invented Xtreme Programming, also popularized the phrase "Do not write a comment, instead, write a method with an easy-to-understand name that communicates what the comment was going to communicate."

I've come to a similar conclusion. Do not write a comment in a code review, instead, write a test that would catch whatever danger you want to warn about. And if you cannot find a way to express your concern as a test, the return-on-investment of worrying about that concern is probably zero, so we should ignore it.

Interesting that the mania for over-investment in devops is beginning to abate. Here on Hacker News I was a steady critic of both Docker and Kubernetes, going to at least 2017, but most of these posts were unpopular. I have to go back to 2019 to find one that sparked a conversation:

https://news.ycombinator.com/item?id=20371961

The stuff I posted about Kubernetes did not draw a conversation, but I was simply documenting what I was seeing: vast over-investment in devops even at tiny startups that were just getting going and could have easily dumped everything on a single server, exactly as we used to do things back in 2005.

There is no money to be made from individual users. All of the money comes from companies building something on top of the LLMs, and those of us building startups on top of LLMs are very much aware of the differences between the LLMs. And, to the point made in the article, it is trivially easy for us to switch from one LLM to another, so the LLMs don't have much of a moat and therefore they cannot charge much money.

I agree that there must have been earlier writing, likely written on wood. Early systems could have evolved from markings on trees, like we still use on the Appalachian Trail, and other trails. Warnings for bears or tigers, symbols for different tribes on different paths. If you've hiked much then you're aware that even experienced woodsmen can get lost as the season changes and a valley changes, or after a hard storm washes away evidence of a trail. Children, in particular, would have been at risk, but would have almost certainly needed to do work over distances, in particular fetching water, which is something that even today children as young as 5 are asked to do. Notches on trees would have been a likely starting point for a system of symbols to communicate.

When I was much younger I used to work as a hike leader for a summer camp in Virginia. We would take a small group of teenagers out for 7 day hikes, during which we could cover something between 70 to 90 miles (112 to 145 kilometers). At one time I knew that stretch of trail so well I thought I could walk it blindfolded. And yet, I only knew it in the summer. One year I went in the fall and I was astonished how different it was. I was helped by the markings on the trees. (This was before cell phones and GPS.)

Progress towards what? Do you have some political goal? What you've written does not make sense.

Dan Morena, CTO at Upright.com, made the point that every startup was unique and therefore every startup had to find out what was best for it, while ignoring whatever was considered "best practice." I wrote what he told me here:

https://respectfulleadership.substack.com/p/dan-morena-is-a-...

My summary of his idea:

No army has ever conquered a country. An army conquers this muddy ditch over here, that open wheat field over there and then the adjoining farm buildings. It conquers that copse of lush oak trees next to the large outcropping of granite rocks. An army seizes that grassy hill top, it digs in on the west side of this particular fast flowing river, it gains control over the 12 story gray and red brick downtown office building, fighting room to room. If you are watching from a great distance, you might think that an army has conquered a country, but if you listen to the people who are involved in the struggle, then you are aware how much "a country" is an abstraction. The real work is made up of specifics: buildings, roads, trees, ditches, rivers, bushes, rocks, fields, houses. When a person talks in abstractions, it only shows how little they know. The people who have meaningful information talk about specifics.

Likewise, no one builds a startup. Instead, you build your startup, and your startup is completely unique, and possesses features that no other startup will ever have. Your success will depend on adapting to those attributes that make it unique.

“progress requires fundamental change”

Progress towards what? Do you have some political goal? There is zero evidence that flatness increases profits. There is no company that we can point to and say “that company was successful because it was flat” (with the obvious exception of franchise models).

"Hierarchy is so baked in to every other company, organization, and education system that people just don't know how to operate absent it."

That's partly because it works. It is proven. It is well-known. We have excellent tooling for it. There is absolutely no need to change it. Changing does not bring any advantages. You cannot point to any company and say "That company was an outstanding success because they were flat and rejected all hierarchy." If you run a business then do you want to make a political point or do you want to make a profit?

"There are examples of stable large companies with flat org charts that do work, they're just not the hypergrowth scale-ups."

So why waste any time with them? What is the advantage? What is your real motivation? You seem more focused on some political agenda than you are focused on making a profit.

If workers cannot do what the leadership needs, then they are bad workers, they need to be fired.

This explains everything:

"It’s not practical. Hybrid work may be technically the best route, but it’s also complicated to oversee."

Who cares if workers are productive, when the leadership is clearly less productive? And the leadership's time is extremely valuable. If remote work makes workers more productive, while the leadership is less productive, then remote work is bad for a company, full stop, no other conversation is needed.

How would that be possible? Novelty is a known weakness of the LLMs and ideally the only things published in peer-reviewed journals are novel.

Completely irrelevant. Humans cannot survive in the conditions of the Cambrian. Merely going back to the Cambrian is enough to ensure our extinction. No one cares that the CO2 levels were higher then. What matters is that we will all be dead if we go anywhere near those levels.

"The way we discover interesting websites needs innovating, why not let anyone contribute to any webpage?"

I remember there was a website that did this in 1999, using frames to allow people to post comments on any website. The courts shot this down as an illegal infringement of trademark. Does anyone remember the name of that website that did this?

"Most of the difference in GDP was because Dollar was a lot cheaper compared to Euro pre Recession."

Yes, but why? There was a theory that massive stimulus spending would weaken the dollar because of the additional debt, but instead, the opposite happened. The USA did more stimulus spending and the dollar became stronger. Many people were surprised by this. Those people need to revisit their initial assumptions.

Since the crisis of 2008 the USA has had several trillion dollars in stimulus spending. Very roughly speaking, Europe pursued policies of austerity, whereas the USA followed a more Keynesian route of big spending. On this matter, the Keynesians have been vindicated. In 2007 the GDP of the EU was slightly larger than the USA, but now it has fallen far behind: $16 trillion ($18 trillion if it still had the UK) versus $25 trillion for the USA. And the most important policy difference since 2008 has been large stimulus spending in the USA, versus relative austerity in Europe. More recently, President Biden was able to push through some big infrastructure bills, which should power the USA through the 2020s. (There are some qualifiers to be added about weaknesses in the USA system of funding and allowing construction, in particular the aggressive system of "substantive due process" that allows for any project to be stalled by lawsuits, but despite that, the USA has done better than Europe.)

"hardware startups that have to raise a lot of money before a Series A are... considerably riskier, no?"

Why riskier? They have more of a moat don't they? The large capitalization needed suggests they will face less competition. It's more difficult for competitors to gather the necessary capital to compete against them?

What I've written here was the conventional wisdom for most of 200 years. Much of the Industrial Revolution played out when merely concentrating together capital was seen as an engine of growth. Rockefeller did not need to sell innovative gasoline, he only needed to use cash flow to buy up monopoly positions, one small region at a time, until he had the cash flow to buy up monopoly positions nationwide. Economies-of-scale meant that merely concentrating together capital was a path to greater profitability.

The last 25 years were an aberration, during which time big companies could be built with small initial investments. But over the course of centuries, the opposite was more common.

On a slightly different topic, we now know that a surprisingly large number of people suffer daytime "hallucinations without delusion" meaning they see things but they know the things are not real. And I've often wondered if that is some kind of dysfunction of the sleep cycle. If we think of dreams as a kind of hallucination, then is daytime "hallucination without delusion" a kind of dreaming while awake? I'd like to see more research on that question.

I agree with the use of Hetzner. I use them. Very cheap machine, very powerful. Very simple and straightforward. People need to think carefully about the possible expense and complexity of AWS. Hetzner is simple and straightforward.

The problem with the title is simply that the English language allows open compound words. I know many Germans wonder why we allow this. Germans push the words together, for clarity. I've suggested that we use hyphens. Hyphens feel natural in English, and could remove the ambiguity that exists whenever we use open compound words (that is, open-compound-words). In this case "not-comments" would have added clarity.

Likewise, the title "World's longest DJ set" was confusing, because most people will assume that the compound word is "DJ-set". But if you read the whole article, then you realize that a python snake fell on the mixing board and accidentally mixed some tunes. So the compound word was actually "longest-DJ" -- a 2.5 meter python.

We should all consider using hyphens for all compound words.

https://x.com/krubner/status/1828155852773113942

Github won in part because git won. And git won because, for complex sociological factors, the software engineers were able to argue that their needs were more important than the needs of other parts of the companies for which they worked.

For a counter-point (which I've made many times before) from 2005 to 2012 we used Subversion. The important thing about Subversion was that it was fun to use, and it was simple, so everyone in the organization enjoyed using it: the graphic designers, the product visionaries, the financial controllers, the operations people, the artists and musicians, the CEO, the CMO, etc. And we threw everything into Subversion: docs about marketing, rough drafts of advertising copy, new artwork, new design ideas, todo lists, software code, etc.

The whole company lived in Subversion and Subversion unified every part of the company. Indeed, many products that grew up later, after 2010 and especially after 2014, grew up because companies turned away from Subversion. Google Sheets became a common way to share spreadsheets, but Google Sheets wasn't necessary back when all spreadsheets lived in Subversion and everyone in the company used Subversion. Likewise, Google Docs. Likewise some design tools. Arguably stuff like Miro would now have a smaller market niche if companies still used Subversion.

At some point between 2008 and 2015 most companies switched over to git. The thing about git is that it is complex and therefore only software engineers can use it. Using git shattered the idea of having a central version control for everything in the company.

Software engineers made several arguments in favor of git.

A somewhat silly argument was that software developers, at corporations, needed the ability to do decentralized development. I'm sure this actually happens somewhere, but I have not seen it. At every company that I've worked, the code is as centralized as it was when we used Subversion.

A stronger argument in favor of git was that branches were expensive in Subversion but cheap in git. I believe this is the main reason that software developers preferred git over Subversion. For my part, during the years that we used Subversion, we almost never used branches, mostly we just developed separate code and then merged it back to main. Our devops guy typically ran 20 or 30 test servers for us, so we could test our changes on some machine that we "owned". For work that would take several weeks, before being merged back to main, we sometimes did setup a branch, and other times we created a new Subversion repo. Starting new repos was reasonably cheap and easy with Subversion, so that was one way to go when some work would take weeks or months of effort. But as ever, with any version control system, merge conflicts become more serious the longer you are away from the main branch, so we tried to avoid the kind of side projects that would take several weeks. Instead, we thought carefully about how to do such work in smaller batches, or how to spin off the work into a separate app, with its own repo.

A few times we had a side project that lasted several months and so we would save it (every day, once a day) to the main branch in Subversion, just to have it in Subversion, and then we would immediately save the "real" main branch as the next version, so it was as if the main branch was still the same main branch as before, unchanged, but in-between versions 984 and 986 there was a version 985 that had the other project that was being worked on. This also worked for us perfectly well.

The point is that the system worked reasonably well, and we built fairly complex software. We also deployed changes several times a day, something which is till rare now, in 2024, at most companies, despite extravagant investments in complex devops setups. I read a study last week that suggested only 18% of companies could deploy multiple times a day. But we were doing that back in 2009.

The non-technical people, the artists and product visionaries and CFOs and and CMOs, would often use folders, when they wanted to track variations of an idea. That was one of the advantages of having something as simple as Subversion: the whole team could work with idioms that they understood. Folders will always be popular with non-technical people.

But software developers preferred git, and they made the argument that they needed cheap branches, needed to run the software in whole, locally on their machines, with multiple variations and easy switching between branches, and needed a smooth path through the CI/CD tools towards deployment to production.

I've two criticisms with this argument:

1. software developers never took seriously how much they were damaging the companies they worked for when they ended the era of unified version control.

2. When using Subversion, we still had reasonably good systems for deployment. For awhile we used Capistrano scripts, and later (after 2010) I wrote some custom deployment code in Jenkins. The whole system could be made to work and it was much simpler than most CI/CD systems that I see now. Simplicity had benefits. In particular, it was possible to hire a junior level engineer and within 6 months have them understand the whole system. That is no longer possible, as devops has become complex, and has evolved into its own specialty. And while there are certainly a few large companies that need the complexity of modern devops, I've seen very few cases myself. I mostly see just the opposite: small startups that get overwhelmed with the cost of implementing modern devops "best practices", small startups that would benefit if they went back to the simplicity that we had 10 to 15 years ago.