The Broken Chain Problem

I don't understand, why do things like this happen? Is someone emotionally invested in upgrading the DFS servers?

A few years ago, i worked on a payment processing system for a high street bank. The system, in its first stage of evolution, would receive a file full of payment instructions, parse each one, validate it, then pass it on to a gateway in a special binary format.

An architect had decided we should build this as microservices, coupled by message queues. There was one microservice which split the file into records and parsed each one, one which validated each one, and one which handed off to the gateway.

In the first demo of the system, it took half an hour to process a file with a few thousand instructions in it. This was a ludicrous amount of time: it really should have been done in seconds.

The team met, and one of the senior developers pronounced the solution: since this was a microservice system, we would simply split one of the microservices in two, so we could scale out the amount of hardware we were using. The service selected was the first: rather than splitting and parsing the file in one pass, a first microservice would split the file into chunks, and pass each one onto a second microservice, which would parse them.

I asked where we thought the problem was, why we thought splitting a service would help, how we would measure the improvement, and, moreover, why we thought that adding a network hop would make things faster rather than slower. My questions were rejected as self-evidently unnecessary: it is well-known that you can improve the performance of a microservice system by scaling out.

The service was split. There was another demo. It was just as slow as before.

The project was removed from that team, and transferred to a new team under my leadership.

(this is a true story; the one fact omitted for the sake of drama is that the transfer of the project wasn't due to the embarrassing demo, it was purely down to staffing and logistics)

Covering others incompetence is breaking a chain.

Uhm, it's interesting, I see a little contradiction in your two points...

I always apply Chesterton’s fence before breaking rules and regulations: if I see what the rule intends to accomplish, I can make sure to accomplish the same thing in another manner (walk across the street even if there’s a red light if there are no cars around). If I don’t see the point of the rule, I’m hesitant to break it.

I apply the scream test when refactoring. Comment out a feature and wait until someone complains. After half a year or a year start removing things. Works pretty well. This works better with internal systems. I wouldn't do it with external customers or at least only with a lot of thought.

> The Scream Test: If you see something and you don't know what it does, remove it and see if anyone screams.

Spoken like someone who works on pure software, with Undo!

Out in the physical world, the screaming would be followed by a loud crash. You don't touch anything if you don't know what it does. That's how equipment is destroyed, and people die.

It only seems to be difficult for philosophers.

They didn't average the length of everyone's nose.

They averaged everyone's estimate of the emperor's nose.

More importantly, it sounds like everyone who's going to actually see the statue got a chance to provide their input. So this method maximizes the number of people happy with that part of the sculpture, anyway!

Obviously, number or weight of standard-compliant nails.

The story seems to be based on a satire cartoon: https://skeptics.stackexchange.com/questions/22375/did-a-sov...

Certainly the practical issue of what to use as performance indicator is a deep one, at least if one wishes to have a system that efficiently satisfies the needs of the population rather than their stated wants.

Today we could probably build some decent abstract indicator out of a myriad of detailed statistics, but the soviets didn't have much IT infrastructure in place and had to make do with simpler indicators. The linked memoir has the following summary of how the planned economy worked in practice:

> The Soviet manager’s success indicator was a measure of gross output, such as weight, quantity, square feet, or surface area. [...] The Soviets experimented by adding other indicators, but in the end a gross output indicator always determined the manager’s success or failure.

> Soviet managers were as autonomous as their market counterparts. They set their own plan targets by disguising their productive capacity and overstating their resource needs. Soviet planners served primarily as supply agents for enterprises, endeavouring to supply the enterprises with sufficient inputs to fulfill their gross output targets. The system of material supply could seldom perform this task, and Soviet factory managers made barter arrangements with one another and produced their own inputs. This activity led me to the conclusion that the Soviet economy, like a market, was organized polycentrically and not hierarchically as a planning system. The “central plan” was little more than the summation of the factory manager's individual plans.

I would caution that this account seems like it overstates the actual independence of the managers, no doubt there were many political restrictions on their freedom of action in practice, but the general account that the plan essentially comes from the low level managers seems sound.

For a much more detailed fictional but based on true events take, there is Francis Spufford's novel Red Plenty. It details how certain factions in the Soviet Union had considered the optimisation problem (including Kantorovich, the inventor of "linear programming"), and were effectively proposing pseudo-market systems. So long as these could be made politically acceptable by not having anything labelled "price".

> what should the central planners give as metrics? Is there anything that works?

According to Goodhart's Law, no. As he originally explained it: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Metrics of GDP, inflation, and employment are commonly used in macroeconomics to approximate quality of life, but they're subject to the same problem. For instance, you may be able to increase GDP and employment by increasing violent criminality, expanding the markets for private security, insurance, weapons and prisons, but harming general quality of life.

Have quality control that verifies the nails meet specifications, and only reward production of good nails. It's better but still lacks normal market feedback.

The system as described actually worked fairly well in an economic sense for a few decades, pulling Russia up from a poor agrarian country to a much wealthier industrialized country. In the 1950s there was a legitimate case to be made that Communism was winning and would soon beat capitalism at its own game. Essentially, if the recipient of the nails had their own large-scale industrial project to complete they could pull strings with the Central Committee to make sure they got the raw materials they needed to get their own bonuses. There wasn't much representation of end consumers, so they tended to get low quality stuff that they didn't want, but the system was actually quite good at producing increasing amounts of steel and cement. However increasing disillusionment and corruption, combined with the complexities of a computer-age economy lead to the gradual erosion of the system. There was a bona fide attempt to build a computer model to gather all the relevant inputs and do the central planning, but it didn't get very far in practice.

The book Red Plenty gives an informative and compelling lightly fictionalized account:

https://slatestarcodex.com/2014/09/24/book-review-red-plenty...

> Is there anything that works?

Markets mostly work.

> Now I wonder: what should the central planners give as metrics? Is there anything that works?

No. You can't set a metric and let the metric do your governing for you. There needs to be a human in the loop. In other words, the system has to be run by people who are making a good-faith effort to achieve the real objectives.

Markets partly solve the problem. You can have a market economy that optimizes the production of nails and other widgets according to the metric of profit, but if that's all you have, you will end up with corporations that produce lots of widgets at a huge profit, but in the process burn out workers, pollute the environment, and fill news channels with scandals about how Acme knew for the last thirty years that its products were poisoning consumers, but kept the truth hidden. So you need regulations to mitigate these problems. But this in turn requires that the regulators are acting in good faith instead of trying to game a metric. There always needs to be someone in the loop.

I don’t think the sculptor was planning to sculpt an outlandish alien nose. The problem is that if your statue doesn’t look right heads will roll.

The story isn’t about the selection of outlier data points as being representative - its more about drawing conclusions through seemingly plausible sounding methods that are nevertheless completely disjoint from the thing you’re reasoning about.