Hacker News Reader

Comment by rossdavidh | Hacker News Reader

While it is obviously not unrelated to the pandemic chaos, there is a deeper issue. As Nassim Taleb has often pointed out, the same things that make a system "efficient" also can often be viewed as making it fragile. "Just-in-time" means "no buffer". We have, for decades, been doing as much as possible to make all our economic systems as efficient as possible, which is to say with no extra capacity.

Thus, if it had not been the pandemic, it would eventually have been something else. Sooner or later, there is always a shock to the system. If you have made sure that every step in a long complex chain is optimized to have very little slack, the thing you have optimized for is fragility.

View on HN

> "Just-in-time" means "no buffer".

Just in time doesn't necessarily mean no buffer, the goal is to minimize excessive stockpiling and keep enough for continuous production, increasing stockpile when required.

Toyota, some might say THE pioneer of Just-in-time manufacturing, was one of the car manufacturers least impacted by the component shortage precisely because they started stockpiling very early on after they saw the upcoming issue.

Here's a quote from another Bloomberg article specifically on Toyota[1]:

"Toyota asks its Tier 1 suppliers to input detailed information about their most obscure parts and materials providers in a complex database that it maintains. Using this system to glean information about, say, a single headlight Toyota purchases for one of its cars, it can get information as granular as the names and locations of the companies that make the materials that go into surface treatments used on those headlights’ lenses and even the producers of the lubricants used on the rubber pieces in the assembly, Toyota spokeswoman Shiori Hashimoto says.

These lines of communication alerted the company early on that it needed to stockpile chips."

[1]: How Toyota Steered Clear of the Chip Shortage Mess - https://www.bloomberg.com/news/articles/2021-04-07/how-toyot...

This is actually also discussed by Nassim Taleb in his book which I assume the parent comment is referencing.

He says Toyota is one of the few examples where just-in-time was not distorted and warped. So Toyota implements true just-in-time, which can be robust, with buffers. But most other companies implement a half-assed version that is very fragile.

While attending a training event, I talked to a pilot who flew cargo for a Tier 1 supplier to North American auto manufacturers. He had several stories of flying a jet with just a handful of boxes on board to avoid the line-down penalties.

My favorite was tower advising them they’d be fined for a departure during local curfew. “Roger, we better get our money’s worth then” and did a max performance takeoff and turn on course with whatever parts they were carrying.

Heh there are airports that have curfew fines that all the airlines just pay.

We have a local field with an 11P-7A curfew (landing/takeoff surcharge technically).

I’m not sure it does much to keep the airport quiet as I’ve been in a holding orbit of three low-altitude airplanes from 6:45 AM to 7:00 with all of us waiting to avoid the curfew fee.

So, instead of slipping relatively quietly into the field from a random route to a straight-in, neighbors got to hear me orbit for three rectangular patterns over the same ground points plus two other airplanes doing the same low-altitude holding. If I were a neighbor, I’d much rather the simple straight-in to be encouraged.

The sports teams all just pay the fee; I never had to break the 11PM side while I was based there, but I would have as well. (It was less than an hour’s worth of gas, so it wouldn’t make sense to divert and reposition later.) I suspect the curfew makes people feel good while making the actual experience worse.

The curfew at Sydney is $1M (USD 730K), so large that airlines go to considerable lengths not to violate it, making sure they have accurate forecasts for the winds from LAX and running predictive models for various landing scenarios. They will really burn avgas if they can squeak it in before curfew, and they have been known to divert flights to Canberra or Brisbane and put the passengers into a hotel overnight rather than risk it.

https://www.infrastructure.gov.au/aviation/environmental/cur...

But those extra 15 min in the air cost money, so it's still an incentive to not plan on arriving before the curfew ends.

Sometimes those curfew fines actually cause flights not to depart. Some years ago, I was sitting on the tarmac at John Wayne for about an hour, hoping to make it just under the nighttime curfew, but ultimately the flight was canceled and I had to get a shuttle to LAX and an alternative flight in the morning.

John Wayne is more forceful than some for sure. I flew a small plane their once and took off just at curfew - the runway lights went off the moment my wheels left the ground.

If they turned off before you passed the departure end, I’d raise that as safety concern (ASRS at a minimum, but this is something that I’d probably take straight to the FSDO).

It's been a while, but flying into Long Beach from a connection in Phoenix, the scheduled plane was delayed somewhere, they had boarded us on a replacement and then realized the plane was too loud, so they rushed us to a different plane (in the next terminal) that had the right noise reduction equipment.

Also, my Dad once had a late-ish flight to Long Beach that got diverted to LAX because delays pushed it past the curfew.

...and Long Beach's stubbornness about that curfew and other things is a big part of the reason JetBlue packed up and moved all its operations to LAX

https://crankyflier.com/2020/10/08/jetblues-two-decade-run-i...

This is not quite accurate. Toyota began stockpiling materials a while back after having its supply chain disrupted. Having extra inventory, no matter how well-implemented, is actually an anti-pattern in lean manufacturing -- the good folks at Toyota are simply wise enough to know when they they should embrace lean vs. when they should back off it a bit.

> Having extra inventory, no matter how well-implemented, is actually an anti-pattern in lean manufacturing

Arguably lean manufacturing is an anti-pattern.

> Arguably lean manufacturing is an anti-pattern.

When taken to the extremes that modern business has taken it to, it is 100% an anti-pattern.

Which book?

I'm not sure, but I guess it's "Antifragile"

so, like Agile then

The difference is that Toyota is a uniquely intelligent company with respect to supply chain.

The average American manufacturing company is run by a CFO operating with reports out of SAP or Peoplesoft. Their performance is measured by fiscal performance, so running to the penny and having no inventory benefits them more than making the company resilient. Wall St rewards quarterly performance, not resilience.

Being a resilient company only means that when it’s time for an industry bailout you get less than your competitors because you’re doing better.

Private profit, socialized risk.

On the other hand, companies that have inventory right now get to increase market share over those that didn’t think ahead and have nothing to sell.

But if the optimal strategy only falls apart for uncommon events that you can't determine when they happen, how much of an incentive is there to run a conservative/gambling strategy like that?

It depends on your objectives. One bad event can put many companies out of business. The auto industry is a good example because US-Canadian border issues and snow can fubar things for half the year.

Look at companies like Boeing with absurdly complex supply chains. The inventory numbers look good, but the factories are idle when a truckload of magic bolts is stuck in a blizzard in South Dakota.

It also increases actual cost. I supported a GE business unit as a supplier for awhile. We hosed them for stupid last minute orders due to this sort of thing. They would pay more for expedited shipping, overtime, waste money on leasing stuff to avoid capex, etc.

Uncommon events are uncommon individually but common in aggregate.

Maintaining resilience and designing antifragile systems usually pays off in missed disasters and unique opportunities.

Alternately, to save money without buffers is to continuously gamble.

There is no optimal strategy because there is no perfect knowledge. That's why good CEOs are paid their weight in gold. A vision and steady hands can pay off handsomely.

Depends completely on the risk vs reward. Black swans are more common than can be assumed from modeling past data so in general a slightly more conservative strategy than industry standard is more likely to win.

For example COVID style shutdowns could be reasonably modeled at X% per year after talking to a disease specialist. But, not all of them are going to be from diseases.

Depends entirely on the time frame. If you have a superior capital allocation scheme, by all means, capture all the value left on the table then.

All of the car companies, dialed down orders at the start of the pandemic. Now they are all trying to stockpile chips, due to not enough supply. I imagine (no data to support) the increased size stockpiles are a large part of what makes the problem worse. Kinda like TP really.

Also, Toyota is now cutting 40 percent of global production due to chip shortage.

https://www.bbc.com/news/business-58266794

It's not mainly due to chip shortage, but other parts made in ASEAN. https://asia.nikkei.com/Business/Automobiles/Toyota-to-cut-g...

Tail whip effect

hmmm interesting - here's an article.

https://qz.com/2004569/the-global-chip-shortage-can-be-expla...

Oh, true! English is not my main language. I meant bullwhip effect, not tail whip (pokémon move). ;)

An interesting failure mode of free markets. Inherent instability.

Any system with signalling delay will experience this effect. Changes will get amplified across layers. That's also a common way to die from a viral infection: your immune system goes into overdrive from too many signals of infection from different tissue cells. A common treatment is steroids, which dampen immune response.

Is an inherent instability in any system with long/slow information chains. It's more a failure mode of central planning than of free markets. We had a few companies making bad decisions based on bad demand forecasting. This is a problem mitigated by stock exchanges (commodities).

What you fail to mention (although you linked to an article that covers it) - is that Toyota was prepared due to the setbacks they suffered in the Tsunami. If it wasn't for that, they would not have stockpiled for and been ready for a pandemic. Even so, they are now dealing with chip shortages as well.

A system that turns set backs into lessons and better future performance is praise worthy...

The thing is that you can not apply this method (suddenly stockpile because of anticipated shortage) globally. Now of course, putting gratuitous buffer everywhere would not be the panacea either. But some buffers may be needed to improve global resiliency.

One has to optimize to reduce fragility, but when massive chip (or anything else) shortages start to appear for extended periods it is obviously already way to late. And way too downstream if your strategy was just a punctual stockpile decided by a single company. Because that probably would have been impossible for everybody to apply this punctual strategy at the same time...

One advantage of starting to stockpile ahead of a crisis is that production is not yet affected much by the crisis, and the total amount buffered could be larger.

They started stockpiling key components after the earthquake roughly 8 years ago. You could do this globally if everyone took the Toyota approach and only stockpiled components likely to suffer disruption.

Seems like a low-on-details article with PR-like flavor.

According to this, Toyota cut production by 40%, though it does acknowledge that Toyota took less of a hit vs industry:

“New cars often include dozens of microchips but Toyota benefited from having built a larger stockpile of chips - also called semiconductors - as part of a revamp to its business continuity plan, developed in the wake of the Fukushima earthquake and tsunami a decade ago.”

https://www.bbc.com/news/business-58266794

What's the quote about selling someone a product whose faults will occur when you're years gone and half way around the planet? I wonder how this mentality is avoided/regarded in Japan.

When many of the quality-focused companies in Japan took off they focused on gaining market share primarily through gaining repeat customers. A lot of their sales machinery was constructed around the idea of building a personal relationship with the customer to predict their needs as they arise.

This was in part because the domestic Japanese market was so small. Of course, it needs support from quality development and manufacturing. But it's also efficienct because it's much cheaper to sell something to someone who trusts you than to get it out by cold calling.

I think this papers backs up pretty well and goes into a lot more details of why it matters to focus on the flow. In fact it’s crucial that such issues that stop production including predicting and identifying demand-supply mismatch is essential for a smooth flow of operations. BUT, humans are humans and we plan for the best case scenario without giving due consideration for failure modes. Manufacturing is hardly an exception.

https://businesswales.gov.wales/sites/business-wales/files/S...

I wonder if they track the financial status of these far upstream suppliers. I've been doing some work with supply chain monitoring and often smaller companies in Europe are hesitant to share information on financial performance etc

"least impacted" = 360,000 Toyotas not being built this month, or 40% of normal production.

https://www.bbc.com/news/business-58266794

As your link says Toyota has not been materially impacted until now, whereas all the other manufacturers had to scale back their production way earlier (I'm seeing articles from January and March). 8 months longer runway than everyone else is a pretty good deal.

Toyota is underwater like everyone else. When you have signal, it is already too late

Just-in-time, done mediocrely, still means “no buffer”. What Toyota does is not all that relevant to the discussion.

It PRIMARILY DOES mean this as it's IMPLEMENT in 99.9% of businesses. Toyota's implementation is NOT NORMAL OR COMMON.

The Toyota Production System is /the/ precursor to “Lean Manufacturing”, you could say that Toyota wrote the book on “lean”. While there are many companies that try (and fail) to replicate TPS, the Toyota Way is the standard by definition.

That makes it the archetype, it doesn’t contradict the claim it is uncommon.

(I have no idea if it is common or not, just that this doesn’t contradict that claim)

perl4ever had an interesting comment that they deleted after being downvoted for some reason. I'm happy to take the downvotes because I think their take was a thoughtworthy counterpoint.

* * *

>"Just-in-time" means "no buffer"

No, it doesn't. It just means the buffer is somewhere other than where you assume it's necessary. That may in fact be a net gain. Computers and cars and whatever don't wear out instantly. It will be a long time until all the newest equipment that already exists is old enough it absolutely has to be scrapped. In some other thread on HN, someone said that everything depends on manufacturing, and services don't mean anything without manufacturing capacity. Services! Like maintaining and repairing what's already been manufactured! Like, for an extreme example, what happened in Cuba when they couldn't get new cars for a long time. Or during WWII when civilians couldn't buy new cars. Day in and day out, people talk about how terrible a disposable society is, and then we have a crisis that requires like 1% less disposal and it's the end of society.

This comment was probably down-voted because it completely misses the fact that not everyone already has access to the needed resources. Sure, even if you already have everything you want/need now, and you own high enough quality parts, you may be golden for a decade or so. Eventually though, you'll need replacements. If access to those replacements is limited, how much value does the rest of your scrap really have?

Creative outlets for junk are going to continue to become more and more fashionable as we struggle to meet our values and practical prices.

The central part of the comment was this:

> It just means the buffer is somewhere other than where you assume it's necessary.

The rest was an example of one place you might see such a buffer. In practice, there are many such buffers, of varying capacity and time duration. For transportation, there’s the households who have 3 cars but could get by with only 2: they’ll sell their extra car once the price makes it worth their while. There’s the cyclist who’d like to have a car for out-of-city trips, but decides to hold off until inventory catches up (a demand-side buffer, if you will). Many more “buffers” could be imagined.

And, well, the market libertarian might feel really satisfied over the last year of commodity prices. Sure, supply constricted and prices rose, but at no point did we see any mass crises where people became stranded somewhere because they had no transportation option. For the vast majority of people, they just had to make convenience/cost tradeoffs that were a little bit different than before.

> but at no point did we see any mass crises where people became stranded somewhere because they had no transportation option.

You are just not paying enough attention. As prices rise the poor become poorer. How can you possibly claim that nobody is stranded by the increase of prices?

I think we’re maybe speaking past each other. I don’t disagree that things have become more expensive — and hence that the poor can own relatively less than they could before.

But, no. I haven’t heard stories of people being stranded in mass. I meant that word literally — because of the context of transportation — if that wasn’t clear? Even in the days of mass lockdown: I saw huge lines at food banks in suburban and even rural areas, but even those who are in a position to wait four hours for a few days of essential goods seem to have still found ways to get there. That observation is subject to some obvious biases, and yes I’m sure some people did fall through the cracks, fatally. But no: I truly haven’t witnessed what I would consider mass crises caused by people losing access to transportation. Despite massive shocks to a system of JIT manufacturing, there wasn’t a collapse of the supply side in the sense that you would have expected had you looked at inventory, divided by the rate of consumption, and thought “that’s how many weeks it’ll be until anyone who doesn’t own some form of transportation will be physically stranded”. Whatever services and goods existed two years ago, equivalents can by and large still be found today, albeit at a possibly-increased price or a possibly-decreased utility. They may have dimmed, but “the lights stayed on”. Is the world free of problems? No.

I don't think the comment completely ignores that. As the Cuba example demonstrates, the potential for a strong second hand market for existing repairable hardware is another form of buffer.

One of the issues is that repairability and/or durability have also been decreasing for many categories of goods.

I also got downvoted for stating that I feel I have enough HW and can live a few years off what is there. I guess people have been trained to get the newest thing every year, that it now hurts them bad to not be able to do so for a few months. And yes, I agree, this could be a wonderful time to actually strengthen our knowhow regarding keeping things alive, instead of just throwing them away. Sounds like a great chance for the climate change people. How is the market for used equipment doing btw?

To speak to a "long perspective" on the issue, that was one of the things that scared people when electronics moved to integrated circuits. There's no "knowhow to keeping them alive". Once the magic smoke gets let out, they're just dead.

There was a time when every part of a computer - including, literally, an individual "bit" of ram, could be hand-repaired, craftsman-style (and even visually assessed for failure), but we've been on a long, long trajectory towards none of this stuff being user-serviceable.

We're now watching the rise of SoC, wherein my graphics card and my ram crawl into my CPU and disappear as discrete components. Strange times.

This can still require a certain amount of DIY.

I've just replaced a Blackmagic Atem Mini: sadly, I tinkered with my previous one trying to keep it alive and couldn't return it. I burned it up trying to do too much with it: ran the internal processing at super HD from a Pocket Cinema 4K, with two separate other inputs coming in and having to be rescaled, color corrections, and then the output downscaled after chromakeying to HD.

The unit basically melted itself until all it could do was flash bright colorful lines. Total loss.

So the replacement (dropping right into the previous workflow, though I'm making a point to not run as many rescaled inputs by default) will sit on a homebrewed heatsink with thermal pads and fan. That way it probably won't burn up, but if it did I won't have taken it apart, and I can most likely return it for a replacement.

The inability to repair is a concern, but on the other hand building the larger environment into the cooling support the unit should have already had, is the DIY aspect. Should work.

Well, I get your somewhat nostalgic attitude. But back then, computing equipment was also way large, louder, and energy consuming. Heck, one of the "museum" disk drives we have at work used to make the light flicker in the lower floor if you powered it on. So yeah, we're past a certain point of repairability. Thats sad, but more a function of progress, not really something we can do about.

But that doesnt mean all our existing tech will emit the magic smoke in the next few months. The endless need for more computing power is mostly driven by software bloat. We could extend the current smartphone and gadget usefulness by a significant amount of time if we stopped to fall for constant featurism and did lets say, a year or two of cleanup. Sure, "the future" like AR and such will have to wait, but heck, this "we will sell you 3d" thing is around since the 80s and didnt take of in the mainstream yet, so what?

If we kept our tech level at around the 80s but only increased the power efficiency, we wouldn't be in such a bad position.

We couldn’t really have achieved this degree of power efficiency on 80’s tech though. Likewise, we couldn’t have gotten to 2020’s tech (always-on wrist-watch displays that last days), without increased power efficiency. Advancements in either one facilitate advancements in the other.

As a fun example, I’ve been simulating core memory at the level of electric/magnetic fields lately. It takes literally trillions of floating point operations to simulate just 1 nanosecond of a single bit of core memory. In doing this, I can identify things like “if we were to arrange the cores like <this>, we could reduce the peak magnitude of the radiated fields when a core flips, thus we could get by with using slightly softer ferrites (which use less energy when switching state) without encountering noise problems.”

But of course, a trillion fp operations was a lot more back then! Having more advanced tech made it substantially easier to optimize the earlier tech. Optimizing compilers are maybe a more direct, but limited, example of this. The faster a CPU can run your code, the more cycles you can have your compiler spend making that code run even faster. I’m not sure there’s a good term for these “mutual feedback loops”, but they seem to pop up a lot.

Aren’t most of the power efficiency gains from integration and miniaturization? If you’re charging/discharging a larger amount per bit, it’s going to take more power than a smaller (physically and electrically) equivalent would. That tends to work against practical field repairs (other than replacing at the IC or subassembly level).

The shorter the distance between two components the less energy you're going to lose to resistance, electromechanical components have physical friction, conversion inefficiencies going from electrical to mechanical energy, and the energy needed to accelerate a mass.

Really, what nukes the repairability of a lot of products these days (ignoring the repair-hostile practices of many companies) is the cost of the item vs the time needed to disassemble the unit, track down where something went wrong, repair it, reassemble, and hope that component didn't blow because something else upstream is faulty that you didn't find. Cuba and Africa the time is cheap while replacement units are somewhere between expensive and unobtainium, USA/Japan/Europe not so much.

If floppy drives were common today I think many of the issues you highlighted like increase power usuage would be solved and applied to this domain. We don't have to be past that power.

Are there any standards bodies or informal organizations that try to quantify software bloat and featurism? Seems like an opportunity for groups that are motivated to keep software lean and consumer friendly to pool some resources ala open source. Anything like that out there already?

> if we stopped to fall for constant featurism and did lets say, a year or two of cleanup

I hear that all the time. Usually, in the same paragraph, they'll also ask for Feature X, Y and Z.

> I have enough HW and can live a few years off what is there. I guess people have been trained to get the newest thing every year

I imagine the downvotes were for implying that everyone else is in a similar situation as you. What about people who were being efficient before the shortage, and now can't keep their aged equipment running any longer? What about businesses who suddenly need much more equipment than they did previously, due to pandemic restrictions?

> How is the market for used equipment doing btw?

I've found it much harder to find available supplies at any given time, but I haven't actually seen prices going up as a result.

>> How is the market for used equipment doing btw?

I sold my stockpile of laptops (anything i5 or newer) for triple it's value last year. People would buy anything at that point.

Crazy. And also good, since that hardware is likely getting used more now.

Have you never had a laptop stolen or a phone destroyed by water damage or a car totaled in an accident? Sometimes you actually need a replacement, and when you do, you need it now because you depend on it for your livelihood. Buying new is not always the mindless plodding on the capitalist treadmill you make it out to be. There are plenty of people who can and should delay purchases right now, but there will always be those who just can't wait.

There's definitely a sort of "keeping up with the Joneses" that goes on in technical fields with regard to hardware. I do all my personal projects on a 2013 model Ultrabook. Sure it's underpowered, but not for editing text, compiling small codebases, or building CRUD WebApps. If I need to do some derp learning, I can just spin up a cloud instance. There are certainly productive tasks that benefit greatly from newer hardware (video editing, compiling the Linux kernel, launching a single instance of the Android Studio emulator), but most of this hardware capacity is probably for gaming and, like many people, I'm not much of a gamer.

Since I am still mostly on a plain terminal, I used to have a time when I used a rpi0w as a laptop, since I really dont need much more then an instance of Emacs and a few ssh sessions in a tmux. I switched to a real laptop by now, and its nice to have a bit of local power. But I could actually do all my dayjob on a rpi0w.

https://blind.guru//blog/2016-06-12-brlpi.html

https://hackaday.com/2019/03/26/not-all-raspberry-pi-laptops...

What's HW in this context? I can only think of homework and hot water, but neither makes sense.

Hardware

I think the message is true but misses a major point: yes, using existing hardware, we generally have some buffer though the simple fact of delaying replacement or new purchases. Equipment can recirculate (second-hand) and be maintained.

What is being missed are the larger societal needs for new hardware. If you build infrastructure (new public transport line, factory, power plant, communications network, etc) you need new components, lots of them.

The impact of shortages gets projects delayed or cancelled due to lack of availability of components or ballooning costs. It's also not just electronics, it's also shipping and raw materials costs that are impacted.

> "Just-in-time" means "no buffer".

Personally I have been ranting about the Modern use of Just in time manufacturing for well over a decade. Just in Time manufacturing as it originate in Japan doesn't actually means no buffering. Somewhere a long the line in typical Chinese whisper fashion the true meaning got loss when it go to the west. If you are software developer, just like at Agile and Kanban board.

The problem make worst when CEO and COO dont actually understand this, and force those in supply chain to comply with their view of JIT. It works in a sense when everything is normal. It doesn't work when there is a shock. Look at Apple, the only company that took JIT to a level beyond Toyota. They are doing just fine while others are fighting for parts.

The other thing worth pointing to is that this isn't just chips, but also every other commodity. The reason behind all these different industry are exactly the same. You could have swap the title for Beef, Poultry, Steel, Toilet Paper, Mask, Chips, Milk, Pencil, paper etc... I really do wish people learn a little about how supply chain works. It is the fabric of our daily life and yet very little attention has been paid to it.

Maybe you should rant a little here about why just-in-time doesn't imply no (or minimal) buffers. If not, what does it mean?

It means not carrying needless excess work-in-progress that cannot be delivered to the customer but may very well go stale or never be sold at all.

This is a balance point, not an optimization problem where you're supposed to go to the extreme.

In fact, one of the books that popularized Lean manufacturing, Eli Goldblatt's "The Goal", explicitly makes the point throughout the book that making work centers within a plant locally optimal will cause your overall operation to fail to deliver efficient outputs.

To make global efficiency improvements you have to accept that individual work centers may be inefficient. Rather, just like a developer optimizing their software program, you need to look at your entire organization and find where the actual bottlenecks are, and improve those specific bottlenecks, potentially by running "inefficient" (but non-bottlenecked) work centers to assist the teams that are the current bottleneck.

And then as you change things you'll have to be willing to address a different bottleneck.

Moreover, unavoidable variances (which the book describes in a story about a Boy Scout camping trip) means it will be potentially disastrous to 'optimize' even your critical path, just like you wouldn't want a network router to be consistently near max capacity.

> one of the books that popularized Lean manufacturing, Eli Goldblatt's "The Goal",

Turns out it's Goldratt. Very difficult to read as it is in the form of a melodrama. I suppose the point was to make it light reading, but to me it's a difficult slog through the protagonist's personal life to get to the data I want.

Thanks for the correction. You're right, the point is to make it easier to approach for business minded readers (his son even recently got someone to turn it into a graphic novel). Same tradition later picked up by Unicorn Project and Phoenix Project by Gene Kim.

> needless excess work-in-progress that cannot be delivered to the customer but may very well go stale or never be sold at all.

And what exactly is a big stockpile of raw materials and components that aren't intended to be delivered to the customer under normal circumstances?

"Inventory" (ideally), "Overhead" or "Waste". Raw materials can at least be resold in concept, but half-assembled gear (i.e. "work-in-progress") can easily end up in a sort of parallel universe where it's not easily sold without the additional investment of time and effort (either to break it back down into components or finish it for sale) but you have to pay each month to maintain it properly until it can be finished.

It's important to remember that in traditional manufacturing at the time, each work center wasn't graded on its ability to contribute to customer sales, they were graded on things like local efficiency on a per sub-assembly basis. The "customer" wasn't the end user, the "customer" was the plant manager or the PMO or some other stand-in.

The work centers were structured around functional tasks like "heat treat" or "machining" rather than an end user product, so your "high priority" customer product had to make it through multiple independent backlogs before it could be shipped, even in theory. God forbid rework would be needed! Quality issues were the job of customer service to address, not the work center that introduced the defect (as after all, it might be years before the sub assembly finally ends up in a user's hands).

So it’s like Factorio? If you accidentally fuck up your critical path you suddenly find that all your production dies.

I've studiously avoided Factorio because I don't need to get hooked on any more video games, but yes it's about thinking of your organization's workflow in terms like a critical path (Goldblatt would describe 'critical chain' in a later book), and then discovering where your bottlenecks are.

To be honest I would recommend a separate book for software practitioners, Reinertsen's Principles of Product Development Flow, because it gives you methods to look at which are more applicable to creative work where you can't easily turn your work into "production lines" to then use Lean manufacturing/JIT principles on.

Either way none of this works if you let every local work center just produce produce produce up to its peak output or most efficient point. At that point it becomes impossible to find the bottleneck because there are so many bottlenecks (including bottlenecks not on the critical path). That's a key reason why all the methods (Goldblatt, Reinertsen) focus on reducing the amount of work-in-progress and reducing batch sizes of each individual work package or work item.

Wendover has a good video on this. Give it a watch (entire channel is highly recommended)

https://youtu.be/b1JlYZQG3lI

Given you can't predict what the shock will be or its consequences, how are you exactly going to create a buffer? Creating broad redundancies would result in lots and lots of waste both damaging to a bottom line and the environment.

Also, is this "fragility" really that big of a problem? Sure, prices are going to move up 20-30% for a year or two until the supply chain issues ease, but is that worth driving up costs for decades preparing for potentially the wrong issue?

Toyota has this figured out. After the 2011 quake they built a system to identify critical, difficult to replace parts in their supply chain and size buffers for those parts. Any they aggregate demand over larger time intervals than other manufacturers. They lasted 18 months into the pandemic shortages before they ran their buffers dry.

We don't really put a price on environmental and societal damage that "just-in-time" does (thinking of truck drivers living on the highway effectively acting as the warehouse, saving the company to build an actual warehouse).

If we did, more robust models might be financially attractive too.

What is the environmental damage from making something not needed? There is a lot of energy in window glass, refined iron,and all the other parts of a car.

You could order adequately, then build and sell those cars or whatever you're making, instead of throwing the parts away. But then, you couldn't change models as swiftly, and need to sell a lot of last year's models.

I’m quite certain that replacing truck drivers with AI drivers would save the environment more than building even more warehouses.

>prices are going to move up 20-30% for a year or two

I have never ever in my whole life seen prices going down after they have gone up. Ever.

Have you not bought hardware?

I've seen every computer part come down in cost over the years even after briefly becoming expensive through my life that recent trends have been odd to me.

RAM. Harddrives. Video cards. Gas. Bitcoin. Gold.

C’mon meow. Hyperbole helps no one.

I counter this. In my country, medical mascs were sold at 100+ their regular price last spring. Now they are at their regular pre-pandemic prices.

Gas prices?

It’s supply and demand. Prices fall if there is a surplus.

oil price do that all the time: up, down...

Yes, publicly traded commodities do all the time because it means there are temporary inefficiencies in the system that can be arbitraged to turn some profit, until they converge at the “correct” price for some period of time.

But GP is correct, products typically don’t come down in pricing. Consumable goods are more elastic in a free market absent collusion, but finished honest-to-god products have a well-known price stickiness problem. (The price will come down, but rarely back to what it was.) Typically the only way that cycle is broken is when the product itself is obviated/supplanted by a replacement (eg an iPhone will never get cheaper but some new phone may come out that is cheaper and everyone moves to it).

These are all obviously generalities though.

I'm surprised you mentioned iPhones, because that's exactly Apple's strategy. They tend to keep three generations around, and each time they introduce a new iPhone, they knock down the price of the previous generations. Here's [0] an article on them reducing the price of the iPhone 8 by $150 (from $559, so a significant cut of 25%) when they introduced the iPhone 11.

[0] https://venturebeat.com/2019/09/10/apple-cuts-iphone-8-and-i...

But they tend to do if there's a temporary price shock. Think hard drives after the thailand floods or gpus during and after a cryptomining boom/bust cycle. Or used car prices after the cash for clunkers bill.

Stickiness seems to apply to MSRP which has mostly not gone up.

You don’t buy gas??

Gasoline?

> the same things that make a system "efficient" also can often be viewed as making it fragile.

Apart from the Toyota approach, discussed, Dell pioneered a different system: they forced "SLA"s on suppliers for whom Dell was a major customer. This was back in the mail order build-to-order business of the 90s; I don't really know what Dell does these days.

It worked like this: Dell took a truckload of parts. They hit the loading dock; were (in the finance sense "received"); Dell accepted the invoice (to be paid 30 days later or whenever) and the goods appeared on their balance sheet. They used them in computers they built, sent out, and then charged the customer's credit card for, getting paid immediately. When they ran low on part P they called up the vendor and said "deliver us some more right away". What was "right away"? Why pretty much immediately; the vendor had them in a truck outside Dell's facility. If they didn't, well, they stopped being Dell's vendor.

Pretty clever: the contents of those trucks were still on the vendor's balance sheet until Dell asked for them; Dell got to use them and get revenue for them long before they needed to be paid for, but basically not before they were needed. They made so much money on the float, and all at their vendors' expense.

They did the opposite too: if they had inventory ready to go but without a customer (prebuilt configs) they put it in trucks and drove the trucks away from the docs. I don't know how they got their auditors to accept it as GAAP but they did and were lauded by Business Week, the WSJ etc for their cleverness.

While impressive, this is only sustainable in a non-sustainable world. The burden that Dell unnecessary put on their suppliers have to be absorbed by the economy at some point. The single most important point that Toyota makes is stressing the importance of supply chain wide sustainability. Eg. by paying your suppliers ASAP and shortening credit lines, you should be able to get the parts cheaper than what you make of the interest.

Agreed! I hope I didn’t come off as enthusiastic about Dell’s parasitic approach.

How much of a buffer can you possibly maintain to deal with a year long economic disruption, which changes both supply (by lockdowns and factories being offline) and demand (by a shift in consumer spending mix and free government money)?

It's impractical to keep 12 months worth of supplies for manufacturing, even when you don't try to do lean manufacturing - just plain old manufacturing. It's something you need to finance, parts getting obsolete and product designs in some categories iterate every year or two.

We just need to deal with this new reality. You can do your best to avoid the worst case scenario but it is after all a statistical inevitability, so you just ride through it.

As I said on another comment here, though, this is going to have long standing effects on the other end of the disruption. We're going to be left with huge over capacity, unclaimed stocks and companies that couldn't produce parts will have their customers design out their parts for whatever else they can get their hands on. This is a golden opportunity for the smaller players in the industry. How giants like ST, TI and NXP will deal with this long term - I'm not quite sure. Many companies in 2021 and possibly well into 2022 who would normally design their products with TI parts will not do that because they can't even get samples, let along ramp up production.

Toyota (famous for lean manufacturing) built redundancy in to it’s supply chain and only recently has run in to chip shortages. It may be hard, but it’s certainly possible.

0: How Toyota Steered Clear of the Chip Shortage Mess

That article was back in April. Then, in August, "Japan’s largest car maker said Thursday it was cutting production in the country by 40% in September because of a shortage of semiconductors. The company declined to say whether it would shut down plants outside of Japan."

Most of automotive does not need AMSL 10nm and below fabs. They need robust 200nm to 300nm chips.

This may yield car redesigns with less touchscreen dependence, simpler electronics in the essential systems, and an "infotainment" system that's less integrated with the vehicle and can be added or replaced later. Already, some new cars have shipped with a blank plate in place of the entertainment system.

There's a mindset that electric cars have to be more complicated. This is strange, because managing the batteries and motors is far simpler than managing an IC engine.

[1] https://www.programbusiness.com/news/toyota-succumbs-chip-sh...

>This may yield car redesigns with less touchscreen dependence, simpler electronics in the essential systems, and an "infotainment" system that's less integrated with the vehicle and can be added or replaced later. Already, some new cars have shipped with a blank plate in place of the entertainment system.

I'd personally see that as a major improvement.

The highly integrated infotainment systems are going to make a lot of these vehicles feel obsolete before their time.

The shortages are hitting ECUs and body controllers, it isn't just the fancy cabin electronics.

The car industry really just needs to standardize a connector (USB-C?) and a protocol for an external device to drive the dashboard touchscreen and any relevant physical buttons. Like Android Auto and Apple CarPlay, but open and universal.

They already have a standard network bus connecting all the things together. There's even a standard debug port that exposes it, somewhere near the driver seat.

https://en.wikipedia.org/wiki/CAN_bus

https://en.wikipedia.org/wiki/On-board_diagnostics

Yeah, but can that drive the display? My impression was that it doesn't have the bandwidth for that. Android Auto requires USB-C, although I'm not quite sure what the actual protocol on the wire is (but I wouldn't be surprised if it's the same as for USB-C displays in general, just with some extensions).

Most of the pieces are already there. The only problem is that they're proprietary.

> This is a golden opportunity for the smaller players in the industry. How giants like ST, TI and NXP will deal with this long term - I'm not quite sure. Many companies in 2021 and possibly well into 2022 who would normally design their products with TI parts will not do that because they can't even get samples, let along ramp up production.

I'm currently designing with STM32 parts and bought the last hundred available anywhere in the world, but that's only enough for a field trial, not for production. The only ones left in any quantity at at all are the insanely high pin count BGAs and chip-scale packages... no way.

I looked at TI, NXP and Microchip parts too. There's plenty of older 8-bit microcontrollers available, so if you want an AVR or similar I think you'll be fine because they're on older processes that aren't in as much demand.

If there really is an alternative from a "smaller player", I'm all ears.

MCUs are more of an issue and of course migrating between different families is a huge headache. However the problem extends to other active components too - regulators, interface ICs, sensors, anything really - and there are opportunities to work with the smaller manufacturers there.

Fair enough, although I'm not sure this is true at our scale or our product type. When designing products for a 20 year industrial product life you really don't want to gamble on chips staying in production.

Definitely shortages on regulators as well, although that's outside my area.

> It's impractical to keep 12 months worth of supplies for manufacturing

They didn't need to. The initial disruption was only a few weeks. The problem is that once companies saw that there would be a shortage, they started buying up all the supplies they could to rapidly build up buffers, which increases the demand for those supplies worsening the shortage, which in turn means even larger buffers are required. It's a vicious cycle that all could have been avoided.

I don’t think it would have been without the pandemic, mostly because remote work and education has dramatically increased demand for PCs. There is not a ton of excess capacity especially at the wafer fab level, so PCs running 25% higher than expected was enough to push the industry into shortage. There still would have been a cycle (driven by inventory stocking / restocking), but the shortage would not have been nearly as acute.

Funny enough I believe it's not even the real consumer/end user demand.

It's just companies anticipating or seeing the same thing (an increase in consumer PC demand) and all releasing their own shitbox/AIO/desktop/laptop.

I bet most of them are sitting in warehouses unsold.

Nope, not just PCs, just electronics in general.

The pandemic allowed tens of millions of well-off western office workers to move out of tiny urban flats into more spacious housing. While having to spend a lot of time in their new homes, which was previously in ”public” places (eg restaurants, bars, what have you)

Now, what to do with all that space? A giant tv, washer and dryer, a nice stereo set, new induction stove, the list goes on and on…

PC demand also increased obviously, but probably bitcoin miners caused the most demand for PC hardware components

The automotive industry scaling back and cancelling orders in the beginning of the pandemic and then being shocked when their orders came back, increased even and the chip manufacturers couldn't just step up was a serious factor too.

That definitely made the inventory cycle more extreme

I'd rather have fragile efficiency over the long run than robust inefficiency to smooth over the rare events like covid.

I mean would we all like the base model Ford Focus to cost $45,000 simply so that they'd be available on the lot during a once in a century pandemic?

Also small-scale fragility makes for large-scale anti-fragility if the broken things are allowed to break.

Toyota actually had a large buffer of chip inventory which allowed them to maintain production longer than most other auto manufacturers. But it wasn't enough and they're finally cutting production. There are limits to buffering.

The pandemic caused companies to buy chips like people were buying toilet paper.

If every company suddenly decided that they now want to keep a years stock of their important parts rather than a week, the demand just went up 52x right away. It's no surprise even basic parts are sold out for the next year at least.

All that does is pull through a year of extra demand. (I’d be more inclined to call it a doubling of annual demand rather than a 52x.)

On a weekly basis that would be a one time 52x increase in incoming orders. On a yearly basis it would be a one time doubling. But that is just semantics.

What is important to discuss is that no one was paying for a extra idling factory before, and now suddenly it is needed for one year to satisfy every customer wanting to keep a one year local buffer.

One also have to take the effect of a one year buffer for the whole supply chain into account. The demand for the manufactured good also increased due to their customers wanting to keep the same one year of buffer, so the one year buffer (based of your previous demand) would be consumed solely by your customers buffer and your orders would have to be larger than one year of inventory for you to keep one year of inventory after the demand spike.

In the end it is easy to imagine that new buffers would lead to a huge scarcity, especially for the longer supply chains.

It is alarming that people don’t understand that this is just another way to say inflation. The government prefers to call it a shortage.

This is a good point. Perhaps it was better for the problem to be revealed during this pandemic rather than something worse like a war.

Fun fact, in the early days of lean manufacturing (of which "just in time" is a part), the practitioners often referred to it as the "fragile production system" for precisely the reasons you mention here. So yes, lean manufacturing is brutally efficient, but incredibly fragile.

I’m fond of the idea that efficiency can also be overfitting.

Interesting analogy!

It got me thinking that a buffer is really just operational insurance, and the cost is "efficiency". I wonder if it's the same average efficiency. If so, my knee-jerk though is that I much prefer the smooth buffered model as opposed to the spiky unbuffered model.

Unfortunately, large buffers lead to more spiky behaviour than well-kempt ones.

One of the reasons is they hide feedback about downstream capacity problems. (See "bufferbloat" for this in internet terms, Principles of Product Development Flow for it in terms of creative work, and any book on lean manufacturing for it in terms of repetitive work.

Scary part same applies to clouds. People spend a ton of effort making things multi-region and multi-cloud yet there is not enouph spare capacity to absorbs AWS US East going out

> "Just-in-time" means "no buffer".

No, that's a degenerate fragile microoptimization inspired by seeing the short-term cost savings of just-in-time and going all in on those savings without understanding just-in-time.

What “just-in-time” actually means is dynamically exactly enough buffering as is cost effective given current expectations of forward conditions considering the likelihood of supply irregularities, cost of production interruption, costs of inventory, etc.

Yeah, the original comment is way off. I worked for a major electronics manufacturing company helping with demand driven MRP (DDMRP, MRP=material requirements planning) and obsolescence (DMSMS - diminishing manufacturing sources and material shortages).

I can definitively confirm that just-in-time is about trying to optimize the amount and strategic positioning of buffer needed to support the "right amount of production" to meet anticipated and actual demand, exactly as you describe.

Now, doing that can be rather difficult in the case of black swan events. But the goal is as you describe.

> dynamically exactly enough buffering as is cost effective given current expectations of forward conditions considering the likelihood of supply irregularities, cost of production interruption, costs of inventory, etc.

"Exactly enough" means "no buffer". You are simply mislabelling inventory as buffer. What people needed was buffer beyond what is cost effective given current expectations because something unexpected happened causing improbable supply irregularities.

> "Exactly enough" means "no buffer".

“Exactly enough” means nothing by itself.

“Exactly enough for X” means different things based on X.

“Exactly enough for current production” means no buffer.

“Exactly enough buffering as is cost effective given current expectations of forward conditions considering the likelihood of supply irregularities, cost of production interruption, costs of inventory, etc.” does not generally mean “no buffering” unless conditions are expected to be extremely stable, or response time to changing conditions from identification to new deliveries are near instant and there is negligible risk of that changing, or there is negligible cost (including opportunity cost) to production interruption.

No, it is the definition of no buffer. Buffer is the excess beyond enough.

If you expect some supply irregularities and you don't have enough to compensate for them, that is less than enough - a shortage. Having enough for normal operations (which includes forseeable irregularities) is no buffer. A buffer is what you have in excess of that.

not only is there a shock to the system, every (sub)system has had shocks.

I was talking to someone who was rebuilding a home, and the price of plywood went from $8/sheet to > $100/sheet

Another person said that the company that makes 75% of the electrical panels had stopped production.

those are just two pieces of the bigger puzzle, but since many if not all pieces have similar shocks, it would really surprise me if capacity could ever increase to meet demand.

> "Just-in-time" means "no buffer". We have, for decades, been doing as much as possible to make all our economic systems as efficient as possible, which is to say with no extra capacity.

This is true, but I don't see an antidote. The government couldn't just tell businesses to "be inefficient"!

There's a really good (free) interview on the Verge with Dr. Willy Shih who explains whats going on:

https://www.theverge.com/2021/8/31/22648372/willy-shih-chip-...

TLDR: it essentially was the pandemic: cancelling orders -> redirecting supply -> too late to uncancel -> low supply + competitive hoarding. Plus a 10 year $150B head start.

Very little margin and too much optimization/efficiency is bad for resilience. Couple that with private equity backed near entire market leverage monopolies/duopolies/oligopolies that control necessary supply and you have trouble.

HBS is even realizing too much optimization/efficiency is a bad thing. The slack/margin is squeezed out and with that, an ability to change vectors quickly. It is the large company/startup agility difference with the added weight of physical/expensive manufacturing.

The High Price of Efficiency, Our Obsession with Efficiency Is Destroying Our Resilience [1]

> Superefficient businesses create the potential for social disorder.

> A superefficient dominant model elevates the risk of catastrophic failure.

> *If a system is highly efficient, odds are that efficient players will game it.*

Over efficiency and supply chain concentration single point(s) of failure (China/Asia with chips especially across Hong Kong and Taiwan that make most of the chips) caused supply chain disruptions, some hoarding is also going on with chips. [2]

The HBA MBA-itis and Chicago style thinking of excessive efficiency made the players that can compete larger and less of them, that takes away resilience and leads the market to gaming. The market has thus been gamed as the players that control it are more powerful in terms of market leverage. When you give up diversification and flexibility you get leverage and added weight to any needed quick changes as the large players attempt to control the market. [3]

US is now building up silicon supply chains again [4][5], it was much bigger in the 90s/early 00s with Intel/Motorola/etc additionally, but business leaders allowed a concentration to happen and it led to market leverage.

Hopefully that same mistake is not made in the future. It will take time to build up diversification of market leverage in terms of chips for availability.

This chip shortage, and all the supply chain problems during the pandemic as well, will hopefully introduce more wisdom and knowledge into business institutions that just because things are ok while being overly super efficient, that is almost a bigger risk than higher prices/costs. Competition is a leverage reducer. Margin is a softer ride even if the profit margins aren't as big.

[1] https://hbr.org/2019/01/the-high-price-of-efficiency

[2] https://finance.yahoo.com/news/china-stockpiles-chips-chip-m...

[3] https://www.bloomberg.com/news/articles/2019-12-03/china-sto...

[4] https://www.cnbc.com/2021/03/23/intel-is-spending-20-billion...

[5] https://www.cnbc.com/2020/05/15/tsmc-to-build-us-chip-factor...

In capitalism there is a very simple solution for shortages: price increases. People still haven't realized that the solution is just to increase prices until only enough people afford the products, so there won't be shortages anymore.

This is exactly what's happening. Chips in high-demand have gotten very expensive. https://blog.flipperzero.one/manufacturing-and-shipping-upda...

Few people consider a deflating economy a "solution".

The point is that artificially keeping prices low does not improve the supply. If anything, it has the opposite effect.

By raising prices on the high demand stuff, you send a signal to the people that can get by with alternative stuff that now is a good time to do so.

So, like, highly hypothetically, if I couldn't find a calculator for a good price, I'm in a good seat to whip out my slide rule. This gives me an edge over the people who have to pay high prices for calculators, and simultaneously smooths out demand. What's good for me is also good for the people who really need calculators.

Yeah, that's high school macro. I didn't need an explanation, but it seems that you do.

Optimally distributing fewer goods isn't a solution to the problem. The problem is a shortage.

I still don't see how reducing demand isn't a solution to excess demand. I'm honestly confused and would be happy if you could take the time to explain.

How is that a solution? What happens when the good in question is food?

It isn't a solution, it is a rejection of the problem. There isn't such a thing as a 'chip shortage', because that implies some natural number of chips that should exist.

That number isn't a constant, it is a function of supply and demand. Therefore there is no such thing as a chip shortage. We'd maybe like prices to be higher or lower depending on our perspective.

> What happens when the good in question is food?

If there are 10 mouths and 9 meals then there is a food shortage. But that is a biological observation, not an economic one.

> That number isn't a constant, it is a function of supply and demand. Therefore there is no such thing as a chip shortage.

It does take some time for prices to adjust, and if the supply and/or demand curves shift very quickly, there absolutely can be a temporary shortage. I think that's what we're going through now.

We've been seeing 'shortages' since 2016; I remember there being a lot of trouble sourcing a good GPU. This is probably some deeper problem than a short term issue.

It isn't like COVID is a surprise any more, it has been with us for nearly 2 years.

When we start hitting food shortages down the line from climate change induced damage, you will get to witness the alternative: powerful countries confiscating supplies for their rich citizens (at stable prices to stave off rebellion of course).

> What happens when the good in question is food?

Prices go up on the food most people prefer to eat, first. People who cannot afford the foods they're accustomed to eating will switch to lower-priced foods they would never have considered otherwise.

That's largely how tomatoes, potatoes, and more came into use... Desperate, hungry peasants.

When food suppliers see the increase in price, they will work hard to squeeze more yield out of their existing supplies, and as soon as possible will invest in producing more (typically for next year). Others who are capable of producing food, but normally produce something else due to economics, will see the equation change in favor of food and switch their output.

Famines are not caused or encouraged by supply/demand economics. It is vastly more common for communist governments to cause them.

I don’t have a problem with supply/demand economics, and certainly the market exerts pressure to economize like you say. I just take issue with the There’s no such thing as a shortage perspective.

There is such thing as a shortage, but persistent shortages are only possible due to government meddling.

Famine is a terrifying word to brandish so flippantly. God help us we don't find ourselves in a JIT situation with our staple crops. A famine would make COVID-19 look like a walk in the park on a Sunday afternoon.

There's a really good (free) interview on the Verge with Dr. Willy Shih who explains whats going on:

https://www.theverge.com/2021/8/31/22648372/willy-shih-chip-...

TLDR: it essentially was the pandemic: cancelling orders -> redirecting supply -> too late to uncancel + competitive hoarding.

Reply on news.ycombinator.com