Hacker News Reader

Comment by alexose | Hacker News Reader

In my experience, it's not that companies are choosing any one strategy over another. It's more that companies have no idea what their supply chain actually looks like. Only that the widgets show up at their loading dock every week, and that the shipments have always been reliable in the past, so that there's no reason to look any further.

There are often strong disincentives, as well. Nike, for example, got themselves into trouble when parts of their supply chain became public. Many more companies regard their unique supply chain as a competitive advantage. So, even if you should take up the time, effort, and expense to research your own supply chain, there's a high likelihood that your upstream vendors will have no interest in sharing their suppliers with you.

There are exceptions to this. Reportedly Apple has excellent supply chain visibility, but they've achieved it at great effort.

(I'm by no means an expert, but I used to work for sourcemap.com, an MIT startup based around the idea of understanding supply chains end-to-end.)

View on HN

This is it 100%. I work at a major financial institution and we invest ridiculous amounts of money, effort and, frankly, delay, in understanding our supply chain, its security posture and risk management model. It's a unbelievably gargantuan effort and if it weren't for the scary shit that we find all the time one could easily argue it's a waste.

What problems have you encountered that others should avoid?

They run the gamut. Lack of a viable risk management program, lack of resiliency planning and testing, poor operational and change management controls, poor visibility and response planning for security events, poor SDLC security, etc. etc. The requirements vary based on our exposure.

Over the years, I have run into:

- Counterfeit components, like memory out of spec that creates scaled failures.

- Fake products from the channel.

- Fundamental mistruths with regard to how services are delivered.

- Poor operational practices for maintaining SLA, especially for off-shored services. Those ops folks will move mountains to meet the minimum standard of "up", but will allow problems to fester.

- Fraud. Fake on-site employees sending work back to a boiler room with people actually doing the work.

- Fraud. Named users actually being fake people sharing a credential. In one case, a colleague in a manufacturing case discovered this when he accidentally received a link to an open video stream of a MFA token.

- Fraud. Stuffing fake or inferior products into orders. Example: Order new laptops, find a bunch of refurbs in the pallets.

Basically, people suck and if you primarily make purchasing decisions based on price, and you care about what you are buying, you need to treat the counter-party as a potentially hostile actor and assume that you are going to be ripped off.

I read when having bicycle frames made in Taiwan you get the best price/quality ratio if you have a permanent onsite observer in the factory. That guy doesn't really do anything but they wont cut corners if he is around.

You get what you pay for. If you aren't paying for the guy to be the observer there's no one to observe.

You don't get what you pay for.

But generally value derives from multiple independent credible verifiable audit mechanisms.

Or from reputation.

A company with a reputation for quality can charge more, because you don't need observers and can just rely on them. And they want to deliver on the reputation, so that they can keep milking it.

The value of trust.

JS Mill has some interesting observations going back a ways.

Auditing is still recommended.

Definitely, auditing is one way to speed up the process.

Ignoring auditing for a moment:

The trustworthy reputation is quicker to build, if the quality of the product is easy to ascertain quickly.

But if you are a company in a sector where that's not possible, the reputation for quality becomes even more valuable.

And you can potentially ride it out for a long time in an exit scam. https://en.wikipedia.org/wiki/Exit_scam

> You don't get what you pay for.

Yes you do. You get exactly what you pay for. If you pay for a dozen of German inspectors who think Chinese factory produces junk that is outside the spec, a dozen of German inspectors with that attitude are going to do inspections. If you do no want to pay those German inspectors, you are not going to get those inspections.

In most of the cases companies do not want to pay for it. Apple manufactures amazing iPhones in China. Apple iPhones easily last 5-6 years. Blu manufactures crappy Android phones. They barely last a year. Apple's iphones cost more to manufacture than Blu phones sell for.

You don’t if you just throw money at it.

But when you build ongoing relationships, you get benefits from not pushing for the bottom dollar at all costs.

> Fraud. Named users actually being fake people sharing a credential. In one case, a colleague in a manufacturing case discovered this when he accidentally received a link to an open video stream of a MFA token.

This is a classic bait and switch in many online work platforms. The person who interviews for your open contract/role is not the person who does the work.

> It's a unbelievably gargantuan effort

Also applies to software supply chains:

https://www.platformsecuritysummit.com/2019/speaker/sherman/

> Today’s software is largely assembled rather than written, and most of the assembly comes from open source components. The creation of components and their inclusion into applications creates a “supply chain” just like in conventional manufacturing. While physical supply chains have well established chains-of-custody to establish properties like refrigeration maintenance, authenticity or spoilage avoidance, the software supply chain is very much a wild, wild west, filled with vulnerabilities that can be (and are) inadvertently inserted into applications. As supply chain risk and mitigations are being explored by government and academia, a larger attack surface is being uncovered that needs to be addressed.

We do that too. :)

What have you found? (Omitting any details that would be confidential of course)

I mean, imagine you have a 3 year contract with a supplier, and when you go to renew they get cagey and seem to be giving you the runaround. Later you find out that it's because they already signed a contract with one of your competitors that begins the day after your old contract expires.

We laugh about golf and Vegas visits, but it's really hard to stab someone in the back when you're used to looking them in the eye. If they're just some suit you meet once every three years? Fuck 'em.

> the scary shit that we find all the time

Curious to hear examples if you can share.

My dad used to work for a big tech OEM and retired a few years back. He had years of supply chain flows, contacts, locations, and more for his old company in notes, his head, and Visio and was hired through layoffs for that knowledge. I think in like 2006 he was picked up by a manager who had heard him speak up in a meeting years before to explain a step in a chain and knew he knew what the company was able to do.

He could jump into the flow of parts and information for a specific printer model from 10 years ago. He knew the reasons for each steps like import tax avoidance, cost/benefit analysis of using one vendor/shipper/etc, or even that they used to have a specific engineer at that one factory who was amazing at QA and tolerance checking or something like that. When teams were behind due to supply chain issues, he would jump in to analyze their process. Sometimes that meant combining it with ongoing flows like "This factory here makes a part like that for product X, I'll get you a sample so you can adapt it into your product to rework."

He had immense company value and history committed to memory that he tried to pass before leaving as teams moved to India or China. But I doubt anyone at a decision making level was aware.

Thanks for the write-up!

> He knew the reasons for each steps like import tax avoidance, cost/benefit analysis of using one vendor/shipper/etc, or even that they used to have a specific engineer at that one factory who was amazing at QA and tolerance checking or something like that.

Excellent example. Some of those you wouldn't be allowed to document in writing, even if you wanted to.

They have a complete idea, its just that the globalized option is cheaper, even fractions of pennies cheaper, and is directly measured in the profit margin, while the risk, if it shows up at all, is buried in the footnote of a 10-Q or annual report.

Ideas like regionalization and flexible manufacturing are great ideas, in most cases, the capital machinery of factories are relatively easy to replicate and setup (harder than software, but much easier now than any time in the past). However, the economic and accounting system needs to balance out the global employee wage differentials for it to gain any traction.

Tariffs have been hated on economically, in part because the analysis values the generic profit creation over the risk; but it seems like one way to balance wage differentials to encourage de-risking our supply chain. I'm sure there are other maybe better methods?

They could introduce some new GAAP rules like...

Add 10% of the value of all inventory sourced outside North America as a "risk liability". Same for 10% of yearly labor that's outsourced.

Basically find ways to make short term thinking visible in their financial reports, since that's what's measured and rewarded.

Not really an accounting issue & messing with those numbers to highlight a risk is just asking for trouble.

Companies are required to disclose significant risks, but the quality of that disclosure...varies.

I suppose this is a useful way to use tariffs. Make production abroad slightly more expensive, or significantly more expensive for vital goods.

Base those premiums on verified actuarially based underwriting risk and you might be on to something.

Companies are free to add them to their accounting already.

Mostly these kinds of risk factors are mentioned verbally in their reports at the moment.

Entities tend strongly to externalise risks unless compelled to do otherwise.

Insurance obligations -- auto, health, life, C&L, bonding, mortgage and title -- are generally required by or offered via third. parties. Sometimes governments, but often creditors, counterparties, employers, or other entities.

There are true externalities in a car accident.

If you have to shut down your business, that's mostly impacting your business in a very direct way.

that's also where i've landed on the tariffs issue. theoretically a bad idea, but has practical uses beyond purely economic considerations.

good tariff policy is hard to achieve in divisive political environments however. for instance, in many cases, you'd want them to go away dynamically as conditions improve, but lawmakers hate giving away their levers of power.

I am convinced a low tariff of ~3% on all goods and services to every country is a good idea. But that’s never what happens, you get a huge mix of zero to insanity based on which industries have political power. As such a blanket ban on tariffs became a rallying cry.

Food is often used as the counter example, but IMO domestic production is less valuable than a multi year stockpile.

why 3%? to overcome inflation? and why on everything instead of grossly disadvantaged industries? i haven't spent much time thinking about such details, so curious about the rationale.

grossly disadvantaged industries = industries labeled as grossly disadvantaged = industries that managed to talk politicians in to giving them a huge advantage

Not the person you're asking, but I could see doing it to "internalize" the risk of rare events like what we're seeing right now. Basically it applies an incentive towards regionalization over globalization, that can make the system more resilient to distant events.

It’s extremely expensive to grow bananas’s in Canada, so the correct response is to import them not add a huge subsidy. The point is to encourage domestic production, but not to the point of stupidity.

Further, unlike a direct subsidy it’s adding tax revenue.

> domestic production is less valuable than a multi year stockpile

nobody is going to stockpile food or consumer goods when you're demonized for charging more for them when the demand goes up. there's no way to do anything but lose money on it.

Governments used to maintain grain stockpiles in case of famine. A modern equivalent could be run for vastly less than current farm subsidies. Especially if stocks where rotated and mostly refilled in years of massive surplus.

PS: At the extreme end Honey can be stored for thousands of years at room temperature.

that's why stockpiles should be kept by government. they don't need to make money on it. should be easy enough, have 6 months worth in a warehouse, ship the oldest out the front of the warehouse to the supply chain while topping up with new product at the back. rotating it like this acts like a buffer,rather than something that is only accessed in dire emergencies and would be out of date or less effective due to age. the relatively small costs of a warehouse would be negligible

Such a tariff could be made more economically neutral if the tariff proceeds were distributed to citizens as a dividend too.

You'd still get deadweight losses.

Easier to image with a 1,000,000% tariff: nobody would actually pay the tariff, so there's no dividend to distribute. Yet, you get an economic impact.

no one is suggesting that tariff level though so it's a pretty much unrealistic value to work throuh.

Well, that's the tariff level that corresponds to a ban on imports, basically.

You still get deadweight losses from lower tariff levels.

> Tariffs have been hated on economically, in part because the analysis values the generic profit creation over the risk; but it seems like one way to balance wage differentials to encourage de-risking our supply chain. I'm sure there are other maybe better methods?

What do you mean by 'balancing wage differentials'?

Global trade is on of the best ways to balance wage differentials. That's how China had crazy catch-up growth in the last few decades.

(Another great way to help poor people get more productive and thus richer is open borders.)

If downstream customers want to de-risk, they should pay for it. Either by setting up their supply chains differently, or by outright purchasing insurance, etc.

If they choose not to engage in this expense, perhaps they are right, and just bearing the risk is overall a better trade-off? Who are we to tell other people what level of risk aversion they should have?

Of course, once the calamity hits everyone wishes they would have bought prepared or bought insurance. But that doesn't mean we should all go out and make our roofs asteroid impact proof, or never ever go out because we might break a leg.

Really the risk sounds more like a job for insurance and warranties tied to product specs than tarrifs.

I don't see a connection between tarrifs and risk in a supply chain. Infamously US autos which refused to adopt Tailorism or other cousin methodologies for reliability wound up being expensive crap compared to Japanese car makers. It is a peripheral variable at best or an irrelevant one at worst.

Tarriffs as a solutio resemble opportunistic timing to not let a crisis go to waste rather than addressing a root cause.

My current position is involved in Supply Chain, and this hits the nail on the head. There are so many different departments with fragmented knowledge of the supply chain that isn't shared or sometimes even documented. One of the senior people trying to fix this specifically mentioned Walmart as an example of having their supply chain being a competitive advantage because they focused on it so much. I think Apple was also one of the ones we discussed.

WalMart turned their supply chain into a competitive advantage 30-40 years ago. I saw it as an employee first, their inventory control in the store was ridiculous back in the 80's. Then I saw it again working with an EDI shop (Sterling Commerce) in the 90's. Everything was electronic. We'd get these little mom and pop shops calling in trying to get their bisync modems working so they could upload their ANSI X12 810's.

The only reason there's any question who's the richest person in the world is that Sam Walton is dead.

Sterling Commerce - that’s a name I haven’t heard in a •long• time. Do you by any chance remember a company called “EDES” (Electronic Data Exchange Service)?

There's a tickle in a neuron but it could be self-induced. What was the relationship?

They bought EDES in 1998 - it was a company I’d done a bit of contract work for.

(Trivia) EDES was one of the few users of RPL (RapidGen Programming Language) - Decision Table based, and running on PDP11s and later MicroVaxs.

https://www.cbronline.com/news/sterling_commerce_buys_into_u...

Hmmm I was there in 98, probably just a hole in my memory.

Before he was CEO, Tim Cook was handling logistics for Apple, wasn't he?

I haven't been at many companies where a customer paid us money specifically to build up capacity for them first. Most of the time I've had to ramp up, we had to pay for it out of investment dollars or scraped out of contracts specifically for deliverables. Even when we special-built a feature for a customer, we still had to find a way to generalize it for others to turn a profit on it, and that was out of pocket.

Apple has apparently paid a number of manufactures to get good at something in exchange for a short lockup period and a permanent discount on orders. Which I think is why when Intel tried to make their first direct Macbook competitor, it turned out their margins were razor thin compared to Apple's.

>but they've achieved it at great effort.

And enormous expense. You shouldn't have to have Apple money to establish visibility into, and security of, your supply chain. I mean, do you really want to be down in the weeds trying to make sure the coltan isn't essentially coming from Rwandan warlords? Apple can afford that, most people can't.

Just adopt the opposite point of view:

You could, by default, have lots of visibility in your supply chain. Or you could get a substantial discount in return for giving up on that visibility.

You are arguing that the discount should be smaller?

> based around the idea of understanding supply chains end-to-end

if you know more companies / projects in this area, I would love to know. It's of course wiser and easier to do b2b in this space, but there's also the dual problem from the consumer side of getting to know the supply chain.

It would be so great to have a portal where I can scan a barcode and get to know the whole chain of production.

Is it possible some legislation requiring resilience and local-sourcing could be possible in this climate? - esp. given the political antipathy to China these days.

I sure get the feel that it would be, but I hope we get moving on that ASAP before this is over, because I suspect minds have only been temporarily opened on this matter. Before too long I suspect our old heuristics will reassert their dominance over logic on topics that are in any way associated with the identity of the person.

I am going to bookmark this thread in anticipation of this phenomenon.

"Nobody could have foreseen what would happen when the world’s second-largest economy went offline and completely shut down external logistics connections."

Exactly what they said about the housing bubble. Honestly, how many people on HN and elsewhere couldn't anticipate this? I think the main thing that has caught everyone by surprise is just how comically unprepared we are for this. And the overwhelming sentiment I've been getting from the memeplex is that this situation is entirely the fault of Trump, when he fired Obama's pandemic team. Not only is this not actually technically true, the idea or insinuation (dog-whistling) that the extension of that team would have mitigated this whole thing seems to be quite unlikely, to put it mildly. Oh yes, of course having this team in place would have improved things - but by how much, and under what counter-factual scenarios?

These are the discussions we need to have, but I don't think Western culture as it is is capable of having them. We need to wake up and realize the human ego is the root of all of these problems.

This is arguing over whether the iceberg sank the Titanic or the decision to remove bulkheads to make room for the grand ballroom.

It is of course both. Global supply chains and JIT delivery have made us more fragile. But China reacted well - locked down a region of 60 M people - then sent in the resources we of 1.5 BN to stop the spread to the rest of the country. Now they are basically hanging on till a vacccine.

USA pandemic teams probably had similar plans - shutdown California for example - in other words steering away from the worst of the icebergs.

Is China's strategy "best"? Hard call to make - especially now. But it beats many many alternatives.

There is a much larger more fundamental problem I'm referring to. It's right in front of us, and yet invisible, for lack of a better term. Kind of like this virus ironically.

>the overwhelming sentiment I've been getting from the memeplex is that this situation is entirely the fault of Trump, when he fired Obama's pandemic team.

Also Trump 'dependence on China is a huge problem'.

Of course, but if we're seeking the truth, Donald Trump isn't a good place to start.

Dependence on <x> (where <x> could be China or many other things) has been an obvious risk across numerous dimensions for a very long time now. Many of these ideas have been discussed here, although not in a manner befitting for the gravity of the matter.

That even on HN we are often unable to engage in unbiased, purely rational discussion has been another big systemic risk that I have been banging the drum about for months. It makes a person start to wonder: what will it take for people to wake up? Maybe it isn't even possible?

What would that legislation accomplish?

Companies already bear the costs and benefits of their resilience. So it's already internalised.

You could, however, pass a law that specifically allowed people to charge whatever prices they can get a customer to agree to pay; and ban all laws outlawing such pricing.

That way you would give companies an incentive to be especially resilient in the face of disruptions.

Right now, you are mostly not allowed to sell hand sanitiser for thousand dollars a bottle. So the incentives to become so resilient that you are still producing when everyone else has shut down and you can charge these kinds of prices, is kind of dulled.

There are regulatory and IP issues to consider as well. Redundancy is not necessarily as straightforward as it seems.

>Only that the widgets show up at their loading dock every week, and that the shipments have always been reliable in the past, so that there's no reason to look any further.

Just want to add that this same story is playing out at the individual level right now, as well. Milk comes from supermarket shelves, not cows.

Arguably a benefit of substantial tariffs is that it gets that message through (i.e., your supply chain may not be nearly as reliable as you think).

My family is in manufacturing and everyone is acutely aware of their supply chain, and there are whole departments in the companies dedicated to sourcing and testing it all.

Supply chain management is like 50%+ of all manufacturing industry. I think you must be responding to the idea of like a hospital not knowing where their masks come from- and I don't think thats the focus of the article.

Does your company monitor its entire supply chain, or its supply links?

If your supplier's supplier would change their supplier of something, is that something your company would know, or are they at liberty to source their materials and parts wherever and however they want and not tell anyone?

Carefully sourcing parts is one thing, tracking the whole supply chain of where your suppliers get their materials is another.

> more that companies have no idea what their supply chain actually looks like.

Except Walmart

It's called the invisible hand the free market for good reason.

Reply on news.ycombinator.com