Supply Chain Security Is the Whole Enchilada, but Who’s Willing to Pay for It? | Hacker News Reader

Supply Chain Security Is the Whole Enchilada, but Who’s Willing to Pay for It?

krebsonsecurity.com

148 pointsjohnshades85 comments

I’m not a protectionist, but we need to start looking USA-based semiconductor supply capacity as a national security imperative. We should never lose our ability to manufacture these critical components, even it means policies that might be viewed as “protectionist”.

I feel the same way in europe, and it applies to much simpler things as well: clothing and shoes. It's incredibly hard to find shoes manufactured locally, or at least regionally.

I don't think these are the kind of items the parent was referring to. Does it really matter if your shoes were manufactured locally?

It could in the event of a trade war, which seems increasingly likely for many countries.

Lack of local manufacturing isn't really important in a world of global commerce, but when governments start using their manufacturing as a weapon in dimplomatic disputes it becomes important.

You could argue that about literally anything though, from clothes to bananas - but it's simply not practical for every country to manufacture and produce everything it needs.

Also, clothing and shoes (mentioned in the comment I responded to) are easy to produce without massive capital outlay, and the raw materials are readily available from several countries. I think the local manufacturing argument makes more sense with regard to items that are seen as critical, and which would be impractical to obtain from elsewhere - such as motherboards, processors and the like.

there's a trade-off for sure, but at the moment some countries make a very small percentage of their goods locally, which leaves them more vulnerable to tariffs in a trade war.

Whilst raw materials could be easily available, there is a ramp-up time to move any industry locally, both in terms of creating the manufacturing infrastructure, and in terms of acquiring the skills needed locally.

Not as much as something like semiconductors. If everyone in the world literally stopped selling us shoes, we could likely figure out something passable locally in a few months. Spinning up the tooling and know how for advanced manufacturing of critical technology goods is a years long proposition.

No, but it matters that you can't get any because some adversary decides to cut your supply. Protecting your supply by force is obviously easier if the source is within the borders of your country.

This is the basic reasoning of every "national security" argument about having domestic production.

The parent is merely stating that basic necessities are also important for the well-being of the nation, it's not just about the IC for ballistic missiles or the 3G circuits that allows Trump to always be on twitter.

I get what the sentiment was, but it could be applied to literally anything.

I don't think it matters for many items, such as clothing and shoes, because they are cheap to make and readily available from several countries. So even if China randomly decided that they wouldn't ship any clothes to the US anymore (which would hurt them just as much, so is unlikely), you can just as easily obtain them from Bangladesh or Vietnam, or indeed make them yourself.

Something’s odd: in the US midrange clothing is from europe. Be they shoes, sweaters (pullovers) shirts, etc. Japan enjoys the same cachet. We also have domestic brands which enjoy similar preference.

While they tend to cost more, for the most part, unless you follow the vagaries of fashion, they are worth the extra money.

They're not just worth it in the sense that the difference is measurable, they're also often worth it in the sense that they're more economical over time.

I bought a belt from an American company after researching different leathers. It's lasting me years at this point. I'm sure it will last decades. It's crazy how much money I wasted before on belts that were a third of the price.

What’s the company?

And which leather did you settle on ?

(I also posted the belt above.)

Both top grain and full grain have their uses, so it depends on whether you want some character in the item as it ages or not. Generally speaking if you don't know what to do, go with full grain. It's expensive, but it's great.

The only case where I'd recommend top grain over full grain is if you're buying something where the leather needs to stay in the background. Think a multi-material item like a bag that has both a bright cloth part and a muted leather part then you'll want top grain for the muted leather because it will keep a more consistent style over time so it won't split your attention when looking at it in the future.

I have found no other use for the other types of leather. I'm far from an expert, but from my research and personal experience I really think that the rest of the leather out there is complete garbage.

Sorry for the delay, I was at thanksgiving. The belt I bought was this one:

https://saddlebackleather.com/full-grain-leather-belt

Make sure you double check sizing. It's not the same as jeans.

meh... clothes and shoes are incredibly easy to make. Sure, it would be expensive to make them in the EU, but I'm sure I could go out of my apartment right now and find 100 people willing and able to do that within an hour.

The issue with chips is, that a modern fab plant costs $1bn. You can't just make it happen, immediately. Same with food production. Technology, guns/steel, energy/oil/gas and food, those are the main "national security" industries.

A lot of "fast fashion" (e.g. Zara et al) is actually made in Portugal to better react to trends. A lot of medium- to high-end shoe brands are made in Italy. A lot of traditional brands (Like Aigle) still produce most of their stock in Europe. Doc Martens have a range still made in the UK (and the quality is notably better). And then there's the bespoke shoe industry in London... just don't expect supermarket prices...

Many shoes are actually still made in Europe (Carmina, Lobb, etc.)

They're usually not the cheap brands though. The low end is dominated by foreign manufacturers, so Europeans have moved up the value chain.

There's Carlos Santos in Portugal which takes advantage of the fact that Portugal has the lowest labor costs in Europe.

Meermin (in Spain) has a hybrid model where they source from Europe (French tanners), start the manufacture in China, and finish in Spain. Their products are reasonably priced, and pretty high quality.

It's a similar story with Dutch clothier SuitSupply. Their designs are European, and they source their fabrics from Italian mills, but sew their suits in China.

If only we could redirect some of the massive defense budget for it, as a contract or something.

This doesn't seem like it would be that hard to happen - doesn't the US already mandate requirements that certain DOD purchases to be from within the US? The DOD probably buys enough to make it worthwhile for there to (continue to) be US-based fabs if they could only purchases those chips.

I think there would be widespread bipartisan support for such a thing. Those are jobs. The downside is that components will invariably cost more.

The stuff the DOD and other branches buys are usually more expensive anyway.

You can and should be protectionist when the situation calls for it. eg: national security. You shouldn't be protectionist because a politician's brother owns a steel mill.

How would that work in practice? Granted, the US might be able to manufacture critical components itself. What abou Italy? Honduras? Liechtenstein? Nigeria? Should they all produce their own hardware for national security concerns? And why stop there? Wouldn’t your suggestion in essence lead to countries fully self-supplying everything from food to cars to communication equipment to power stations and so on and so on. And not just manufacture every single component, but also source every base materials, crops, fertilizer and more. Arguably food safety or energy supply safety might even be more critical than semiconductor safety. Even when you reach at level of self-supply capability, you still need a chain of trust to ensure that you’re not compromised by your own government and intelligence agencies.

IMHO, such an concept would lead to a perfect protectionist regime while not giving any real enhanced security at all.

Final assembly of computers are a reasonable place. Destructively test critical parts to validate then, randomize usage to prevent targeted attacks.

It's not even terribly hard. AFAIK you can buy all the tooling quite easily, and it's a well understood problem.

Having your key components produced somewhere where you have a measure of oversight and inspection abilities isn't the worst idea if you're concerned about this kind of attack.

I would think countries would take a cost and risk-based approach. Countries that have the means, and who face major risks from a compromised supply chain compared to other standing risks, might want to think about moving things in-house.

Honduras and Nigeria might have other, more pressing concerns. Liechtenstein in particular is a financial center with an extensive manufacturing base and attendant expertise, so it might be worth considering for them.

Probably something similar to other aspects of the military-industrial complex. We buy guns and planes and rockets and ships and tanks we don't need because strategically, it's worthwhile to keep that manufacturing infrastructure (and more importantly, expertise) operating in case of the next world war.

>What abou Italy? Honduras? Liechtenstein? Nigeria? Should they all produce their own hardware for national security concerns? And why stop there?

wrt computers, why not if they're capable and willing? it has increasingly obvious and critical security considerations. russia already has a fledgling domestic processor industry for these reasons:

https://thenextweb.com/insider/2017/05/25/russia-showcases-f...

Russia's processor industry is also still dependent on foreign fabs like TSMC. https://en.wikipedia.org/wiki/Elbrus-8S

It seems like the main fabricators based in Russia are still working on getting below 90nm.

Sure, not many nations would be producing current-gen chips anytime soon. But in case the fan is hit, I imagine ~15 year old tech will make do just fine.

> What abou Italy? Honduras? Liechtenstein? Nigeria? Should they all produce their own hardware for national security concerns?

There are two paths to making this work. One is domestic production, which as you say might work for the United States, but the other is supply diversity. If you don't have a local industry but there is one in Canada, Germany and South Korea, and those countries are your allies, you're probably fine. The issue comes when you're entirely reliant on a single country (e.g. China), especially when that country is undemocratic/authoritarian. Because that makes it too easy for them to betray you without you being able to do anything about it, significantly increasing the probability that they actually do it.

> Arguably food safety or energy supply safety might even be more critical than semiconductor safety.

Poisoning a population or depriving them of power is a great way to start a war. Peeking at their computers is merely espionage. So yes, food and power might be more important, but there exists sufficient political deterrent already.

That's already a thing. Intel has its most advanced fabs in the US for exactly this reason.

Indeed but the "security" doesn't actually making the final product secure (as the Krebs article notes). Rather, it's securing the highest tech to ... keep Intel's market position secure.

(theoretically about having "capacities" that Chinese, Russians etc don't have but that's clearly an outdated concept - having the fastest processors in some defense system hardly matters compared to having secure electronic system broadly).

So that the USA can do the exact same thing to other countries? I have zero reason to believe that the us intelligence apparatus won't be exactly as intrusive as chinas when it comes to inserting themselves in tech companies, and plenty of reasons to believe otherwise (intentionally inserting back doors in cryptography standards, wire tapping fiber owned by tech companies, forcing tech companies to provide data on their customers, etc)

Yes. It will be exactly the same - but US companies would only be under the threat of the US government, which they already are.

And yes, if these are sold to other countries, they get to choose who owns them, the US or China. Unless they start their own manufacturing.

You are talking sideways to the person you are replying to. The US national security imperative should be securing manufacture of critical components.

If the intelligence apparatus abuses that for the purpose of spying on foreign governments that is an entirely unrelated issue. Their security is their problem.

Yes it'd be the same. So either your country become strong and handle your own affair, at a cost acceptable to your population, or you align with country(ies) with that capability that share your interest and belief, such as EU. But depending on your adversaries puts you on a very shaky ground.

The DOD does this for equipment related to national security. A lot of the cryptographic equipment supplied by General Dynamics for instance has US made semiconductors.

The concept of a strategic supply chain is not new, as semiconductors play an increasingly strategic role in the US' defense and industrial policies - treating them as strategic is becoming less and less controversial. In fact China has recognized the importance of semiconductors and taken measures to boost their domestic capabilities in this space[1]

[1] https://www.wsj.com/articles/china-plans-47-billion-fund-to-...

The US government has had this concept for as long as there have been foreign producers of electronics, if not longer. You probably haven’t heard about it, but it has been there.

Companies doing photo masks for chip lithography do exist in the US. Some of them have been bought by international parents. And those international parents jump through plenty of hoops to keep their certification for being able to continue to produce photo masks for classified military projects — only US citizens are allowed to work on those projects, they may need their own clearances, etc....

Don’t think just because SuperMicro may not have been a party to that system doesn’t mean the system in question doesn’t exist.

I don’t understand this comment.

The USA has a strong semiconductor industry. Probably stronger than China’s. China is traditionally not very strong an semiconductor fabrication.

What they are strong at is PCB assembly. Or rather cheap PCB assembly. You can do that in the USA too, it’s just not as cheap.

I’d guess most military stuff is done in the USA of domestic and mostly non-China made parts.

You’d think so, but a lot of stuff still comes from outside the States. I was an electrician on aircraft in the Navy and got to see a lot of the electronic stuff. Some stuff comes from overseas

How much comes from China or Russia?

The alternative, and often simpler one, is to tie your suppliers' economies and needs to tightly that they can't afford to screw you (and vice versa of course).

For example until the tariff war China was highly dependent on US agriculture (not just buying soybeans from American farmers, but by owning the largest pork producer in the US, Smithfield farms). Well they still are, of course, but they are aggressively looking for alternate trade ties.

The US and major European economies are intermingled tightly enough that it's unlikely that there could be a war between them. The NATO umbrella discouraged them from investing themselves in war-fighting capability (costing the US a lot less than the cost of fighting another war in Europe!). To that extent NATO is a fig leaf, an important one.

Right before WW I European economies had become integrated to an unprecedented extent, and people were saying that the integration would discourage war.

Yes. It's a very strange and basically un-contested idea that trade integration discourages war. If that was true then civil wars would be the rarest form of war, but the only type of war going on in the world today is civil war. And if you look through the 20th century, then other than the World Wars, most conflict was civil in nature. You can't get more integrated than a countries internal trade. So this idea that trade deals make war hard doesn't seem empirically supportable.

Good point. The third biggest war of the 20th Century was the Chinese Civil War.

> Right before WW I European economies had become integrated to an unprecedented extent, and people were saying that the integration would discourage war.

And maybe it did; unfortunately, the myth of offense dominance guaranteed that diplomatic crises would escalate uncontrollably into war, masking any effect which discouraged war as a choice in its own right rather than as a preventive measure against the risk that an opponent would choose war.

It's very difficult to isolate and evaluate the effect of one factor.

The DOD has been paying it through the Trusted Foundry Program. Probably not enough use of that, though. ;)

https://www.dmea.osd.mil/trustedic.html

If you mean among themselves fine - but among others along the peripheral I think it is for a good reason. It would be something of them isn't something likely to be piggybacked on for several reasons - even aside from the increased cost from overspeccing and reduced economies of scale the DOD easily have the muscle to extort built in spyware to the specification to be trusted to if they want big contracts.

And given history believing such is not paranoid and wise in a sense of 'not being stupid' as opposed to prescience given the past history of exposed similar moves and a lack of transparency. Probably a borderline 'blasphemous' statement like other uncomfortable truths about things not working.

I had a recommendation for this during Snowden leaks: use hardware from threat model's most capable opponents. As in, assume it's all backdoored by at least one party. Most threatening are specific, nation states. Use their opponents' hardware.

If stopping Chinese hackers, use stuff from Trusted Foundry running most secure software you can.

If stopping Five Eyes and Israel, use Russian or Chinese hardware running most secure stuff they support.

If not trusting anyone, use a computer made before 1997 that's not on a network and usually hidden in a tamper-evident compartment.

If paranoid, get rid of all electronic devices except those that detect electronic devices and emissions. Keep it far away from you itself sealed so it's not a point of attack. Periodically get it out to conduct a sweep. Your brain, pencil, paper, and hiding places are what you trust in this model. Even mechanical typewriters have acoustic and active RF weaknesses.

So, now you know how to be paranoid. Have at it. ;)

Why 1997?

As an opponent of NSA, I was studying them closely to assess capabilities. Anderson, Schell, and Karger were pushing "COMPUSEC" in the 1970's on the military who mostly didnt believe in it. They and NSA thought all you needed was access controls on computers plus COMSEC between them. The BLACKER VPN paper mentioned the rivalry between COMSEC vs COMPUSEC people, too. That was somewhere between 1980-1990. It wasn't until around 1990 after many pentests by INFOSEC pioneers and hackers in the media convinced them to hire their own hackers. They were still focused on crypto and software flaws since they were low-hanging fruit. I figured subversion would be high-profile targets only.

How Patriot act got passed was straight outta Nazi playbook. If a power grab, I knew secret backdoors, surveillance, and disappearances would follow. They'd launch USAP's forcing people to backdoor stuff or be held indefinitely under Patriot Act. Despite Haydens work, NSA still moves slow. I figured a few years before large-scale capabilities came online. Said 2004. Far as I recall, nothing in Snowden leaks refuted that estimate.

What about TAO? Hayden was forcing management to listen to engineers. They'd propose backdoors. Still software focused mostly, though. I estimated 1999. Later article on TAO history said 1997 was their start. Damn. Now I say 1995-1997.

Only thing left off was use obscure, less-popular hardware IP holders and terroristd probably werent using. Macs, Amigas, SGI... or just off-brand x86.

“If it’s a core business function — do it yourself, no matter what.”

— Joel Spolsky, talking about Not-Invented-Here programming

https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

Is security considered a “core business function”? I wouldn’t believe so.

Joel also writes “If you have customers, never outsource customer service.”. I think “security” is in the same category.

How does this apply to the types of programming like encryption where it is oft advised to not roll-your-own?

Encrypt multiple times with multiple encryption methods?

Including an in-house built cipher (task assigned to intern).

If there's reason to believe your team is better at cryptography implementation than the people behind the libraries you'd otherwise use, maybe. (This probably applies to the US national security establishment, not some random internet product).

If encryption is a core business function, you need as a prerequisite to having a business at all a couple of folks who know encryption well enough to roll their own.

> How does this apply to the types of programming like encryption where it is oft advised to not roll-your-own?

It basically doesn't apply to widely-used open source software.

Or to put it another way, you can always maintain an internal fork that currently has zero changes from upstream.

Can someone explain why a semiconductor production facility would be “punishingly” expensive to operate in the US?

Maybe the cost of electricity, but it seems like the labor cost wouldn’t make a huge difference since most of the production work (I presume) is automated.

I don't see how, either. TI, Intel, GloFo, and a few others have facilities here. Sure China can do it cheaper, but IMO the race to the bottom is over. People want quality, which explains why Apple and Samsung outsell, say, BLU.

I guess it's could also be

a) A missing local supply chain for the raw resources, making them more expensive

b) Additional shipping cost, since every chip will still be assembled into the product in Asia anyway

i would imagine at least part of it is supply chain centralization around eg shenzhen. it would have to be much more than just the fabs to be competitive

Safety, and environment costs. Many of the former semiconductor sites in the US are now superfund sites. Doing work exactly the same way it is done overseas would be illegal in the US, and doing it legally is expensive.

Once upon a time computers came with a full schematic diagram and enthusiasts could actually verify what they received. My point isn't so much the diagram as it is the culture of [showing/]knowing what's inside your black box.

This seems like as good an excuse as any to post a link to the seminal “Reflections on Trusting Trust”:

https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...

It’s an absolute classic that I’m sure 97% of HN readers are familiar with, but for the 3% that aren’t it’s a really great meditation on just how difficult it is to trust any computing system, without somehow recursively assessing the trustworthiness of everything that has ever happened before.

That was a paper reflecting on Paul Kargers compiler-compiler subversion he did in the 1970's evaluation of MULTICS.

https://www.acsac.org/2002/papers/classic-multics.pdf

The seminal paper on subversion in the lifecycle was from another high-assurace, security researcher named Philip Myers in 1980:

https://csrc.nist.gov/csrc/media/publications/conference-pap...

The recommendations were carefully specifying what things do, implementing them in a structured way to inspect for backdoors, using safe languages to block regular vulnerabilities, using covert-channel analysis to find leaks, verifying things down to object code or transistors, letting people build from crypto-signed source, and using couriers for the hardware from trusted facilities onshore ("trusted trucks").

People ignored most of that. Even Thompson as he was obsessed with unsafe language vs Wirth whose work let you choose per module. It's coming back to bite everyone.

Oh yeah, if any of you enjoyed Thompson's paper, you might enjoy this write-up that talks about discovering something like that:

https://www.teamten.com/lawrence/writings/coding-machines/

It took me 30min-1hr to read. I can't recall specifically. Worth it. :)

This is the kind of attack that the trustless.ai guys are fighting. They've secured their supply chain down to the silicon vendor at a fab in Italy. Really cool project.

Is Apple willing to pay for supply chain security? Could they afford it?

    Writing for this week’s newsletter put out by the SANS Institute, 
    a security training company based in Bethesda, Md., editorial board 
    member William Hugh Murray has a few provocative thoughts:

    1. Abandon the password for all but trivial applications. 
    2. Abandon the flat network. 
    3. Move traffic monitoring from encouraged to essential.
    4. Establish and maintain end-to-end encryption for all applications. 
    5. Abandon the convenient but dangerously permissive default access control rule of “read/write/execute”

This is going to be a huge problem. If that backdoor exists, someone will find it and use it. Not necessarily the people who put it there.

The supply chain is secure. But that’s not FUD that sells. The gap between mainstream news and tech news is closing.

> Indeed, noted security expert Bruce Schneier calls supply-chain security “an insurmountably hard problem.”

Centrally, yes. But supply chain verification & tracking is one thing blockchains are genuinely good at. There are actual blockchain-based products on the market for that.

Imagine a company such as Apple forcing their suppliers to authenticate each production step from raw material to shipped good on a proprietary blockchain. It's certainly doable from a technical standpoint, and Apple's suppliers are probably eager enough not to lose Apple's business to comply.

I don't see how this would help here. If you have a nefarious party adding skullduggerous chips to motherboards, they are hardly likely to attest to that on a blockchain.

My point is, just because someone added a verification message to a blockchain, doesn't mean it actually happened in real life - or indeed preclude additional things from happening.

I don't understand how this solves the problem:

- if all links in the supply-chain are properly verifying/tracking components, why can't they send this verif/tracking data to their client's server?

- if they're not making a good-faith effort, what does blockchain solve?

Blockchain is genuinely good at maintaining an immutable, trust-free ledger. It's NOT good at making sure the data that's written to it is true.

It does not, that's why a blockchain on it's own does not solve the problem but in conjunction with other technologies it could. E.g. food - you have to make sure your sensors (Temperature etc) are tamper proof, but you also have to find a way to make it hard to seperate the sensor from the goods. Only then it would make sense to use it as an oracle. Also if you make it hard enough to cheat the sensor at some point it's just easier to cool the goods and transport them accordingly.

Right, so it really doesn't solve anything. It just passes the buck further down.

Blockchain is a solution in search of a problem, here.

>It's NOT good at making sure the data that's written to it is true.

if you can't trust the manufacturer to begin with (as is the alleged case with super micro) then this is totally useless.

Yeah. That’s what the adults were talking about.

We can’t trust the supply chain.

I know from the car industrie, from each car, they know from each part the whole production history. A factory, for example, is certified to produce part "A". The factory has two machines (machine "B" and "C") who could produce part "A". The factory has to produce samples with machine "B". The samples have to pass several test and get approved by all the companys, who produce bigger and bigger parts with part "A" until you get the full car. So maybe a handfull of companys have to say ok to part "A". This can take like a half year or more.

So now is machine "B" certified to produce "A" but not machine "C".

This takes all pretty long and I don't think it's done this way in the electronics world, because there are almost no callbacks. So why bother so much with the supply chain.

And not to forget, the test is just if the part does pass the tolerances in the factory. It does not test, if a guy in the factory does cheat.

Even if the factory checked "everyting" and all the details. Then comes somebody like the NSA and they just grab the package on the transport from the factory to the customer (Cisco).

What does this actually solve? If you are surreptitiously adding chips to a motherboard, you'd just do it between whatever steps are written to the blockchain.

Even if you can totally secure the factory, you can just redirect the parts in shipping. I think "insurmountably hard problem" is a fair description.

That you for bringing for this intriguing idea for discussion!

Supply chain tracking of the sort you describe is excellent for tracking packages and the movement of goods. It's very useful for tracking the inputs to and outputs from a manufacturing process. Perhaps it's just my personal failure to understand your proposal, but this approach seems mainly limited to tracking known and expected inputs. So it might have a hard time detecting some nigh-invisible addition that takes place within an existing stage of manufacture and does not result in any additional attestation.

Can you help me understand? How does a blockchain solve the problem of a fabrication step compromised?

Reply on news.ycombinator.com