The Big Hack: How China Used a Tiny Chip to Infiltrate Amazon and Apple | Hacker News Reader

The Big Hack: How China Used a Tiny Chip to Infiltrate Amazon and Apple

bloomberg.com

2,493 pointsOsiris30770 comments

For those interested, there are additional threads discussing Apple's and Amazon's denials:

https://news.ycombinator.com/item?id=18142277

https://news.ycombinator.com/item?id=18138990

https://news.ycombinator.com/item?id=18143569

I have worked in card payment industry. We would be getting products from China with added boards to beam credit card information. This wasn't state-sponsored attack. Devices were modified while on production line (most likely by bribed employees) as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag.

Once this was noticed we started weighing the terminals because we could not open the devices (once opened they become useless).

They have learned of this so they started scraping non-essential plastic from inside the device to offset the weight of the added board.

We have ended up measuring angular momentum on a special fixture. There are very expensive laboratory tables to measure angular momentum. I have created a fixture where the device could be placed in two separate positions. The theory is that if the weight and all possible angular momentums match then the devices have to be identical. We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.

First, wow this is both incredible and crazy! Both the China-side hacks and your side's anti-hack. Mind. Blown.

Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?

I think what is described is an issue with process.

If the device is sealed with an anti-tampering system then the contents must be checked by a trusted entity before being sealed.

Trying to guess the contents of a box that you cannot open sounds a bit like madness.

> guess the contents of a box

Use X-ray? or whatever can penetrate the exterior shell

All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia and these are quarantined immediately after they return. They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).

The more critical the field, the more you have to treat those devices as untrusted before attaching them to your trusted zone.

I would imagine you would treat any kit that has been to those countries as disposable on return and crush them

Add the USA to that list.

Are you saying companies should or that you know of companies that do?

As far as I know, big EU companys do that, when they visit US, or they don't have sensitive information with them in the first place.

The problem is not the data on the machine, it's that you connect that machine to your trusted network when you're back.

Especially when border cops can seize your phone / ask for your passwords

Some companies certainly do that (in case the question was for me).

One of the companies named in thr Bloomberg article does. They just deatroy your laptop if it was in the hands of customs without your supervision for any length. US customs explicitly included, which is kind of wierd if you ask me.

That doesn't strike me as odd at all.

That’s perfectly expected. It’s not a stretch of the imagination to think border checks are abused for industrial espionage. If it gives your country a major advantge nothing is off limits these days.

My employer - doing lots of interesting engineering in among others the marine and aerospace sectors - certainly do that; we're based in Europe.

Basically, any device brought to the US and a number of other countries are issued for one-time use; if they leave our custody for even an instant, they are to be scrapped.

> They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).

Given how sophisticated these attacks can be, I'd think they'd issue disposable equipment to be destroyed on return, like a cheap netbook or something. I don't see how you could trust an individual viewing a simple X-ray scan to detect some extra microchip the size of a signal conditioning coupler.

Procedures change, as attacks get more sophisticated the next step could be disposable devices. But an attack like the one described in this article won't be mitigated by having a disposable device. On the other hand having your laptop "hijacked" while on a business trip will most likely involve some extra PCB or components that are a little more obvious that something that's "built in".

Then again many companies or public institutions would find it hard to justify shredding each week maybe tens of laptops and phones that still have to be good enough to work on. Basically they still have to be a "standard issue" device with your company's software stack, config, etc.

I'm sure someone can find a good compromise between security and wastefulness.

Does the security team have any motivation to reduce waste? Unless you're finding hacks in your devices already I see no reason to think they aren't just making you jump through hoops because it's funny.

Of course not. But remember that someone else is holding their purse strings so they might care about that.

The obvious solution would be to earmark some number of dedicated "china laptops" that will never be trusted with anything important, and get reused everytime someone has to visit China. If they get backdoored... who cares? They can only spy on things that were getting spyed on anyway due to being in China.

The post-trip inspection is not so that the device can be reused, it's so you can (try to) find out if it was compromised. A $3000 laptop is not a significant cost compared to the airfare, hotel bills, etc.

But it's useful to know when/if you're being targeted.

And none of those measures would have protected against the compromise detailed in the article.

Sure, we were discussing someone saying their devices came with extra PCBs inside (it's a bit hard to follow but scroll up to the original comment, currently first on the page).

> We would be getting products from China with added boards to beam credit card information.

>> Trying to guess the contents of a box that you cannot open sounds a bit like madness.

>>> Use X-ray? or whatever can penetrate the exterior shell

2 different types of attacks, 2 different types of responses.

> All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia

From what I understand, Boeing does this when employees visit France.

They have had problems with men in nice suits going through laptops stored hotel safes.

Yes, irrespective of country where its manufactured, if there are compliance requirements around an un-openable box, then some process becomes required.

But I think the GP's question is: "Whether it would be cheaper" - in the sense whether such an expensive QA process could have been averted by having a more trustworthy partner. One whom you're not on a race hack after hack.

The point is that if the devices are sensitive with compliance requirements then you must be able to verify them irrespective of who you hired to manufacture them.

You cannot just trust the word of a contractor on this because it's your ass on the line.

The point is that the process was to assure the device wasn't tampered AFTER shipped from manufacturer. Nobody thought it could already have been modified so early in the process. This is the eternal cat and mouse game. When I started in IT in 90s it was assumed that company network was quite safe and you didn't always need passwords, maybe for critical resources only.

I would think that, logically, and as illustrated, "the device wasn't tampered AFTER shipped from manufacturer" means after YOU have shipped it to customers. The anti-tampering system is to prevent modifications in the field.

The manufacturer shipped the device to us from China. We were already customer. The device would already have been locked. We would customize it some more (injecting cryptographic keys, application, placing our labels on the device) and then send them to merchants. The merchants were never customers, they would get it on loan from us. This was the only way to do it as the device could not be re-used with other acquirer so it only functioned as long as the merchant had valid merchant account with us.

Transparent plastic?

They may find a way to hide things out of view. My thinking would be to x-ray it. Airports are pretty good at that.

See, the article showed that even largest companies are not completely immune to the problem. This was decade ago and payment card industry, not exactly national security matters.

The PLA can lean on factory managers very effectively but they're not going to be interested in small time stuff like credit card numbers. The sort of sophisticated criminal gang doing something like this will have fewer coercive tools at its disposal and I'd imagine would target lower level employees with bribes.

I'm not sure; the DPRK are certainly known to make their espionage units self-funding through credit card fraud.

The problem highlighted in the article is very tricky because verifying a motherboard to that level of details is difficult.

As I hinted in another comment I suspect that they had a suspicion and checked those motherboards very, very carefully.

India

> Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?

I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?

It's not the chips that are the problem. Most of Intel's fabs are in the US, and their assembly sites are in a number of countries.

I think people should know how stark the differences for assembly in the US are vs outside of it. Something that costs, at low prototype volumes mind you, $20 in China for a dozen boards or so, would run hundreds of dollars in the US and still take the same amount of time. As it scales up, the ratio might improve, but these aren't like 10%-20% differences.

This is not microchips, this is basic PCB assembly

But I'd guess momentum is hard to change.

America has fabs, both old and leading edge, but ask industry giants like Gemalto to even bother to manufacture chips anywhere outside of Taiwan, assemble the final product outside of China.

They will never do that, because they look for the cheapest solution.

The bigger the company, the less it cares about things other than cost. This is why Mediatek and Broadcom can usurp the market of network SoCs, while making products with atrociously bad support. I personally dealt with both, and say that they wholely match their popular culture image.

I don't know how it is with USA, but for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres, for as long as they get them made inside the country.

What about Japan? I know they've lost most of their semiconductor business as well, but they still have some capacity no?

I don't know that.

I’m under the impression that China does not make chips, but they do final assembly cheaper and faster than everyone else.

I don’t know if any companies do PCB manufacturing and assembly outside of China in large numbers.

Not exactly, PCB assembly is super cheap everywhere thanks to propagation of chipshooters, what makes the cost go up is logistics - what do you do after you populate the board for your part? Ship it across the world, or to another factory behind the corner?

That largely depends on the size of your board and the total number you want to ship. As soon as you reach full truck loads or full container loads that additional shipping cost is marginal on a board level.

But still, that complicates the whole operation immensely in comparison to "just go next city block"

You are right in a certain way. What increases with distance is lead time and as result inventory. Both of which are off set by volume. Cross-border adds customs issues. Complexity is not that much of a problem nowadays, maybe it never was. You are right that all of this oblyakes sense with large numbers, small scale production is better done locally as a rule of thumb.

Sony production lines assemble Raspberry PIs in the UK:

https://www.theinquirer.net/inquirer/news/3014651/sonys-penc...

That is true I think because of what happened to ZTE earlier this year.

Gemalto manufactures and assembles many, many products in Europe. The entire European security cluster won't use anything else. I know because I played a part in smart card development there. And all of that was from Europe. Wafers, chips, holograms, mag stripes...

> for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres

Well, if your chips are bigger and slower, you will need more chips and mounts/packaging to place them. If you need more chips then the weight of missile/plane/tank will be increased and available space decreased.

So at the end, the 'uncaring' military will receive a weapon which is worse than competition.

All thanks to an outdated chip.

Offshoring has its costs.

It's the unknown unknowns that get you.

> Has the West completely lost the ability ...

at first I had the same thought. but i have to question how securely the same manufacturing could be done in a US plant.

the US employee base has its fair share of desperate, ethically challenged individuals. and plenty of incentives to make a quick buck could be offered here too. idk.

The consequences if US citizens or residents get caught engaging in espionage for a foreign government are go-to-jail-for-years serious.

And it's not white collar resort prison ... no no no!

https://www.youtube.com/watch?v=oBzvMLW0ii4

Death penalty.

ok. a fair point.

OTOH, given US law enforcement's low efficiency what are the chances of being caught? and what if it's merely corporate espionage?

finally, the US is an open society with strong personal freedom guarantees builtin. what if the perpetrator has ties to a foreign country and simply leaves the US after they've installed the vulnerabilities?

You are underestimating the FUN of playing anti-anti-^N-hacks. I have had the privilege to be paid to so anti-anti-^N-hacking on a firewall thingy in the past and it was a challenge and a joy!

The day I figured out to measure the angular momentums and calculated the feasibility I was walking around the office proud like a peacock.

Honestly if it weren't programming I feel like this is a movie-worthy story. To me it sounds so thrilling like a spy movie plot but am I just imagining that or was it actually this crazy / cool / integral to way bigger moving parts / things like I'm assuming? Regardless kudos. Your story definitely started my day on a happy note, thanks!

I'd very much love to hear more stories if you have any!

We had MasterCard end-to-end test auditor on site. This is the first time ever you get to do a transaction with real transaction system with real credit card.

Due to requirements we opted to have the only large meeting room to have outside our secure zone. This created an issue as we had no network access from there and in the end we decided to use slow GPRS terminal for the test.

The end-to-end test starts with offline transactions which by their very nature are quite fast (it is negotiated between terminal and card).

But then we went to online transaction and it finished instantly too.

The auditor, bewildered, proclaimed the test failed as he assumed it was incorrectly processed offline instead of going online. But then I pointed out to the printout to show ARQC (basically says it was certified online).

Now, the real discussion started. The terminal was very slow taking quite few seconds to establish GPRS and then even more for the SSL handshake so the auditor said it was not possible to make it work.

How it worked was that I have completely gutted OpenSSL and had entire cryptographic state stored locally (safely, using internal HSM) so the SSL session could be optimistically re-established without another handshake even after TCP connection was closed. The first message the terminal sends is already encrypted transaction message, there is no SSL handshake. I wrote an application to terminate the connection in our data center so that it stored the states of each connection in the database. The entire handshake was only done if the first message could not be decrypted successfully.

The operating system was single-threaded with no multitasking of any kind. This meant that all applications on this device did their operations sequentially. Send network message, print something, display something, etc.

I wrote a cooperative multitasking functionality into the application (using coroutines) so that it could work on multiple tasks at the same time (like talking to network and printing).

I then have segregated all data on the printouts so that it can start printing without having to already have response from network. Hopefully if everything went right, the response would come before it even came to that place on the printout effectively looking as if it was done in zero time.

Seriously, pitch this story to an editor at one of the major tech blogs. It would be an incredible read.

Agreeing with mortenjorck. I'd love to read longer stories.

FWIW, the described technique (or something roughly equivalent) is now standardized as 0-rtt early data in TLS 1.3. (you still need 1-rtt for TCP, unless you can combine this with tcp fast open, or run TLs over UDP)

Impressive.

But am I wrong to have my hackles raised by a) the roll-your-own security nature of this, b) the reliance on a single developer's single stack implementation as what guarantees the integrity of the system? It seems like there are a lot of assumptions baked in.

I, too, would love to see a more detailed write-up--if there's a big idea here (almost a unikernel thought), it deserves to be shared and tried by fire.

i love this ~

quote captures the human element playing strong in face of bad system

^ This gave me a much needed smile today, thanks for sharing.

The fact that you keep referring to moment of inertia as "angular momentums" make me doubt that your story is true.

English isn't my native language. Of course you are correct it is called moment of inertia.

how would you measure the moment of inertia though? probably by checking your changes in angular momentum when applying a moment

a good hack can sometimes be it's own reward

This wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China.

Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.

Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.

How was it finally "dealt with?"

I wouldn't know. We were just buying the stuff.

They got better at hiding and his company couldn't detect it. Problem solved.

When I worked in telecom (a while ago) the manufacturing was shifted from China to Thailand/Other SE Asia due to this. The Thai companies weren't as efficient, but were much more open and honest when problems would arise., plus they didn't blatantly steal tech

Get an x-ray machine? They are surprisingly cheap pieces of hardware if you are willing to deal with a small area, low penetration image. Low penetration means no lead, which makes for something that's about as cumbersome as a large bar fridge.

It's mentioned in the article that X-Ray didn't help much: 'Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment'

This comment is specific to the parent talking about their experiences producing credit card terminals that ended up with PCBs implanted in them. Here it is appropriate.

And as soon as they know you are using X-Ray to test ( I would be surprised if there were no insider information leaking on the OP's end ), they would develop a way so that somehow the X-Ray images looks identical. May be a heat spreader like metal as cover up.

Can't the image be diffed with one of a trusted system?

Sorry, but this doesn't sound as true or there is huge mistakes done in choosing supply chain for such sensitive matter.

How come company keep ordering devices from some unverified sources from China, and after hitting a wall keep doing same?

How do you accept shipment of such devices without randomly opening and inspecting sample(yes losing all data, but electronic inspection can be done).

How you didn't investigate that with Visa/Mastercard? Whoever does that, he will lose his payment terminal certification after such incident, because they will track them down by IC serials very quickly.

What if vendor changed power supply board or even components type on it, and your momentum or weight test will make false positive?

Unless... your employer or you buy single devices, on demand, from some shady aliexpress seller. But then, it is plain suicide.

It sounds like they are just ordering stock products from Amazon.

While it may sound sensational this was more of an operational issue, really. We were told by Visa and Mastercard that it is not even a question if we are going to be targeted. If you work in payment card industry you are constantly being attacked and the only way is to make it part of the process to deal with those things. Our network was hacked but what was important was tight, almost mathematical processes around protecting very specific material like credit card data and PINs.

For example, PINs are only ever being in unencrypted form inside of Hardware Security Modules and only for the purpose of being encrypted with Visa/Mastercard exchange keys. The process was designed so that nobody has enough access to ever get enough cryptographic material to be able to decrypt anything, at least two or three people would have to collude to do anything.

It also happens that we put all our resources in investment in software for the platform locking ourselves in. It would be rash decision to change the platform and it would probably kill our company. Also we (correctly) gambled that it would be dealt with quickly.

Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking of malicious implants

Look, there are enough supply chain problems with counterfeits already, you don’t want to start thinking about malicious implants. Just google for it, it’s massive

Why don't you guys consider to expose this by suing the bad manufacturer? I believe this could help other truely honest manufacturers both in and outside China to beat the wrong doers.

The Chinese operation would simply shut down and reopen under a different name. And the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.

But I don't think it's that good to keep playing the rat-cat game.

> the credit card companies are always very worried about their brand image, so they are not interested in any negative publicity.

Letting the bad guys keep doing what they do while putting it's own customers under shadow, to me, it sounds like the credit card companies are helping the bad guys.

As a Chinese myself and a credit card user, I'm more worried than you guys do, because we are more likely to receive that kind of treatment (compromised credit card and computer chips etc). I really hope somebody can teach those bad doers a unforgettable lesson.

This is fairly analogous to my comment here: https://news.ycombinator.com/item?id=18138847

Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.

Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab. Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.

Presumably.

I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.

> as once they were closed they would have anti-tampering mechanism activated so that later it would not be possible to open the device without setting the tamper flag

You didn’t specify what type of anti-tamper was used, but I wanted to jump in and say usually that means nothing. The US government intercepted packages [0] and put in back doors (removing and replacing the seals), so I’m not sure why you were so quick to dismiss state sponsored attacks by something as simple as an anti-tamper seal. You can learn how to do it yourself at most medium to large hacker conferences too (DEFCON, BlackHat, HOPE, and CCC to name a few, but there’s more with it).

[0] https://www.techradar.com/news/networking/routers-storage/ph...

The chinese government is probably not interested enough in credit card numbers to warrant involvement.

It would be foolish to show your hand for few pennies.

Might make a difference if those pennies can't be tracked back to the government.

Yup. This would be the modern equivalent of Air America and other schemes by the CIA to raise money to operate by involving themselves in illegal activity like the drug trade.

One would consider those activities to have taken place in "modern" times as well...

You're not suggesting the CIA is out of the drug business...are you?

Iran/Contra got the CIA into a lot of hot water with Congress. They had some pretty stringent restrictions put in place as a result.

https://www.csmonitor.com/1987/1113/aintel.html

People said the same thing about personal detail metadata as well. While theft would be foregone foolishness tracking transactions could have value to them. Granted it would probably be easier to snoop on the fiscal institutions.

You can just buy counterfeit anti-tamper stickers but if there is a switch inside the unit that flips a bit in some sort of write-once memory, then that would require removal of an entire chip and replacing it with another that may not be 100% the same. You can have a chain of trust in the system where chips will only talk to each other if they all spit out the right hash. Bury the SPI/I2C lines you use for this trust check within the PCB so you can't access it without drilling the card and add a layer of anti-tamper traces that trigger another tamper event if disturbed. Now what was just a quick install has turned into a whole PCB rework job where you are having to swap all the chips with a virgin set, assuming you can't get your hardware in there prior to final assembly.

Use an AES256 key from the factory to hash the chip's burned-in serial number and the time from the RTC. Lock out JTAG interfaces so once the chips are burned, they are inside their own fortress. There are a ton of ways to really lock down the hardware other than a shiny sticker and weird screws. Those keep people from breaking their own hardware, anti-tamper tech in chips keep out bad guys.

The device outer enclosure was tamper evident but the device itself was tamper proof HSM, basically. Any kind of intrusion (melting, dissolving, drilling, etc.) into a secure internal enclosure (separate processor, memory and battery) would cause internal battery to be disconnected from internal SRAM and basically the device would loose all cryptographic material and then self-destruct.

To give a bit of background, when you type your PIN on credit card terminal it is not the terminal application that is really getting the pin (well, except for special credit cards but that is really problem of the Bank that issued the card). The Visa/Mastercard mandate that the application don't have control over the PIN and that the PIN entry uses physically separate keyboard and display.

To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry and the PIN is transferred directly to the HSM where it is being encrypted before it is transferred to the application processor for the rest of processing.

Those VISA/MasterCard rules can't be universal because there's at least one bank issuing merchant terminals that run Android and take the PIN on the touchscreen:

https://www.commbank.com.au/business/merchant-services/eftpo...

This only accept contact-less payment who doesn't require pincode.

PED/PTS devices have even stricter guidelines than contactless payments.

But this device doesn't need a way for the user to enter the pincode. So, all the sensitive part of the terminal is probably completely isolated from the android part.

I don't know this device internal and the PED/PTS exact requirement but it seems plausible for me.

You have something like a physical compartment who include the NFC and everything needed to process it like in a classical terminal. This compartment is highly secured as requested by the specification with just a very simple interface for the android part to send the amount to bill.

I've seen a lot of each-machine running on windows. Doesn't they work like this with the windows machine just managing the display buttons to select the amount and sending this information to the secure part who handle card interaction, pincode and delivery of the money ?

Yes this is a common design in fact most current solution segregated the POS and POI completely anything that handles the actual credit card whether it's C&P, Track2 or NFC is a closed black box with the required PED/PTS/POI and P2PE certifications the merchant never sees what's going on they only can talk to the thing in bill the next card X and get a confirmation of the transaction that's it they don't see any of the card data they don't even see any card holder data unless they collect it in a side channel e.g. a loyalty program.

Now none of these certifications or standards is bullet proof but people have a very skewed vision of the PCI certification process likely due to bias of only having interacting with the PCI-DSS requirements for merchants and low levels to boot meaning they didn't had to do anything but to fill the SAQ themselves and be on their way.

It accepts contactless payments over the threshold where a PIN is required, and does in fact prompt for a PIN using an on-screen keypad.

Those are universal PCI council rules the fact that the terminal runs android doesn't mean anything it still needs to be compliant with the PIN entry device requirements as well as PTS and POI requirements depending on the exact nature of the device:

https://www.pcisecuritystandards.org/documents/pos_ped_secur...

https://www.pcisecuritystandards.org/documents/PCI_PTS_POI_S...

The requirement for the tamper proofings is literally the first requirement in the PED standard:

A1 Vendors must comply with all components of A1.

A1.1

The PED uses tamper-detection and response mechanisms that cause the PED to become immediately inoperable and result in the automatic and immediate erasure of any secret information that may be stored in the PED, such that it becomes infeasible to recover the secret information. These mechanisms protect against physical penetration of the device by means of (but not limited to) drills, lasers, chemical solvents, opening covers, splitting the casing (seams), and using ventilation openings, and there is not any demonstrable way to disable or defeat the mechanism and insert a PIN-disclosing bug or gain access to secret information without requiring an attack potential of at least 25 per PED, exclusive of the IC card reader, for identification and initial exploitation as defined in Appendix B of the PCI POS PED DTRS

Clover CEO here. Won't comment on a competing device but this may not work the way you think. In Clover's approach the touch controller input isn't reaching the Application Processor running Android when in PIN entry mode. You can do patent search if you're interested.

That would be in line with the requirements. You go through stringent certification with the software and hardware that has access to the actual PIN and then show that the application and application hardware never really has any access to it so that you can customize/update your software.

This is the easy part.

The hard part I remember was establishing secure communication between all components in the system (initializing HSMs, injecting keys). I remember helping designing the process and writing hundreds of documents describing various security-related procedures like how the HSM racks are inspected, how the keys to the racks are fetched from the safes, how there are multiple safes for multiple security officers, how the officers are prevented from ever having access to other safes, how fetching anything from safes requires logging and using tamper-evident containers, how the logs are inspected, and so on.

I have designed a special cryptographic protocol so that we could generate and inject keys to the devices in KIF (Key Injection Facility) and separately to our database (to establish communication with the terminal). Fun.

Agree the people and process side is very difficult to do well. Familiar with all those and more -- we have extremely good, dedicated employees who care deeply about doing those things right.

We have some fun stories on this topic, like when we were using our PCI PIN approved secure room in our development office for the first time. We papered over the cage to prevent a security camera from being able to see employees entering PINs on the HSM. An eager employee papered over this cage a little too well cutting off the natural flow of air. And then there was a bug in our offline CA code and we spent 30 minutes in that air deprived cage while debugging occured :) finally the bug was fixed, we issued the cert on our first production device, and stepped out to get a breath of fresh air. Obviously this isn't our daily driver secure CA room :)

(If anyone reading is looking for a job in security engineering, we're hiring! https://www.clover.com/careers/engineering)

I have few more stories like the time when I closed the HSM rack door a bit too energetically and caused outage to entire company as we had to bring in third security officer to re-initialize it.

We also had special screens created for all cameras in the datacenter to block view on the HSM racks.

The biggest issue was, just before end-to-end test we figured out we forgot one of critical procedures (it was establishing authenticity of the HSM used) and we had to scramble to get new HSM and to re-establish all cryptographic material (so new storage keys, etc.)

404

What you're describing sounds like another backdoor of its own!

The general problem with most "industry security" approaches is that they simplistically attempt to wrestle ultimate Godmode-control for themselves, rather than working towards eliminating it.

Worked in the payment industry for years. Visa/Mastercard do absolutely nothing to verify that companies are not storing Pin codes. The HSM is required for communication with them only.

That's not correct the QSA will validate that the device does not store PIN codes or the that the merchant does not store anything they are not allowed.

Devices that accept cards need to comply with PED/PTS security requirements including very strict physical security requirements which are validated by PCI council approved laboratories and firms.

You are not getting a device on the market or usable with any merchanet network without complying with this: https://www.pcisecuritystandards.org/documents/pos_ped_secur... and a few other standards.

About that, I can only say that Chinese android POSes do everything in software, for sure, without any hsm present.

The question is, how Chinese banks coax Visa into allowing them using them.

Probably it just goes through as a card not present (CNP) transaction?

So you mean it first authenticates the pin, and initiates CNP after? Never thought of that as possible.

It's not CNP doesn't use the pin it uses the CVV2, you also can't use the chip and pin or track 2 swipe data for a CNP transaction.

I think the GP is confused on how a POS works, POS isn't a POI most of them don't touch the credit card they just talk to the reader, most readers today are P2PE closed loop solutions so the only thing the POS does is sends to the reader charge the next card $X the reader will then reply if the transaction went through or not and that's it.

The reader itself will talk to the acquiring bank or the payment provider in a point to point encrypted closed loop and the merchant would never see any credit card details.

Simple greed? As in: "play ball, and you get access to the gigantic Chinese market"

The POS and the PED/PTS isn't the same thing the POS can complete the transaction without touching the credit card in fact most of them do exactly that the only thing that it does is communicate with the PED/PTS to send the amount and get a confirmation/denial.

Second this, after having to go through a service level 1 DSS review for a few years. Lower level reviews (3,etc) just require self validation.

SAQs don’t involve QSAs. They are also intended for merchants which are a rounding error also there is no SAQ for PA, PED, PTS etc. certifications only for merchant PCI-DSS.

Wrong. PIN codes are entered into a damn mobile app and passed through an API. Billions of times per day. You guys are clearly missing the card serciving aspect of the industry.

Please show me the device that transmits the pin of a chip and pin card to an API while not being compatible with PED and P2PEE requirements.

You can totally fake your way through PCI audits. I know of a company that did it for years using a fake network and servers. Not sophisticated at all. Most auditors do not find all of the compliance violations. They have one person do it. It's all about money.

You can fake a lot of things so what? That’s not the point, also PCI DSS is pretty crappy but the hardware vendor, payment provider and P2PEE certifications are a completely different story good luck faking it.

Sure you can send fake devices to be certified and sell something completely different but the same can be said for any certification and if you get caught boy or boy...

That's not correct.

HSMs are required so that the company does not need to have PIN codes exposed anywhere. Not having PINs or full credit card data makes your life easier as there is nothing to steal from you in the first place.

If your company stored PIN codes it means you were in breach of the contract and it had to lie to the auditors to pass the certification.

"If someone was going to break the law, they would have to lie about it first."

Legit companies don't want the info and anyone that wants the info isn't doing anything legal with it.

It is correct. Incompetence abounds. You are correct about the HSM but it does not enforce anything except for the exchange between whoever and MC. You do realize that pin codes are entered into a UI and phone system as plain text right? There are PCI audits but they are a joke.

You realize when we talk pins we mean authorizing your credit card chip&pin transaction? Nothing to do with your phone pin or maybe som other pins.

The pin we are talking about is what is customized on your credit card (directly in its memory) or its equivalent in your bank's HSM for the sole purpose of performing CVM step negotiated by yor card and payment terminal.

> To achieve this, the keyboard and the display is galvanically separated for the duration of the PIN entry

Perhaps this sounds too dull to ask, but what stops the terminal from just ... not separating the keyboard and display?

Certification. Worked in the same industry, and there were very strict both hardware and software requirements for POS software. Having gone trough credit-card audits, early EMV certification programs, and certification to place non-payment software next to payment software on such systems, I can tell you - it's no joke :)

> I can tell you - it's no joke :)

But still as a user I have no idea if I'm talking to a certified machine or not.

As a user, fraud is not your problem. Security isn’t there for your benefit.

It's not your problem. For the transaction to take place it has to go to your bank and your bank trusts Visa/Mastercard to manage risks. You would be surprised to know most frauds are absorbed by banks. Nobody is interested in people fearing plastic because it causes people to increase very expensive debt plus additional interchange fee from every transaction.

Certification [...] I can tell you - it's no joke :)

I'm not so sure. "And then initial supplier inserted hardware that thwarted all those pressing considerations" sounds a punchline to me. :(.

That, at least, is a problem that cab be solved by tightening security at supplier site.

The safeties are mainly to guard against the rest of the world. For example it prevents tampering in transit, or even in our own company - disgruntled employee can't do anything.

Or think for a second about the fact that we leave the device for, hopefully, entirety if its life at the client site. We had clients that were shady businesses like strip clubs that no other companies would touch with a stick.

But in the US we just use signatures for card transactions instead of chip PIN :/

Do you honestly think any of that is not something a nation state actor could pull off though?

I never said a nation state couldn't attempt this but just that if you try really hard to lock down your hardware, it becomes very expensive and risky. You may be better off attacking the system in another way other than cracking it open and soldering your payload in there.

Payment systems are typically better defended than by just a sticker.

It’s not surprising to see a ton of tamper switches, vibration/shock sensors, even light sensors. And they’re all powered by an internal batter and separate MCU that will brick the device upon open.

All of which are overcome by nation state actors if they want too.

Depends on the hardware and the anti-tamper measures. I've seen POS terminals where the pcb was completely encased in security plastic, where any attempted breach would wipe the internal security keys, which meant the hardware just became a useless. They're so sensitive that these things enter "tampered state" from time to time without any tampering. I developed software on these things - and bricked multiple devices by accident, even though this was 'development' hardware.

I have caused production outage during one of our reviews of the rack with the HSM. The procedure required opening the rack, inspecting and accounting for all devices and cables and then checking the status of the HSM itself. At the end of the procedure the rack had to be closed.

Visa/Mastercard required that there are two people present and that there are two people required to open the rack.

We had it modified so that it has second lock.

The additional lock did not fit perfectly. During the procedure when I tried to close the door the door snagged on the lock and then slammed shut. It wasn't a lot of force but it caused the HSM to loose its keys and required a lengthy procedure to get three security officers to initialize the HSM with components stored on their smart cards during which the whole system was unavailable.

What's the motive? If we assume a motive of theft, seems more reasonable that there's organized crime involved than a state actor.

> something as simple as an anti-tamper seal

To echo, it's actually quite trivial to bypass anti-tamper stickers with acetone and a needle.

https://www.youtube.com/watch?v=SqkMIek8sqI

Maybe a setup to measure inertia tensor and center of mass (in that setup's axes) will be easier and I think it's what your call "measuring all angular momentums".

Gosh, I would love to read everything about this. I know there are some videos about anti-tampering card readers on youtube, but not on the feedback race between hackers and security.

A truly sophisticated attacker would leave some easily detectable bad eggs in the stream after countering a countermeasure like the weighing.

> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.

You only need to measure three angular momentums, all other can be calculated. See https://en.wikipedia.org/wiki/Moment_of_inertia#Motion_in_sp...

"This shows that the inertia matrix can be used to calculate the moment of inertia of a body around any specified rotation axis in the body."

On the attacker side, they only need to make sure three angular momentums match.

Maybe they're also measuring centrifugal force for rotations with axes not intersecting the centre of mass.

I believe you are talking about "Rotational Moment of Inertia" about various axis, instead of "Angular Momentum".

Angular Momentum of a body at rest is zero.

Did you look into Terahertz scanning?

Very interesting methods you used to detect the changes! Out of curiosity: was there a reason that taking an xray of the devices was not an option? Industrial/veterinary xray machines can often be had quite cheaply...

> We could not measure all possible angular momentums but it was possible to measure one or two that would not be known to the attacker.

You mean moment of inertia, not angular momentum.

You could measure all of them! Given the moments for the three principal axes at any point, you can use the parallel axis theorem to calculate all the rest. In general, there are 10 degrees of freedom: 3 for the position of the center of mass, 3 for the axes, three for the moments, and for the the total mass.

For a nicer way to count and to do the math, you have the inertia tensor at the CM (a 3-dimensional rank-2 symmetric tensor, 6 DoF) plus the location of the CM and the mass.

In any event, this is a cute tampering-detection trick, but I would have started with an X-ray or CT scan.

Another implication of the parallel axis theorem is that the attacker could perfectly mimic every moment of inertia by shaving plastic. They wouldn't have to know which two axes were being tested because there are only three real numbers worth of information in the system to begin with (once center of mass and total mass have been dealt with.) In the whole MOI tensor there are only six free numbers which sounds like a reasonable number of parameters to fix by adding and removing small amounts of material.

What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start. Not every problem is technical!

I honestly did not know about that. I thought that if you move any mass (remove non zero mass and place it somewhere else) there must be at least one axis which you can use to detect the change in moment of inertia.

whatshisface is correct.

If the mass had to all move to one other place this would be correct (the center of mass would have to change), but the attackers are able to move the mass to multiple other places.

Another problem is anti-tampering measure is applied before the initial tampering check have been applied. Anyway, that wouldn't solve everything considering that if the chip itself is tampered with, that's undetectable short of an electron microscope analysis and even that wouldn't solve the problem of backdoor in the original chip design.

As you say, that's un-winnable. The only way to really build trust is the capacity to sue your manufacturer to oblivion - I mean the real oblivion (destruction of shareholder value) with criminal charge for the company officers, not the lame single digit percent of a single year of profit with a discount if you settle.

> the real oblivion (destruction of shareholder value) with criminal charge for the company officers

This very rarely happens even if people have got killed. Also, good luck suing a Chinese company in China.

I reckon this could be addressed with contractual guarantees in exchange for doing business in the first place. Make it a contractual requirement that the supplier agree to adjudication in the United States in the case of security related concerns or other tampering with the final product or with the designs. They could still refuse if something happens, but I reckon at that point you can just work to shut them out entirely from the US market.

At this point, I think it's fair to establish an industry watchdog group that works as a clearinghouse for knowing all the American companies sourcing parts from the PRC and which companies they source from. This watchdog organization would be responsible for blacklisting suppliers from the entire US market if they don't agree to adjudicate in the US or if they fail to meet that contractual obligation).

This won't bankrupt them outright, but it will sever them from the US and if this watchdog group expands to aid non-US companies, could help shut them out of the entire global market.

I would also require registration of all top executives of these corporations so that you would make the punishment sticky if they moved to other companies. Overall, you've got to set a high punitive cost to non-compliance and non-cooperation.

You can't lock China out of the market without diplomatic implications. Even assuming you managed to get a purely private boycott of China together without somehow violating anti-trust law, by the time it started making a serious dent in the market the Chinese government would have started making representations to the US State Department. Where it goes from there is anyone's guess.

(Ironically the other direction prevails in regard to Israel: there are anti-boycott laws! https://www.lrb.co.uk/v40/n14/amjad-iraqi/short-cuts )

> What was this company doing in hiring an untrustworthy manufacturer to handle secure devices? That's playing a game you've lost from the start

You're assuming that there were feasible alternatives; from their comment below:

"wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China. ... Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device. ... Fortunately this only lasted for few months until it was dealt with"

Business lesson to include a 'no China' if the hardware handles anything sensitive, with a large fee for violating it for the middle man servicing the contract.

Ah, you are right. English is not my native language.

Indeed there are only a finite number of moments. If this was not true you could effectively 3d scan an object by just measuring its inertia response.

I enjoyed your comment. But you are looking at the difference in the inertia tensor of an adulterated board from a non-adulterated one. Not the inertia tensor itself. Signal to noise problems with the measurement.

Could this be solved by a clause in the contract that specified if any randomly sampled devices came tampered with, the manufacturer did not get paid?

Ah, yes, and then we would voluntarily go out of business. And the supplier will just have another customer that would not be so principled.

Huh? This would be between you and the Chinese manufacturer. If you found s tampered unit, you don’t pay them. How should that make you go out of business?

They need the units from somewhere, with a deadline determined by their own runway, and presumably only some of the Chinese ones were compromised. Sure, they could avoid paying, but then they have to spend months getting the production run going somewhere else .. which might also be compromised.

I'm sure the Chinese government would be willing to keep afloat companies that allow them to tamper with.

At what point does it get cheaper to buy from sources that are more expensive but more trustworthy?

Maybe time to buy your financial hardware from somewhere not china?

I know it's nice to blame China for everything... but it's not really the root of the problem here, supply chain management and control is.

China is 100% to blame here.

Absolutely.

Why do you use those suppliers?

That's kinda amazing

So you had hard evidence that your manufacturer was vandalising your property? Couldn't you drop them?

The main lesson I learned from my Dad’s employer (twice) was always have two vendors. You can play them off of each other. When I did contract work I saw how powerful getting out or putting for away the checkbook can be. Large vendors ignore you if the checkbook isn’t moving.

I say “twice” because they were also the biggest employer in town. I got out of there. Slim pickins for career opportunities that didn’t revolve around BigCo.

Seriously, why are we still outsourcing chip manufacturing to other countries? Sure it's cheaper, but we sacrifice a lot to have a society of corporate slaves build our tech. Security, core domain knowledge, capability, corporate secrets, patent rewards and enforcement, etc... All of it you throw away the minute you ship your manufacturing out of the country. I've seen enough board printing machines out there to start working on our own. As a country, we need to close this gap, more automation and capability and there will be no need to outsource circuit board printing and manufacturing. We will be much better off.

It's hard to compete domestically against low prices caused by China's completely different standards for wages and human rights for laborers. Plus, here we don't like the idea of manufacturing industries being subsidized by the government (except in the case of "defense" of course...), while China obviously has no qualms with doing so.

> except in the case of "defense" of course...

Well, this thread is exactly the reason we subsidize the defense industry to make sure things are produced in the US.

Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?

The moment the option of taking control of a production line of something _this important_ becomes available, your local specialized organized crime outfits will start to figure out ways to insert themselves into those production lines, learning the ins and outs, and figuring out a way to get something, anything, in there that won't be noticed but will give them a hook into millions of systems.

The law does not prevent crime. It just puts a price on it. While that price is typically too high for individuals, for organizations that have no business registration to revoke, and no CEO to drag to court, it is an entirely trivial cost.

>Counterpoint: even if we ignored the fact that you cannot possibly produce the volumes of chips necessary at the price necessary in your country rather than in "we don't have to acknowledge all the human rights violations" countries, why would you believe this problem goes away if chip manufacturing were done in your own country, rather than another?

It's certainly easier to enforce laws and observe manufacturing processes at domestic factories than it is at factories thousands of miles away in a country that actively encourages IP theft and other wrongdoings, don't you think?

Not really, no. Why do you think it be easier to police? In concrete terms: who do you expect to do the policing? Because this is the kind of work that requires an expert, to catch an expert, messing with a product created by people 100% ignorant of _what_ they're making. Just _that_ they're making them. Where are you going find enough experts to check the work of even a single production line?

Because "thousands of miles away in a country that actively encourages IP theft and other wrongdoings". It makes perfect sense. If you don't think that, try to run any business in another country. It's just obvious that it would be easier to do domestically. You just list other set of problems. Those problems only get compounded if we'are talking about another country.

Not all societies are equally corrupt.

But no societies are free from organized crime. And last time I checked, no western country was free from card skimmers either.

You can prosecute someone in your home country and even change policies with enough political support. Best of luck doing this in China without getting 're-educated'.

But note that as always: that's too late. By then, the damange has already been done, and the person you take out of the system is simply replaced by anyone else. Card skimmers sure as hell haven't gone away just because we wised up to them: we now just pay A LOT MORE on law enforcement, and all that gets us is law enforcement now at least staying in lockstep with the skimmer's increased sophistication.

Now imagine how much it'll cost to stay in lockstep when the level of sophistication becomes "a single component the size of half a grain of rice".

"Free", no. Much less affected, yes. Organised crime in Mexico and Italy is a much greater problem than in, say, the UK or Germany.

You're letting the perfect become the enemy of the good here

Not really: I'm letting the practical dictate the process. Catching this kind of inflitration requires experts to scrutinise the work of people who know nothing about what they're making, to catch experts with a much more vested interest in success, and you aren't going to find enough experts to take that job, at a good pay, to check the output of even a single plant. Let alone an entire industry.

If the question is "why don't we move chip production to the US, so this doesn't happen" the answer is "because you might be able to do that for one or two plants. And it'll probably drive up the price by an order of magnitutde, so they'll go out of business soon after. And you sure as hell can't move the entire industry because of prohibitive cost and shortage of manpower to do the bit that you want tacked on that can't be trusted to be done abroad".

Canadian steel is considered by this administration to be a national security risk. But Chinese made boards and chips installed in weapons systems and crucial data centers? No problem.

Let that sink in for a moment.

https://www.wsj.com/articles/dont-trust-the-chinese-to-make-...

I love a good Trump bashing moment as much as the next guy, but this is inaccurate. The DoD has stringent requirements and quality control procedures in place for their chip procurement. Not to say they couldn't be improved, but the DoD has been aware of this threat for a while, and seems to be mitigating the risk fairly well.

Also, as far as I understand the argument, it goes beyond "Canadian steel is a national security risk". A couple of years ago, Mexico was caught laundering $2B of Chinese aluminum to avoid US taxes.

http://fortune.com/2016/09/09/chinese-aluminum-giant-is-tied...

The theory, from the Trump crowd, is that Canada is also engaged in similar shady dealings with China. If true, that would put the US at risk.

Even if true, how does that make it a national security risk?

The concern was the depression in prices was cratering local high-quality smelting capacity.

However, the DOD issued a memo (penned by Mattis iirc) indicating that there was no supply related concerns. The lack of aluminum used to justify the tariffs was in fact just the result of the LME's anti-market-tampering rules creating an incentive for metals traders to stockpile the material in private stores rather than in LME warehouses.

There was a nice article about this yesterday, but I don't have the link.

It doesn't. They claimed that so they could enact the tariffs, otherwise it would be a WTO illegal tariff. I think Canada/others are arguing that it is not a security risk and therefore is indeed an illegal tariff. This is what I remember from some articles. Please correct/elaborate.

Their argument is that you need strong domestic steel industry to build tanks, ships, etc, in case of war. Not saying I'm agreeing with it just pointing out the stated rationale behind the tarrifs.

If the source is China, in a hostile period they might stop sending it. But according reportings I have read, the US DoD only uses about 0.5% of steel in the US, and only ~30% of US steel is imported.

Maybe because it undermines fundamental US business interests, and therefore wellbeing? If so it still seems like a stretch to me. Doubtful we'd have no where else to source steel if war or disaster struck.

The thing being called a national security risk is a lack of domestic production capability. No one is saying Canadian steel is sabotaged or something.

Yes, obviously- but one of our closest allies shorting us on steel supplies in a time of crisis has never happened, while espionage from China-produced tech is happening right now. Which should be our priority?

Though it is, because it happened with Kobe Steel. (Not saying Canada specifically.)

They faked the data on what it could handle; and with things such as aircraft, lives depend on that data.

That isn't a Japanese steel concern though, that's a steel manufacturing concern. U.S. Steel could do the same thing.

But they are saying that Canada would cut us off from their steel production in a time of need, which is a pretty absurd assessment of the situation.

>Seriously, why are we still outsourcing chip manufacturing to other countries?

I actually agree it might be better for the Americans to manufacture things in America - especially things used in critical government systems.

But this seems like a human problem - if all the factories were moved to America, couldn't those factory managers etc also be bribed?

American manufacturing in the 60s was rife with unions w/ ties to organized crime.

Business culture is different in the US and China. If you tamper a chip in China and go unnoticed you are considered smart & the winner. If you do so here you'll be getting a more negative reaction.

Western manufacturers have a tendency to buy from the cheapest supplier, even if it isn't physically possible to manufacture at that price point without cutting massive corners or making profit off of some externality. However, when Chinese manufacturers build their own brand, you'd be surprised how the business culture suddenly creates a different set of incentives.

Trust alignment of incentives, as opposed to hey-they-promised-an-actually-impossible-pricepoint-on-paper.

I think I saw this on another thread. You're saying if someone bids that they can do something for a price that's not reasonable, it's my fault for believing their mischaracterization? "Fool me once, shame on me."

I much prefer the other mode of operation: "Fool me once, shame on you. Fool me twice, shame on me."

> Sure it's cheaper

That's enough, full stop, say no more. The other costs are real yet they're either not marginal, are borne by others, or both.

You're conflating a few things there. Whilst companies like to make noises occasionally about saving the planet, doing the right thing, making (your country goes here) great etc, it's just horseshit. They exist to make money. That's it.

x-ray would have been easier, as others have said.

but you're talking about the addition of an entire board! probably on the order of 10% of the size of the main boards.

in this article, perhaps dumbed down or altered, they are talking about the addition of a single, tiny chip, too small to even be an MCU let alone have wireless capability (which BTW requires an antenna).

For the folks commenting below that we should bring the manufacturing back to the US, why wouldn't the bad guys just start bribing American workers to insert the attack hardware into devices made here?

It's not like Americans are somehow above being bribed.

Americans can be bribed to do anything. However, you don't really need to bribe or ask someone to betray their country and fellow citizens when it's done on Chinese soil...

The bad actors could be brought to trial in a United States court, which is a level of deterrent not included in offshore manufacturing. If a US manufacturer was found selling tampered chips then the company itself could be held liable. This would create a general pressure to increase onsight security.

There are other benefits to us based manufacturing, but you only brought up the crime aspect so I will leave the other benefits unsaid.

It's possible, but it's much, much, much easier for an American company to hold another American company accountable when something like this happens. Instead of having to go through all those hoops, you sue the pants off the manufacturer.

Wait a minute... So your company has a Chinese equipment supplier, finds out that the supplier is tampering with your purchased equipment, and your solution is to add criteria to the incoming inspection?

No wonder China keeps screwing with you guys. You aren't supposed to eat that cost! Write a PO with tons of fine print that says "We will disassembly units at random for compliance inspection. Non compliant products will be returned at the suppliers expense." And then add a clause that says ">3 non-compliance events in under # months will result in the entire PO (10 or 20 units) being returned and all contracts cancelled."

I cannot believe you are getting screwed by a company you choose to do business with and yet you eat cost to ensure they aren't screwing you. Just get a new supplier! Do on-site inspections at their facility. This is nuts.

There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company.

The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty (assuming the company was involved which they could not have been as they have been the ones damaged the most).

If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

The way to work is then verify everything and politely point it out. If you notice they will correct apparent mistake.

>There were other considerations like the fact we were actually buying it from large reputable company and what happened was that some employees were doing it with no involvement of the company.

It's their responsibility to police for that. From the perspective of their business arrangement with you, it doesn't matter whether their left hand or right hand is evil; it's not your problem either way.

Is it really hard to find alternatives? Are they really cheaper than non-Chinese sources when you account for these inspection costs?

"There were other considerations like the fact we were actually buing it from large reputable company and what happened was that some employees were doing it with no involvement of the company."

I fail to see how that has anything to do with it. The company is responsible for the product, full stop. If they can't stop their employees from tampering with the product, that's entirely their fault.

> some employees were doing it with no involvement of the company

Again, I don't see why that matters. I understand the latter part, but if this is one employee without the knowledge of the company, then pushing it on them forces them to add checks, not you.

Also it comes across a little naive. Oh they told us it was just an employee and would never happen again, OK!

This still sounds nuts to me.

Two thoughts:

> they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

This is how children behave. Still feels like you’re rewarding bad behavior. You’re enabling them.

I think it’s more that you’ve valued the low per unit price over having a healthy contract. Your vendor is incentivized to make all of their money off of externalities and your company think it’s cheaper to outwit them than to demand QC on their end. Whose brand will be sullied if you miss some of these units and a customer finds the spyware? Not theirs.

And second thought, shouldn’t serial numbers be coming off the line in ascending order? The kind of work they are doing would require taking parts off the line and putting them back later so odd lots of SNs are the ones you need to verify.

You could also be mandating how many boards are allowed or that the SNs go on early in the build process.

Altering components to something cheaper is common practice for every company that is willing to exploit the fuck out of their customers. Look at the food industry. In EU there is control, in US thare are sanctions. But it is up to the governments because companies are to opportunistic to change. As long you can (re)sell the junk you get, you're fine, if you can't you upgrade "QA".

Both the EU and US have internal controls on food quality and sanctions.

Contrary to the impression your post gives, the EU has much more "sanctions" against food (esp. imports) than the US does, and is generally taxing much heavier and restricting many more things. Importing essentially any milk product into the EU is only possible on an exception basis. Trump has a LOT more work before extra sanctions in the US will become comparable to the EU ones.

What is often not said is that the EU not only has sanctions on its borders against food products, it actually has internal sanctions and regulations that are purely political in nature.

For example "champagne" in the EU can only be made within a particular region in France. Which leads to funny, in a sad way, consequences:

https://www.theguardian.com/business/2017/feb/15/eu-fears-in...

This applies to many products. Wines, cheeses, cakes, fruit, meat, potatoes, candy ... and is generally absolutely ridiculous:

https://www.foodmanufacture.co.uk/Article/2014/11/13/Interac...

For your other part of your comment, I am unwilling to give a real answer. Obviously the US checks it's food quality. It's made different choices than the EU, famously the US allows hormone "enhanced" meat, but it does check food quality and safety. There are heavy penalties for breaking them.

and no kinder surprise in the us.

Champagne is literally the name of the area where the product comes from. It makes perfect sense that anything that doesn't come from the region Champagne, isn't champagne... Call it sparkling wine, it's fine, but not champagne. And it applies to most local products. Camembert? Comes from a specific town, which is called Camembert. Beaujolais? The name of the province. And the list goes on. It doesn't sound "absolutely ridiculous" to me.

"generally absolutely ridiculous"

Did you look at the map? It's nearly all things with a place name in the product name. For example, "Welsh Beef". Why is it reasonable for a consumer to buy Welsh Beef and for it to possibly have come from somewhere other than Wales?

It seems to me, more like generalising the concept of trademarks to place names rather than just company names and brand name.

This is just how low trust societies function for hundreds if not thousands of years. This is also why families tend to be much stronger in these places. The American idea of a strong family is different from what I'm describing, since it just centers on immediate family. My definition of strong family bonds, implies both closeness and dependency not just on your immediate family, but even on your distant cousins on both sides. It's close if not the same to how Mediterranean and Hispanic cultures view families. I guess you can call it a clan mentality. I'm now wary of countries where families are really strong ie clans. It's not always the case, but it tends to mean the rule of law is weak and corruption is crazy.

When you can't even trust that food is real, and when you have to bribe even low level provincial government employees (can you imagine needing to bribe DMW workers or even police?) - what else is an average person to do except to treat it as the norm in order to survive? Unless you're an elite, the only other option is to leave, and not everyone has that choice. Very little is considered wrong over there, aside from criticizing the powers that be; as long as it helps your clan. It doesn't help when Western companies and governments look the other way, as we see in past news stories and even the comments here.

I'm not condoning the behavior; just explaining.

Not much to add, but want to note that you're spot on. I've encountered this SO many times over the years and this is always the explanation I'm given. At first it just made me indignant, too, and to some degree I guess it still does, but now I just try to let it go.

It's not rewarding bad behavior when you take it into account in the deal. Moralization is useless when you do business with different cultures. When you notice something what would be out of ordinary in your culture, you estimate the cost and find out how locals solve it. Different cultures have different kinks and people living there know to adjust to them. It can be fun to figure out how other cultures function.

From a naive Finnish perspective you could see American business culture as dishonest. People lie all the time and promise things that are false and you need massive amount of legalese or they fuck you over. When they say something, you can't trust it as much as you can trust a Finn (or Swede). American way is to lie to your face in pre-negotiation but then be truthful to the legally binding agreement. Germans or Belgians are more truthful beforehand and they stick to deals better without legal enforcement looming over them, but require more detailed written deals than Finns. Who is right? Should we punish Americans for their behavior or adopt their culture?

Of course not. You must understand that that the level of acceptable exaggeration, overpromise and bullshit and trickery is strictly culture dependent and relative. Someone is dishonest if they "lie this amount above average" in their cultural context. After you make an adjustment, you know how to get the truth.

Chinese manufacturers have the capability to stick to standards, take responsibility and deliver high quality stuff. It's just more dependent on personal relationship. Just placing an order has less trust as default. Either you develop "quanxi" with your suppliers or work trough some third party that understands both worlds. Third party understands what you need and knows how to get it from the Chinese suppliers because they have a relationship.

Chinese pay the cost that comes with their business culture. They can't network as fast as you can in open western cultures with more default trust between strangers.

It is called business. The company's ONLY duty is to bring profit to its investors and management is legally bound to maximise it. Now, the definition of profit may differ as well as what is profitable and what is not, but the company is legally obligated to work to bring profit and if you throw a good deal because you don't like it you better explain it to your investors.

Whether you like what your supplier does or does not is of no concern. If couple of units can be detected and written off it is treated as cost and you move on to decide if it is still profitable and whether you can get better deal somewhere else.

You let you emotions rule and it just means you are not fit for the job.

All employee actions are company actions. You partnered with a company that can’t control what it’s employees do? No internal audits to make sure their reputation wasn’t being tarnished by a few employees?! Your loss.

I understand the indignation etc etc. And the suggestion to not use these kind of companies anymore.

And that sounds really reasonable, until you realize that pretty much all contract manufacturers in the Far East will source cheaper or off-spec components than those on the BOM if they can get away with it.

One of my friends supplied small widgets for a well known consumer electronics maker. He routinely gets widgets returned to him as defect for inspection, which then inevitably turn out to be clones of his widgets.

The only way to make sure that your product rolls of the product line as expected, is to have people on-site with continuous inspection (and pray that they're not the cousin of somebody who's on the other side.)

If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.

How much do you think you would have to pay them to make it worth their while to not use cheap components (so as not to risk losing you as a customer)? It’s amazing how people never consider using economics.

Edit: But let’s not get away from the matter at hand. The issue wasn’t cheap components, it was full-on credit skimmers installed in yor hardware (IIRC). Should be easier to incentivize the conpany into halting those before they get shipped to you, no?

As somebody else noted: it doesn't have to be the company that does it, but a lower level employee who cuts a deal. So paying the company more wouldn't solve the issue completely.

All of these things are extremely cost sensitive. Your suggestion that people don't consider using economics is simply wrong. If you can manufacture a million pieces for a few cent less per piece, and the only negative is having to paying a bunch of inspectors who are fixed cost, it's an easy choice.

If the solution to the problem involves line checks, independent audits, personnel checks, increased pay, etc., thenyes, throwing money at the problem is a large part of the solution.

And again, you should make it the company's problem if they can't control their own employees. They can learn to put some controls in place once it starts hurting their bottom line.

> All of these things are extremely cost sensitive.

Say their profit margin is 1%. If you offer to pay them a 1% higher price, you're offering to nearly double their profits on whatever they sell to you.

Admittedly, that only works if you are a sufficiently large portion of their business. If you're a tiny percentage, then it may take a lot more to motivate them if their other customers aren't willing to pay more.

You have to pay them as much as it would cost to manufacture in the Western country doing the out-sourcing...so unless the cost of these extra quality checks outweighs the cost of on-shoring you might as well measure angular momentum.

Madness if you ask me but there we are.

> I understand the indignation

IMO this is not indignation, it's supplier sourcing 101.

> If you want the benefit of dirt cheap manufacturing, you need to have a system in place to deal with these practices.

I will definitely agree that holding suppliers to standards regarding consistent output, unadulterated products, and conducting audits all make production much more expensive.

But if you're a device manufacturer, these sourcing controls are key to shipping quality products.

lol. you haven't worked with a vendor in China before, have you?

But make no mistake, this yet another globalized race to the bottom, but one that hits capital (or the end consumer), rather than labor or the environment.

the indignation was not that. you are still missing the point.

after the 1st device found, you should have contacted the manufacturer and said that you will start a department that has the capability of opening the device, inspecting and re-sealing in a way that it won't impact any guarantee the factory provides. If they denied this very sensible request, you had proof that it wasn't a isolated employee doing the hack.

Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.

> Not really familliar with PCI DSS but it might be that the card-readers/terminals aren't PCI-compliant if opened? So not the manufacturer's issue but the customer's.

I think that's the case. The EEV Blog guy did a teardown of and old one once and pointed out the numerous tamper-detection features that would clear the device if opened.

However, if I were the customer here, I'd tell the supplier that from that point forward they need to supply me free extra product with my orders, so I can do my own random destructive testing to look for implants. I order 100, they send me 105 for the price of 100.

You all recognize the irony, right?

PCI DSS allows for "Mitigating Controls" if you need to deviate from specified requirements, provided it is well documented and is equal to or greater in security. Doing teardowns to review circumspect hardware, and applying one's own tamper protection deal (and with accompanying documentation and tracking/logged information) would very likely be sufficient to maintain complaince.

>The fact is, doing any kind of hardware production in China, you have to be aware Chineese have different value system and you would not be suited doing any business if you throw tantrum at any sign of apparent dishonesty

Sounds like the solution is not doing business with them and pushing for a ban on others doing business with them (since this largely has a socialized cost when things go wrong, such as individual people having their credit cards stolen).

OMG, that is so so so so brilliant! I'm going to go tell my boss right now that the suppliers better suite up, otherwise we are going right to their more expensive, less experienced vendor and will delay our product launch for a year....and likely still suffer the same problem. Those Chinese vendors better shape up or we're going to really teach them a lesson by driving ourself out of business right quick!

Just imagine for a second if someone was advocating on HN for the use of a web framework that has knowingly allowed itself to be compromised to steal passwords because it was cheaper to use.

How about a compromise? If any of your customers are a victim of a crime because you continued to use a shady but cheap vendor after seeing them trying to slip past tampered hardware, your company is held fiscally and criminally responsible.

So no more using Intel?

You mean like couple of well known platforms that let you reach their users and in the meantime steal, mine and monetize user information?

You can be as sarcastic as you want, but these stories are absolutely indicative of a much larger problem. It might be beyond the capacity of one company to fix, but in the aggregate they represent a serious political and economic threat and need to be dealt with one way or another. Preferably by literally anyone other than Donald Trump.

Speaking as someone who is _not_ a citizen of the USA and is well aware of his issues, he seems to be first US President who actually appears to have the guts to do something about it. I doubt an establishment politician would ever have imposed tariffs on 200 billion dollars of China imports.

The tariffs aren’t the problem, it’s the stupidity of going it alone and alienating allies that could have turned the thumbscrews with you on China to get them to really capitulate.

If he stuck to just China I would agree. But by trying to change up numerous different agreements, he lost the goodwill needed to have a collective crackdown on China. Even if he is the first to be willing to pick these sort of battles in a long while, he has a new issue of picking too many battles. Seems like a Goldilocks problem.

Leaving an abusive relationship tends to be scary, especially if the victim is convinced that the abuser is the least bad. Doesn't mean the thought of ending that relationship should be dismissed out of hand.

I used to work on the server-side stuff for telecom devices. We designed hardware that went into customer homes, but only downloaded the (encrypted) firmware upon home activation as otherwise the Chinese manufacturers would have ripped us off and sold them to telecom companies without our cut.

So, realising they could copy our hardware, but didn't have our software, they responded by trying to hack my servers, multiple times, from the same IP they sent manufacturing data from. A quiet word with their management would stop it, and it'd start again a couple of days later.

These people have no shame, and if we are going to go for lowest cost at all times it is what manufacturers should expect to happen.

I am reading it as "we don't any any option outside China and we have to live with this". I guess even after adding cost of doing additional of products you buy from China its less than what you would pay to buy it from other place.

Thinking about it more, what you are doing make sense also because tampering can happen with any supplier, irrespective of factory location, given how aggressively states are pushing for spying on each other and their people.

Sounds like a libertarian oasis.

>If the company does screw you (like replacing components for something cheaper) they typically will not be thinking they are doing anything wrong. They are just testing if you notice and if you do not they will say it makes no difference for you but saves them costs.

They don't pass the savings to you so they know very well what they're doing, just playing fool to see if it slips.

This is also such a good anonymous story that I'd give decent odds it's made up.

He's not anonymous, though.

I've heard very similar stories from people in the card industry before. I don't think it's made up, unfortunately. Card readers from China are systematically untrustworthy.

Using the technically incorrect term "angular momentum" rather than the more correct term "moment of inertia" made me think the same thing. I would think a person tasked with building such a device would know their physics well enough to use the right term, but I may be wrong.

Maybe he is afraid of giving too much info away. By mentioning it, now the opposing side will be measuring angular momentum.

So now he needs to make a new test. Ow that that is out of the bag

The fact that they didn't just run the units through an xray machine would seem to support your theory.

Probably the cost per unit is too attractive

Also it forces their customer's customer to essentially subsidize production cost.

Some amount of financial theft happens to random consumers who don't know where it came from and this stolen money is paid in bribes to employees as a subsidy to their low wage.

As a result this immoral company externalizes aspects of their cost structure to greater society.

he's saying that his company knowingly allows a percentage of fraud, identity, and financial theft to occur against his customers' customers because there are enough layers of indirection where they (he and his bosses) probably can't be held personally responsible. Enough efforts (fancy tables with big receipts, etc) have been provided for coverage in case of a court case or media blitz but actually using a reputable supplier is clearly too expensive.

I can't believe the parent comment is so brazen. It makes me physically sick.

I guess it's news to you that the financial sector and especially banks in fact do exactly that:

https://www.creditcards.com/credit-card-news/credit-card-sec...

But it doesn't stop with the cards. ATMs, Tax Fraud, ... almost all businesses within banks have to deal with fraud and there is some threshold that they just allow to happen.

knowing about it and willfully facilitating it are two different things

like shoplifting. Most large chains know exactly how much "loss" they experience, and they just raise their prices to cover it. Theft and fraud is a part of life unfortunately...

Maybe you and me are missing something here. It seems crazy that somebody would go to these lengths without switching suppliers, there must be an underlying reason that is assumed to be understood by the informed reader. (But isn't understood by me and you it seems.)

Profit. Changing suppliers is going to cost money.

Or: you can expect this kind of shit from pretty much any supplier. Even if you go to a western company, they all source their components from China anyway.

A good read is 'Poorly Made in China'.

The product is different, but the problems are the same.

* They drop changes and problems at the last minute, so you're over a barrel with your customers.

* Relationships take months, maybe years to build. Switching suppliers is a long and costly exercise.

* Often suppliers themselves are in communication, so your attempt to build a new relationship is scuppered by your current (corrupted) supplier before you've got far.

* The new supplier will 100% play the same games.

* Suppliers/factories are often expert at these games - they're not idiots, their bread and butter is manipulating managers and big companies into corners and fleecing them along the way.

* Large suppliers often have strong local/regional government connections. Once you start causing serious problems, they'll have no trouble causing massive, massive problems for your business.

* Are you going to admit to your customers and bosses that your products were faulty and you knew?

It's a poisoned chalice. Companies want low cost manufacturing, and sometimes China is the only place a product can be manufactured competitively. But this kind of behavior is standard for Chinese companies.

> * Are you going to admit to your customers and bosses that your products were faulty and you knew?

If you can't answer that with a yes, maybe you don't have the backbone to work in anything critical.

When you discover a fault in something, particularly a fault that might hurt someone, you have a moral obligation to speak up. To do otherwise is cowardice.

Failure to speak up when we see shit is how stuff like the VW emissions gate happen, and also why security professionals can make a career out of ferreting out your mistakes and engaging in responsible disclosure.

All this tells me is that Made in the USA is more valuable than I once believed.

Not having the backbone to reveal it when you discover a fault is not what that comment was getting at

I think you’re right about the value of Made in America and societies where honesty (even about problems or risks) is valued.

The book I referenced included a number of instances where factories actively undermined individuals so that they had very little choice than to either go along with the situation, or press the ‘nuclear’ button and scupper the entire arrangement, including their own livelihood and career. That’s a tough choice for people to make.

This problem is not specific to China, it's a problem of going with the lowest priced supplier. Your proposed recourse presumes that the supplier is making enough money on the deal to engage in a contract with teeth... many are not. Also, even if they moved everything to the U.S. they'd likely have different flavors of the same types of issues (i.e. did the NSA embed someone in the company to implant backdoors etc.) Fraud, sabotage and espionage aren't confined to specific countries. When greed drives you to move operations to the absolute cheapest places you can find on Earth, that's not going to be without its own set of risks and problems.

Honestly, to anyone bashing GP, look into your pockets, laptops, watches, cars, TVs, routers, CCs, singing toys, bitcoin mining ASICs. All with sealed black-box chips. Sure we can x-ray a couple of randomly stripped chips, but each one?

Now show me alternatives when most consumer grade electronic parts are fully or partially made in China.

This reminds me of a story about, IIRC, Soviet intelligence personnel determining that a photocopy machine at their consulate in the USA was bugged by the CIA by measuring it's weight and comparing that value to the standard value published by Xerox.

Modern times, but same old methods of "debugging"

The angular momentum stuff is innovative though.

Now I’m picturing a 300lbs photocopier being spun on three axes. How much tape would it take to keep doors and parts from flying off?

is measuring the moment of inertia not standard? I think any mechanical engineer would suggest it for that kind of problem (disclaimer; I've suggested it before for the same problem)

>measuring angular momentum on a special fixture

Can you elaborate on this concept a bit? I'm not familiar with this term and the sources I looked up were pretty physics-y and out of my depth.

This sounds like a very interesting and creative solution. Good lateral thinking on your part! :)

This feels cargo cultish. Products drop from the sky. One day they become poisonous. You have no idea how to reproduce them locally. So you come up with hacks to make then less dangerous.

We really need to get back into manufacturing if this is our brave new world.

The better question is how did we stop manufacturing.

What good is angular momentum when the producer can have fluctuations in its supply chain? Yes you can see when devices are not the same, but what if that happens all the time, legitimately?

I do not for one minute believe, that china, whose production floors are spotless, watched like a hawk and every product tested, did not know this was being added. They would have been able to tell themselves by weight. Yes, product weight is used for shipping manifests too.

This is obfuscation of the fact, that china as a state actor has perpetrated this crime against our country period. full stop.

Thanks for sharing this story, and I hope you aren't put off by the huge thread of people second-guessing your competence. We need more of your kind of story.

Was the secret service contacted? Card skimmers are a big no no. Family friend works for Dept of weights and measures and finds skimmers all the time on gas pumps. Scary stuff.

Is there an article that describes a bit more in detail what the chips actually did (or were capable of doing)? They only say "the microchip altered the operating system’s core so it could accept modifications.", which I might interpret as circumventing signature checks to allow installing modified firmware on the systems? But how does the chip connect to the network and how does it receive commands?

That said, it's pretty scary that you can hide so much malicious functionality in such a small device, makes me wonder what might be hidden in my Lenovo. In any case it speaks highly of the auditing firm that they were able to locate this. I wonder if they performed an x-ray analysis of the board, as given the size of these chips it should be possible to embed such devices in one of the internal layers of the board as well, making them essentially invisible to optical inspection.

SuperMicro hardware has very extensive IPMI integration into the mothebroard, which amongst other things can take over and inject frames into the network interface, emulate a VGA device, talk to the CPUs serial lines directly, flash firmware, control the state of a number of physical devices- and this is what it supports just from the web interface it presents by default with the password "ADMIN:ADMIN". My money, based on experience attempting to harden their devices, is that any modification were injected into the IPMI hardware where most of this was already supported.

This stuff ends up being extremely difficult to disable. The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC invisibly to the OS and DHCPs on its own to open up an administration port, web interface, and some assorted call homes. You have to explicitly tell it to use the non-connected port, change credentials, and modify it so that it is not accessible within operating system as well. Hopefully while the machine is offline to prevented any automated scanning finding it within your network.

The number of times I'd end up nmaping our local networks and being able to remotely access production hardware with an interface that allowed me to reach this interface was maddening. The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely nu-authenticated avenues exposed. The course of action was always just to write off the hardware and bin it, because god knows what impact you could actually have using that interface.

OpenBMC (Facebook, Google, Microsoft, Intel, IBM and others) is working on open-source baseboard management software, https://www.linuxfoundation.org/blog/2018/03/openbmc-project...

> The organizations behind the new project each have already made substantial contributions to creating open source baseboard management controller (BMC) firmware. Now, working together, they will define the vision for a standard stack that can be used across systems and computing environments.

LinuxBoot and Open Compute OSF are working on open-source server firmware that can be measured on every boot and validated against hardware root of trust keys controlled by the server owner instead of the server OEM, https://www.platformsecuritysummit.com/2018/speaker/hudson/

Wouldn't help. The BMC hardware has direct serial access to CPUs and other hardware in the machine. Communication is unencrypted. A hardware modification attack wouldn't touch the firmware at all and could still compromise IPMI functionality.

Reducing the attack surface does help -- you're making perfection the enemy of the "somewhat better".

No, against determined adversary it doesn't.

Even now, supply chain hardware modification attacks remind people of fiction. However, the number of people known to be affected by buggy BMC firmware is orders of magnitude larger, as described two comments upthread:

> The system is basically designed to be as insecure as possible by default, and allow for the maximum possible persistent threats with BIOS flashing, IPMI flashing, and other completely un-authenticated avenues exposed.

It sometimes feels like certain hardware protocols were designed to be insecure. I remember reading about IPMI issues back in 2013:

https://www.itworld.com/article/2708437/security/ipmi--the-m...

SuperMicro hardware in particular always struck me as such. Asking users to pay a license fee to be able to update the BIOS on their devices (after paying tens of thousands for the hardware itself) is a kick in the teeth.

https://www.virtuallifestyle.nl/wp-content/uploads/2016/08/S...

http://www.supermicro.com/products/nfo/SMS_SUM.cfm

To be clear, this is only for the ability to update the BIOS over the BMC interface, not for BIOS updates in general. (unlike some other vendors, where you need a support contract to be even able to download the updates)

At least on some boards you can boot the USB drive image containing the BIOS updater through the BMC and do a remote update that way.

For the models I was looking at a number of years ago, the options were through IPMI or via a USB Floppy Drive. Perhaps my memory is failing me here, but I seem to remember being quite enraged at the prospect and would have done a lot not to have to pay the license fee on principle.

Supermicro doesn't publish changelogs for their BIOSes. Their disclaimer on their BIOS download page says, "don't update the BIOS unless you have to". It's a ridiculous stance for a server company. You buy Supermicro over Dell for the price. That's pretty much it.

Price, variation, and time to market.

"Designed to be insecure" is probably unfair to the designers of IPMI. Security was just not as big a concern as it is today.

This is only really valid for protocols or products designed before the Morris worm of 1988. Anything designed beyond 2000 has no excuse for not thinking about internet security.

Well, IPMI isn't supposed to be exposed to the internet. Best practices have you running your BMC's on a completely separate, highly locked down administrative network.

Well yes, typically you'd 1) configure IPMI to use the dedicated port, 2) put those ports into a VLAN with no outgoing internet access. But since this is BMC, what's stopping it from just using your management or production port to fire off its secret phone-homes and whatnot?

Best practice would be for the BMC to not have access to the regular network ports when it has a dedicated network jack. All the ones I've looked at don't have any kind of physical interlock or switch, it's a software interlock.

Now even a hardware interlock could be subverted, but that's harder than sticking code in the bmc firmware, which does tend to get updated during the life of a server.

I did a security audit on a company that had a setup similar to this.

After popping an internet facing web server, I was able to compromise the IPMI system and use the management network to bounce around to any server in the enterprise completely bypassing all the firewalls and segmentation on the production network.

Management networks need rack level isolation.

Computer security as a field and area of concern is significantly older than IPMI.

Cannot it be physically disabled with a jumper or maybe by cutting some lines on the motherboard?

No. SuperMicro make some models which appear to be for government application which lack the ASIC completely, they were available on NewEgg with no further description of purpose at one point.

https://my.mixtape.moe/sorqql.jpeg

But without the IPMI kernel modules loaded, IPMI is harmless, right ?

No, since it can configure the BMC, which works without the OS.

No, the controller runs it's own totally separate OS, has connections to most/all the buses and is able to emulate devices whether you have drivers installed or not.

No, it boots prior/separate to the board itself. It's basically a mini-PC embedded in the board that has it's own CPU/Memory and tentacles attached to everything in the mainboard.

If people are interested in digging into how IPMI works every SuperMicro board I've ever seen uses some variety of ASPEED Baseboard Management Controller (BMC). Facebook uses the same chip in their open hardware projects.

The problem with an IPMI BMC is that if you have malware that roots the OS from the BMC that in turn roots the BMC to reinstall itself from the OS, then you can never actually get rid of it. I actually opened a bug with a major clustering vendor in about 2012 because of this. Their response was a docfix.

> The naive approach would be to not connect to the dedicated NIC that's indicated on the back and in the instruction manual, but if you do this it masquerades onto the main NIC

cool, thanks for that info.

> just to write off the hardware

maybe you could just standup the mgmt network but blackhole route it at each switch port. The mgmt NIC thinks it's working properly but it can't talk to anyone nor can anyone talk to it.

at the expense of a dedicated switch.

Or, if the server has a free PCI slot, stick the cheapest possible NIC in that.

  5. When a server was installed and 
     switched on, the microchip altered 
     the operating system’s core so it 
     could accept modifications. The chip 
     could also contact computers controlled 
     by the attackers in search of further 
     instructions and code.

So, in typical vulnerability/payload/exploit fashion, the board's bus is vulnerable by default, because the chip pierces all the usual lines of defense protecting against network and operator I/O. It carries a payload intended to target very common features used everywhere commodity servers are used, one that likely listens for DMA traffic on the bus, and alters the signal stream, by escaping upon the occurrence of a magic sequence, and inserting its own signal, before resuming the authentic stream in flight.

The payload could be pretty small, since the server boards are likely using OS packages that match the chipset. This limits the software to a small set of well known targets, Linux, Windows, Apple. Target their kernels, and you only have to snip out a small chunk of bytes, and splice your own pre-defined package in. Splice in a miniature runtime, that operates a turing complete set of operations, and open up a listener that waits for network access, and now, your payload can enable arbitrary code execution, irrespective of permissions.

Now, to exploit, the payload needs to time the opportunity to splice itself onto the disk correctly. If certain well-known chunks of code will always exist in each given operating system, then with every disk access event one just needs to wait for the inevitable moment those magic system-specific bytes travel over the bus, in order to replace the known bytes with the poisoned modification. Events might target when the bytes are originally installed with the OS, or every time the OS reads those known bytes back into live memory, from any source.

The total payload package could probably fit inside a couple of megabytes, pack on a few more for the "listen & splice" part of the attack to round out the entire mass, and all we know how much data an SD card can fit into say... five grains of rice?

It's not particularly magical, there's consumer chips around which are not a whole lot bigger (though obviously in a more standard package). You don't get a lot of resources, but you don't really need it if all the other frameworks are in place in other software. If this sort of thing is something you can buy on Mouser for a few cents, the espionage grade material is probably an order or magnitude more higher quality.

https://www.microchip.com/wwwproducts/en/ATtiny4

For scale, this alone is about the size of a large SMD capacitor and would basically be lost in most designs today.

Figure it's custom silicon, given the nature of the story, and "magic" in the sense of "magic number programming" to time the attack.

https://en.wikipedia.org/wiki/Magic_number_%28programming%29

For example, looking for ELF or Portable Executable headers, as a crude estimate to determine attack opportunities. In this case, the magic numbers would probably be more selective and sophisticated, but still have an aspect of hard-coded values, since we're talking custom silicon.

For another example, I have a couple of these which are a bit bigger but have an ARM SoC and onboard Bluetooth (with antenna):

https://www.digikey.com/en/product-highlight/t/taiyo-yuden/e...

That's a crazy module, I had no idea anything of that scale existed for Bluetooth radios.

This sounds like speculation. I'm quite capable of coming up with my own unfounded speculation, but there is a real report out there with the actual details in that really needs to be made public, legally or otherwise. There ought to be a CVE about this. Where is it?

They didn't do anything to the CPU, what they did is the modchipped the line from EEPROM and the board management controller.

They probably found it out when they were repeatedly tried to reflash the BMC flash, and saw that checksums did not match.

That would make a lot of sense and would give the attacker a way to interface with all of the other hardware (network, disk etc.). Do you have a source for this information?

I looked up supermicro blade motherboards, and saw that the chip was right near the IPMI chip's line to spi flash.

And prior to that, there were already persistent rumors in the Chinese interney of certain Chinese mobos sending "weird garbage on ICMP," and "BMCs that somehow boot and work with their flash memory soldered off"

Remembering that, I might even suggest that this is not a modchip that does something with signal on the go, but just a very tiny flash chip that has the modded firmware.

Going further from that, to pack, say, 16 mB on a sandgrain sized chip, the densities need to be like that of best flash chips out there, which also means that they have access to last gen flash fab.

A photo of such a motherboard with a big arrow pointed at the additional chip would be a useful addition to this discussion.

Supermicro 6128 aka x10 series microblade. Those were very popular among Chinese DC operators during Broadwel era.

https://www.itcreations.com/dist/landing/i/MBI-6128R-T2/MBI-...

Left of the sata connector. An empty space with 8 pads for an smt eeprom or flash. It is occupied by the thingy on bugged boards.

Right below is the Aspeed chip - the BMC

Is there a list of known compromised Supermicro SKUs?

There really, really needs to be. If this story is real and things have been known compromised since 2015, it’s unacceptable that this information isn’t shared more widely. The “we’re going to follow them” idea doesn’t hold water over 3 years, countless small vendors using Supermicro and not being well-connected enough to know (like Apple or Amazon) about it.

An update on that theory: AST2400 has option for two SPI memories, one main, one "recovery."

https://download.csdn.net/download/duanzhang512/10385038

The recovery overrides the primary if detected by default.

The place they put their "filter cap" is right on top the empty TSOP8 pad for the recovery flash. And they probably ordered the factory to sneak the traces just a little bit more, or put hidden vias under it, or simply had somebody very dexterous to solder it to pads with hair thin wires.

eh, does this mean that the motherboard is not tampered with ?

Well, it means that provision for the second flash was already there, and PLA simply exploited that fact that Aspeed chips are virtually omnipresent in higher end servers.

It also means that the extend of intervention into board design was minimal, and that a trivial automatic xray would not have picked it up. And as implied in the article, later they buried the bug to beat the AOI, if it was done higher upstream.

So, they would've been screwed even if they were doing board testing outside of China.

That's a clever trick.

But the sole fact that the chip has "to phone home" makes detection trivial, and puts the usefulness of the method to nil - anybody sees the router blink when it shouldn't and your bug's cover is blown.

This is the most plausible theory I've read in this thread. Assuming the image in the article is a stock image (there isn't yet a clear image of a definitely compromised board), then the added part could simply be another TSOP8 Flash part. This implies the firmware to the AST2400 is unsigned (which it appears to be, as there's coreboot options for it).

That makes the whole thing gloriously simple. A part "stuck on" afterwards is obvious. A part fitted into a no-fit footprint after optical inspection is not, it looks exactly as if it was meant to be there.

To me it looks like that bloomberg intentionally photoshoped chip labels to avoid risk of upsetting their manufacturers.

This would hardly fit the description of a chip "smaller than a grain of rice", though.

Is this datasheet available outside of CSDN? Do you by chance have a copy you would be willing to share?

Hm, but DMA messages get distributed over a parallel bus and this chip seems to employ a serial interface, so I would assume that it's not directly connected to anything that requires high throughput (i.e. memory, disk and peripheral access).

The article says:

But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

To me, that makes it sound like they could download from a remote host and inject code and do literally anything.

If this is indeed the case, I'm surprised someone didn't catch something earlier when the device was calling "home" over their network. I'm wondering if China stole BlackRidge First Packet Authentication tech [1] to keep things dark. BlackRidge is... "involved" in IC and defense projects.

[1]https://patents.google.com/patent/US8346951B2/en

Not the chip itself but there's an investigation which I've already submitted:

https://news.ycombinator.com/item?id=18144519

You're not the only one worried about Lenovo.

https://freebeacon.com/national-security/military-warns-chin...

They attacked the Base Management Controller. There's an article by Bruce Schneier from 2013 warning about exactly this attack. Quoting:

"Basically, it's a perfect spying platform. You can't control it. You can't patch it. It can completely control your computer's hardware and software. And its purpose is remote monitoring. At the very least, we need to be able to look into these devices and see what's running on them."

https://www.schneier.com/blog/archives/2013/01/the_eavesdrop...

> They attacked the Base Management Controller.

Do you know this, or are you speculating?

It's in the article:

"The illicit chips could do all this because they were connected to the baseboard management controller..."

Can you quote a single source from that article, or is it all anonymous?

That is also in the article...

"17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information."

So not a single source. Might as well be going to war over invisible weapons of mass destruction. Also, Apple and Amazon both has said they do not agree with these claims. So far this is nothing more than propaganda.

so you prefer to believe official corporate statements saying they didn’t have security leaks, rather than a news agency with 17 independent sources confirming they had ??

That’s an interesting choice.

17 independent sources doesn't mean anything. Again, this is the same as the sources that said their were weapons of mass destruction in Iraq. Companies and government use propaganda all the time, so until you have more information confirming these claims, then it is best to hold off before trying to say that Apple and Amazon denying the claims str proof that they are real.

Are you saying Bloomberg made the sources up, or that they exist but are all lying, because they won't go on the record?

Def possible they are lying. There are a lot of companies competing with white box manufacturers. If you can't name a single source, and you have two major companies saying these claims aren't accurate... then maybe you should hold off before making judgments.

> You can't patch it.

Sure you can. OEMs regularly release patches for platform BMCs.

Not sure about you, but I'm not an "OEM".

Well, you cannot patch the vast majority of the software in your computer (assuming you are like the vast majority of users using proprietary crap for everything). That does not mean it is all unpatchable. If supermicro care, they could release a BMC update, for example.

"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

Well played, Bloomberg. Well played.

I LOL'ed when I read that line too

To be fair, removing the last three words could make it sound like the Mormon church was beaming sermons to the adult film industry which would probably be even worse...

I laughed at that line too.

Interesting about how some of the trojan chips were hidden in PCB substrate layers to avoid optical detection. For a much stealthier approach again, it was shown in 2013 that slighting changing the dopant mask for a few gates on Intel Ivy Bridge chips could render RNGs insecure[1][2]. Mask inspection systems[3] are used to detect mask manufacturing defects, but the question then is, do those inspection systems use Super Micro motherboards?

[1] Paper: http://www.emsec.rub.de/media/crypto/veroeffentlichungen/201...

[2] Presentation: https://www.iacr.org/workshops/ches/ches2013/presentations/C...

[3] Example system: https://www.lasertec.co.jp/en/products/semiconductor/mask_se...

We need to get the fuck out of China. It is becoming less credible to throw our hands up and say "China has all the silicon manufacturing, guess we have to put up with it!" - this is national security, both directly via hardware in the DoD and through our economic stability.

Saying "Well the Chinese companies are different" or "It's just rogue employees" or "We just have to accept it" is not good enough.

We need a little bit more economic nationalism and realize that this shit matters. How is it so shocking? How did a Canadian security firm find something before the CIA/DoD? How is this not a NatSec program to be inspecting this stuff through shell company orders and honeypots, like we have on the Internet?

How do you know that the DoD/CIA isn't behind this?

Because it came from Chinese manufacturers. Did you read the article?

Why does that matter at all? How do you know the CIA didn't use Chinese spies to do this? Although from a different agency, the motto "Nothing Is Beyond Our Reach" is telling.

What evidence do you have to suggest that?

None at all. I'm not saying it is the CIA. But it isn't like they do't have a history of wanting to spy on anyone they could.

my bets on the lizard people

> It is becoming less credible to throw our hands up and say "China has all the silicon manufacturing, guess we have to put up with it!"

The thing is, China doesn't. Most expensive chips are fabricated outside of China in nearby countries and shipped there for final assembly.

What goes on in China is usually human assembly. And that's what seems to have happened here -- in the process of putting chips on motherboards, someone added some bonus chips. It seems like it would be hard to do that without dedicated traces / solder points.

Relevant: https://www.wired.com/2011/11/counterfeit-missile-defense/am...

So the chip shown in the article looks like a typical SMD balun, it is a type of transformer used to adapt impedance between two transmission line. It’s designed to replace a series a lumped element (capacitor, inductors, resistors) normally used for impedance adaptation (in a T or Pi network). The most common used for the device is directly between an antenna an a RF front-end to serve as an antenna tuner.

Technically you could embed and power an RF front-end inside a “flavored” balun to intercept or alter any communication passing through that front-end or even use the antenna to communicate during the down time. So this literally would hack Wifi / Bluetooth at low level and inject code and at the same time create a mesh network of malicious devices to relay information. Welcome to IoT Cyberwarfare.

But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines. Common mode chokes (some SMD chokes also look like the chip they are showing) or even some integrated ESD protection solution would be an ideal target as they are inserted in series of the signal.

Wow that’s pretty neat.

Terrifying and neat; thanks for the explanation!

How easy is it for us (or rather the companies making the hardware) to detect and mitigate this?

I don't know enough about this, but isn't the article saying that the appearance is deceptive:

The chips on Elemental servers were designed to be as inconspicuous as possible, according to one person who saw a detailed report prepared for Amazon by its third-party security contractor, as well as a second person who saw digital photos and X-ray images of the chips incorporated into a later report prepared by Amazon’s security team. Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment.

If the photo in the article is real you wouldnt be able to identify this component as compromised "just" by visual (even xray augmented) inspection. TVS Diode Array looks the same from the outside, whats more its build in same way with silicon die embedded in tis structure. Other than signal analysis it would take decapping every single component of a motherboard to find this implant.

How specialised? ohmmeters inductance capacitance meters are not exactly exotic.

The chip shown in the article most likely just something journalists found in photo stock by keywords.

Yes, TVS Diode Array (image google to see they look just like in the article) for ESD protection is the obvious candidate, inserted on serial communication lines - perfect spot for signal interception/injection.

"But this clever hack is probably not limited to RF and is likely to also be embedded in transformers used for isolating Ethernet lines."

I was thinking along the same lines: a balun could harvest energy and use it for other purposes, but it should either store it for later use using maybe a tiny supercapacitor inside, or inject it into the data stream on the fly not unlike RFID dongles do. Or maybe just have enough power and memory to store a few hundred bytes and alter a few fields here and there, so that for example a network frame coming from a compromised machine can have its source field rewritten as it came from a trusted source. In theory small pull up resistors also could be swapped with lookalike malicious parts capable of tampering with i2c traffic between boards peripherals so that devices can be activated/deactivated no matter what the CPU tells them.

We're going straight to the point when there won't be a single device in the world, from toasters to supercomputers through top brand network gear, that can claim and guarantee to be secure; inserting malicious hardware and firmware is getting just too easy for those with the necessary knowledge and resources. A technically interesting and scary scenario.

The article specifically says that the attack goes after the Baseboard Management Controller (BMC). In Supermicro equipment this takes the form of a separate SOC made by ASPEED that runs it's own operating system entirely separate from the main CPU but with access into the main system through various mechanisms. Facebook uses the same ASPEED chips in their open hardware projects.

It seems to me that embedding an RF front-end, back-end, and signal manipulation logic in a chip of the size shown in the article would be extraordinarily difficult. Assuming that feat was achievable, it would likely result in serious performance hits on that transmission line right? Signal latency would probably be a dead giveaway for something like EEPROM reads being manipulated.

you would be extremely hard pressed to fit that amount of logic (and code) into that small of a package. and to operate intelligently, as you suggest, it needs a CPU clock, not available to it from where it would sit. And it would need to do signal analysis in real time, with no power and no clock.

it's not possible.

Clockless CPUs exist.

grain-sized? and they still need power.

Honestly, I'm too ignorant to judge the rest, I just think you weren't accounting for (1) you don't need a clock signal for fancy logic and (2) you can fit a lot of logic in a cubic mm. The other points may rule this out, I don't know. Note that the async CPUs I've read about tend to be lower power than clocked ones.

Clockless CPUs with RF transceiver and signal manipulation logic, however, do not exist.

It does look like a balun which is clever but based on the pinout I think it is meant to intercept SPI flash and patch it as needed.

Looks like I was correct. This image from Bloomberg article seems to indicate SPI flash (https://twitter.com/jamesdotcuff/status/1048221163607007233)

It's been a few years I've given up on the idea of privacy with technology. The number of security flaws that get discovered daily is only the tip of the iceberg. I'm pretty sure some governments (or organizations) have had backdoors, be they hardware or software, in place for more than 20 years. We simply don't know about it yet (and probably never will). Would that actually be that far-fetched? I think not sadly. Even the Intel Spectre et Meltdown fiascos are a sign that we have no idea how to actually secure this stuff. And that's normal, the very definition of IT security is that nothing can be secure. Take the whole antiquated concept of processor rings for instance, we are adding a new level every other year now it feels like... I find it way more interesting (even if it is ultimately "worse") to adapt to the mentality that "nothing is secure" than "let's try and make it secure", which as stated is in itself a fallacy...

I think the root of the problem is backwards compatibility and the fact that initially all these components were not designed with security in mind. So we're adding more insecure components in spirit of "move fast, break things" than fixing the debt incurred.

> It's been a few years I've given up on the idea of privacy with technology.

The problem with tech is that it's modularity and dependency on other people/tools/hardware/etc ultimately requires trust.

> in place for more than 20 years.

Far longer than 20 years. The idea of trust, privacy and security has been discussed for a long time. The complexity of security on just on one specific technology ( compilers ) was discussed in the early 1980s by Ken Thompson.

"Reflections on Trusting Trust"

https://news.ycombinator.com/item?id=13569275

Compilers are just one part of a complex ecosystem. Now imagine having to check ABI or the physical chips/hardware themselves. Where you require sophisticated hardware.

The motto of technology is "In Trust We Trust".

The more I understand software the less I trust it (given the current state of engineering practices). Meanwhile all my friends/family are scrambling to install all the latest new "smart home" gadgets and I just look like a paranoid kook trying to talk them out of it.

This. I've even had an in-law say, "My brother works for the Defense Intelligence Agency, and he uses smart devices in his home, so they must be safe!", with no consideration that tech may not be his specialty, or he doesn't follow the daily IoT fiascos, or maybe he just thinks he won't get hacked. Dunno. Meanwhile, my year-old thermostat still wants me to connect it to wi-fi, and that will never happen.

Wait, can you use a Nest without connecting it to the internet?

Dunno - my thermostat is not a Nest. Works great without the internet.

You're definitely not a paranoid.

However, IMO, the "cost" of ignorance such area is not that bad either. I will let them live their way to be honest.

Amazon are going all out on the denial

https://aws.amazon.com/blogs/security/setting-the-record-str...

Apple is very strongly denying it as well

https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

Never believe anything until it has been officially denied.

These are such detailed denials that it's hard to believe the companies are lying.

hard to believe the companies are knowingly lying.

If you're working with the feds on a national security incident, and the PR department comes round because some reporters were asking about it, the correct answer is "I don't know what you're talking about."

What is more telling for me is that Apple has cut off all contracts with the company involved and will have nothing more to do with them - despite their denial.

They cut off contact for a separate reason - 2000 (IIRC) supermicro boards which did have issues.

If they wanted to cut ties for a different reason, execs could have just said "find a problem with some supermicro boards, no matter how much it costs to do so, and never repeat this conversation". They are complicated human-designed and -made artifacts; of course some problem exists. This is known in other situations as "parallel construction".

Apple’s denial was quite strong as well, though brief. I wouldn’t be surprised to see the Bloomberg article retracted someday.

Yeah, I mean that's what I would say too if some government intelligence agency told me I cannot say a word about this and have to deny it vehemently! haha :)

There's definitely something afoot. Bloomberg probably wouldn't have gone forward with just an in-depth piece referencing so many major tech companies unless it had substance. But generally even when legally compelled, companies tend to prefer silence or curt denials over lengthy detailed contrary pieces.

Is there a federal investigation going on into some sort of sabotage by the Chinese government? Possibly, and if so, there's a lot of reason these companies could even be willingly participating in covering it up. All of them benefit from the investigation being uninterrupted, and nobody needs the bad press.

Could the story be, in fact, fabricated? Sure. Russian trolls have seen to sowing discord in far more than just our election. They stoked drama about The Last Jedi even, as recently claimed. Is it possible they're seeing if they can get a major news outlet to publish something with absolutely no basis in fact? Maybe.

There's a lot of interesting possible angles here, but with the number of large public companies involved, this morning is probably the start of this drama, not the end of it.

I think it is a win-win for Apple and Amazon to spin it this way. They get to deny that they were compromised, and if that gets disproved they can easily say they were cooperating with national security operations and get away without any damage.

...they do? Why?

Investors don't seem to care, stock has slid over 2% today and it's still going down.

I wouldn't read too much into it. Stocks in general are down today, as bond yields go up.

Frankly, I trust Bloomberg more than Amazon and Apple's PR departments combined.

You trust anonymous sources, over a company that is willing to back their claims?

However Apple or Amazon didn't back up their claims. If Bloomberg published an article of this magnitude which could be demonstrably disproved, it'd kill Newsweek's reputation.

More importantly, think of what would be Bloomberg's reasons to publish a false story versus Amazon and Apple's reason to deny a true story.

Of course. Apple/Amazon can be coerced into denying this (via government requests or otherwise). Bloomberg can't, nor is there any advantage for them to publish false information.

> nor is there any advantage for them to publish false information.

of course there is. It's called ad revenue and page views, both of which they're raking in today with these allegations. This story is all over the place. I don't trust Bloomberg with anything Apple at all and haven't for years now.

Here is one motivation for false information. Financial gain. Someone could have made a lot of money with put options.

Yes, this too. Some of the Bloomberg and WSJ "leaks" feel like their primary motive has been subtle stock manipulation. In particular, the "demand is slowing for $PRODUCT" articles that we continue to see.

Even if a national security letter wouldn't compel Apple or Amazon to lie, couldn't it compel the security team not to inform the executives or lawyers? Or at least this is an argument that's been going on for a while, can't remember if that issue has been settled.

Fascinating. A company I used to work for dismissed this sort of situation as a problem and ended up using SuperMicro boards in a security crucial product. Their hardware has always been notably very crappy, with the IPMI interface defaulting to world-open unsafe parameters, but I'd not expected it to be this cleverly hardware backdoored. It's possible to neuter Intel ME, but that's only a small comfort with these motherboards.

>Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered.

>...the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached

>...that generation of chips was smaller than a sharpened pencil tip, the person says.

I hope this corrects the mistaken believe that China can't home grow sophisticated tech.

This sounds strange but it's commonplace to make parts like this. Some DCDC converter modules have silicon dies embedded into their PCBs to save physical footprint, most notably there's a family of Texas Instruments ones which use this technique.

https://www.electronicdesign.com/sites/electronicdesign.com/...

> I hope this corrects the mistaken believe that China can't home grow sophisticated tech.

There's nothing particularly sophisticated about what they did, especially given what China has access to as one of the central hubs of tech manufacturing. There are two dozen nations (or more) that could do this from a strictly technical standpoint (few have the kind of required supply chain access to pull it off at scale in actuality). It's the audacity that is primarily impressive. China is encouraging all of their richest trading partners to further isolate them when it comes to supply chain and advanced tech. They're confirming for the 107th time what everyone already believes.

You're right.

One question though. Would a politically correct, by-the-book US president have had the balls to sanction China? Considering such sanctions could affect the US economy.

> Would a politically correct, by-the-book US president have had the balls to sanction China?

Does Trump even have a goal in mind with his tariffs? It seems like he just wants to score political points, rather than achieve any actual outcomes.

Remove reliance from the Chinese supply chain?

Given that there’s video from 4 decades in which Trump advocates for tariffs, during which time he wasn’t a politician, it would seem that they are something he believes strongly in.

The goal, explicitly stated and often acted on in many contexts, is to disrupt international order and cooperation (including trade), and promote nationalistic competition by all countries. Recently his UN speech, for example, advocated it.

Please take this political posturing elsewhere.

IC fabrication and packaged component assembly are very different things. China doesn't have anything near this level of sophistication, at least not without any outside help.

This seems either incredibly suspicious to me or indicates that Apple is way more protective of security and distrustful of the USGov than I would have thought: “Because Apple didn’t, according to a U.S. official, provide government investigators with access to its facilities or the tampered hardware, the extent of the attack there remained outside their view.”

Why wouldn’t the FBI be able to compel Apple to turn over such key evidence in such an important investigation? Or why wouldn't Apple be willing to do it even without compulsion?

Apple seems cagey about its reputation with the Chinese government. Their balls are directly in the Chinese government's hands, they don't want to do anything that might make the government upset.

Beyond having their manufacturing there, it's also the biggest untapped market in the world.

I hate leaving a brief unsubstantiated comment, but this is the new Cold War and I think it needs to be acknowledged as such and brought out of the dark.

I don’t want my intelligence agencies fighting a war we aren’t even publically acknowledging and subjecting to public congressional oversight. I don’t trust Congress, but I trust the NSA/CIA far, far less.

The economic sanctions against China are disproportionality minuscule if they are meant as a response to this level of attack.

If this is true, it should be proven in a court of law and trade sanctions should be a positive integer trillions of dollars.

Anyone remember the hack of Apple via Supermicro firmware.... https://arstechnica.com/information-technology/2017/02/apple...

This is what Bloomberg's article refers to as "unrelated reasons" for Apple cutting ties with Supermicro in 2016.

> Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.

And then as an "unrelated and relatively minor security incident" later on.

> One country in particular has an advantage executing this kind of attack: China, which by some estimates makes 75 percent of the world’s mobile phones and 90 percent of its PCs.

Intel and AMD are both USA based companies.

Is it conceivable their processors contain backdoors in a similar vein?

I'm pretty certain they do. AMD's PSP and Intel's ME are really shady and are still enabled in every CPU sold today (and full of disclosed and undisclosed flaws for that matter).

Not to mention that these back doors can be easily disabled, e.g intels high assurance mode, but normal "owners" are specifically not allowed to disable them.

The whole thing is a fucking mess.

No, it is totally inconceivable AMD and Intel CPUs are backdoored this way. Inserting a microcontroller somewhere on the Ethernet traces and then using IPMI is not as sophisticated as this article wants to describe. Sophistication is necessary from the payload to be stealthy but not the hardware.

There are at least two problems with China messing with CPUs: a) they are not made there. TSMC is Taiwan, the Asian parts of Global Foundries are in Singapore, Intel is manufacturing CPUs mostly in the USA with some in Ireland and Israel, only 3DNAND / 3DXPoint ICs are made in China. b) the hardware knowledge to change here, well, think of it, the microcontroller on the motherboard is on the few millimetres scale, the parts here are on the few ten nanometres scale so as a rough but not unjust quantifications would say it's 100 000 times harder.

Whether the platforms are hackable / hacked, that's harder to say thanks Intel ME and such but the CPUs aren't.

I don't think the parent was talking about this specific type of attacks but other types of hardware backdoors. I think the answer to that is very obviously yes, and given what we know about intelligence agencies I'm even willing to go as far as saying that it is likely (or at least, if you have reasons to be worried about Uncle Sam getting to your stuff you should consider it a very real possibility).

Modern ASICs are so complex that I'm sure that sneaking a tiny backdoor into the behemoth that's a modern CPU or embedded SoC would be almost trivial. They'd also have great plausible deniability in case they're found, if something like SPECTRE was an intentional backdoor how would you ever prove it for instance?

> Modern ASICs are so complex that I'm sure that sneaking a tiny backdoor into the behemoth that's a modern CPU or embedded SoC would be almost trivial.

I suspect putting in a backdoor would be difficult because they are complex. Wouldn’t it be far too easy for the backdoor to inadvertently cause reliability or performance issues? And the bug would have to be useful enough to warrant potentially destroying the semiconductor business of a nation, not just some difficult to trigger side channel.

> No, it is totally inconceivable AMD and Intel CPUs are backdoored this way.

That’s a fairly strong statement.

It could be argued it is conceivable as evidenced by the fact we are here talking about it.

Also, I’ve read that it’s impossible for any one person to fully comprehend the circuit diagram of a modern CPU. In light of that it seems conceviable any number of wayward circuits could be hidden.

can you be sure that fabs haven't been infiltrated and masks changed between design and production in any factory, be it tsmc, samsung, glofo or intel?

Presumably the organisations responsible for hard coding backdoors in chip designs know how to test to confirm their presence.

Presumably some adversarial nation-states have moles inside these organisations > know how to remove them prior to fab.

Presumably these adversaries export genuine chips to their adversaries, thereby tricking them in to thinking the backdoors made it through the fab process, and only use chips that have the backdoors removed in their own critical infrastructure.

Presumably.

I’ve always had this fantasy of being a hextuple agent involved in this type of deep espionage.

The US did that in the late 70's or early 80's to the USSR and it resulted in one of the largest non nuclear explosions in history.

* allegedly

https://en.m.wikipedia.org/wiki/At_the_Abyss

And how about the software that runs on these things?

they're up front and open that they do

also, they're not that hard to dig out

https://www.youtube.com/watch?v=_eSAF_qT_FY

It makes me wonder how compromised the prosumer/enthusiast motherboards are. All of my important home systems are homebuilds using off-the-shelf motherboards purchased in person, and lack any IPMI access. Sure there's AMD's PSP (grumble, grumble) in my Ryzen box, but I can't help wonder what might be lurking on the motherboards, or for that matter in my network switch, router, access point, etc.

I've sometimes thought about picking up some type of used rackmount server, but hacks like this give me cold feet - aside from the usual issue of 1U/2U boxes sounding like jets taking off.

The denials by Amazon, Apple, Supermicro and the Chinese Ministry of Foreign Affairs [1] are relatively pro forma, both directed by respective nation states involved in this matter. One of the reporters interviewed on Bloomberg noted Amazon and Apple could be directed by US national security interests to deny to protect the ongoing US investigation. Supermicro could similarly be directed by Chinese national security interests to protect plausible deniability.

There was a sense of realpolitik by one UK guest commentator on Bloomberg, comments along the lines of "hey, spying happens since time immemorial, put on some big boy pants, yes there is shock but not horror when the Snowden revelations came out, the US does it, too, etc.". I disagree with this sentiment, as while the attack was quite targeted, it puts into question a quite large supply chain network.

Kudos to Bloomberg putting in the 12+ month investigative journalism to pull off this scoop. Yet another validation of the reasons I subscribe to listening/watching them.

[1] https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

> Supermicro could similarly be directed by Chinese national security interests to protect plausible deniability.

Supermicro is an American corporation, headquartered in San Jose. They're not directed by Chinese national security interests, they'll do anything the US Government tells them to do when it comes to US national security.

I stand corrected, thanks.

Supermicro's vendors they use in their Chinese manufacutring are the likely vectors for the implantation.

As someone who has used a few SuperMicro boards in custom builds, I would like to know which motherboard models have been affected. It's not clear to me from the article whether that is even known at this time.

I'd also like to know of a recommended work-around. I have always had IPMI "disabled" as much as I can; assigned a non-routable IP address in the BIOS in case it flips to a failsafe mode by piggybacking on the main NICs. But is that sufficient? Is BMC ~= IPMI?

Has anyone found more technical details yet for system administrators who don't work at the thirty companies contacted?

Technically BMC is the hardware and IPMI is the software, but yes they're the same thing.

It's good to change the network port settings to "dedicated" so it doesn't failover... but if it was malicious, and it clearly has the ability to piggyback on your production interface, why wouldn't it? There's honestly no way to secure this without cutting off your hardware from the internet completely (or firewalling it off similarly).

This is just the hack that was discovered because there was macroscopic evidence of it. All it would take to pull off a similar hack that was undetectable is one well placed mole in the company that designed a key piece of silicon or software.

no way, an intentional design-level plant would have to pass through many eyeballs. a single mole wouldn't be enough.

I have worked in both chip design and security so I feel well qualified to make this assessment: a single mole in the right place who knew what they were doing would definitely be enough. Security holes get past design review all the time by accident. A skilled hacker could easily insert an intentionally obfuscated one that would escape detection.

Yup. And that single well placed mole could have 100 or 1,000 security experts working on their behalf. It is not as though the mole would have to design the hack, only pass necessary documents to a military grade hacking group then implement their modifications.

Why aren't these attacks constrained by normal corporate firewalls? How does a random server on a navy ship start contacting baddie.china.com without raising red flags?

Who's saying the firewall isn't compromised too? ;)

Right so all the stars need to align for it to go unnoticed - compromised server, firewall and other alerting/monitoring tools.

I would have thought one single unexpected packet in these high security environments would raise significant alarm bells and any anomaly would be found very quickly.

I've worked on systems deployed in the financial sector in high risk environments.

This sort of monitoring doesn't happen in the real world.

Intrusion Detection Systems are basic network security 101 type stuff. I'd be surprised if anything that was really "high risk" didn't use an IDS.

There's a big difference between "using an IDS" and "a few unintended/unexpected packets are noticed and properly followed up on".

Intrusion detection involves connections coming from the outside. These attacks originate inside the network, from the compromised equipment.

Except on extremely controlled networks, this would be very hard to detect. It gets even worse when you consider that the Chinese had/have a distributed network of compromised machines. Imagine using a Google edge server as a dead drop...

An IDS is perfectly capable of flagging outgoing or internal traffic.

We had something similar where anything outside "normal" generated a ticket. It was disabled after 1 week because the support teams where getting more then 5000+ tickets each day. And this was after filtering etc....

Now this does not make it impossible, just very complex. In a more "controlled" environment such as a naval ship, i could see this actually working better, especially if the system is supposed to talk to very few external systems.

It does. I've deployed systems that would not only notify staff when novel packets were observed but immediately isolate anomalous hardware through a combination of powerdown and network fabric reconfiguration.

The amount of false positives a system like that would generate would rapidly render such a system entirely unusable.

Not in a secure environment... where you are supposed to control the hardware, the software, and the network absolutely.

The only way to create such an environment is to totally disconnect it from the outside world -- I'm talking even power source, phones, internet, all of it has to be disconnected or else I can exfil data all day long and nobody would ever know.

Yes, but eventually you have to get something done.

Security engineering is about tolerable failure modes. - Dan Geer (2014)

He was the "first employee and architect @ Kraken (2011-2015)". That might explain why Kraken spews out 502s all the time.

Dear snarky anonymous coward, as stated I left in 2015. I think you will find referenced issues occurred subsequent to that date under very different technical leadership.

Only if you’re lucky. Priority is checking boxes and meeting audit requirements.

If you’re running an IDS on a big 100 Gb datacenter network, you’re literally processing millions of events. Very few places would notice such a thing unless they were investigating something related, and the ones with the capability are going to be for static workloads as getting anything done will be slow and painful.

I suppose the hardware on the navy ship is communicating with a server that is also compromised, and so on and so forth.

Effectively nothing can be constrained by a whitelisting firewall if you have a sufficiently bored actor. You can smuggle data through a variety of benign looking protocols, things that wouldn't matter in the least generally. Your average server contacts hundreds of different public NTP servers, binary repositories, domain name servers every day. If the keys to the kingdom are a 32 byte ECDSA private key, you've lost if you think you can protect this from reaching the outside world.

A method that wouldn't show up on any firewall in the world is simply to delay or drop certain SYN packets. Even if you only intended to transmit a bit at a time through this, any unauthenticated host on the internet could use this without raising any suspicion or even printing log lines in most environments. As soon are you're making an assumption that you're trying to prevent what's inside from getting out things become substantially closer to impossible than anybody would like.

Firewalls in high security environments aren't just port/protocol based. You lock everything down - source ip/port and destination ip/port. You should know where it is coming from and where it is going to.

Navy ships don't upload via Dropbox.

In the parent I described a system which would be able to communicate through those restrictions to another compromised host (remember we're assuming everything is compromised for the sake of this article, which actually seems like a good assumption now).

Networks where security is taken seriously implement data diodes, and this attack vector is mitigated.

> remember we're assuming everything is compromised

I think Bloomberg (and all related) web servers displaying the article are compromised and they're leaving out critical facts the point the finger elsewhere.

Certainly true, since Dropbox doesn't work on Windows XP anymore.

Presumably it contacts somewhere ubiquitous like AWS or Cloudflare.

We need open source hardware designs that can be built locally (where ever your local might be). This black box hardware crap has to stop. Smart people who know how all this works need to dump all their knowledge in to a design and a process. Trade secrets are keeping us not only limited in choices but exposed to bad actors who can control a link in the supply chain.

This sounds beyond complex to manufacture. Especially at Fortune 500 levels.

DIY? Well its going to be larger, hotter, more failures, and probably more expensive.

There is a reason companies specialize.

Sounds like a great idea, but who will fund it?

Won't that just get us NSA backdoors instead of PLA backdoors?

>Smart people who know how all this works need to dump all their knowledge in to a design and a process.

Great, are you paying?

Ok, I’ll design the logo!

My gut feel about this for many years has been outsourcing chip manufacturing to another country is a serious security risk. This was a hardware device, hard to detect as it was, how much code is in the chips we already expect on a motherboard?

It seems flat out foolish for one country to own the world’s computer manufacturing.

This is the logical conclusion of companies giving in to shareholders frothing at the mouth for increased profitability by any means necessary. Manufacturing crucial components like motherboards in China is just absurd.

When will this stuff finally have consequences for China?

Their behavior, not their communication, has been overtly hostile for a while. Yet, very few politicians openly adress the issue.

>When will this stuff finally have consequences for China?

Never, unless hardware manufacturing will take off somewhere else.

Somewhere poor, where labor is cheap and people are still susceptible to bribes?

Rather not, China's economic rise started in Jiang's era was both due to it being cheap, AND due to Jiang's era officials being more competent and business friendly than those of an average bantustan (really sorry having to use the term)

Do you really think that another world power would behave differently if they were the factory of the world?

Ha. This looks more like good old capitalism rather than any state sponsored spying.

"Nobody's making you buy from China"

It’s a cat and mouse game, where escalation can hurt. For example, China might take issue with the systematic compromise of undersea cables.

There’s a certain Casablanca-esque “what? There’s gambling happening here?” element here too — Huawei was shipping fake Cisco gear 20 years ago.

> When will this stuff finally have consequences for China?

Hahaha funny thing to say after disclosers from NSA/CIA/FBI/GCQH/BND...

It already has. Look at the tariffs against them.

I am sincerly impressed by the amount of supply chain analysis and operative supply chain management that went into that hack. And once the Chinese identified a distribution node in that particular supply chain, supermicro, they opted for a brute force attack by seeding these backdoor chips into supermicros servers and wating where they ended up. That was one hell of a hack.

It also gives you pause. Did that happen only once? For how long? Where esle did these chips and servers end up in the end?

Im impressed as well. But I'm sure its not as impressive as what the NSA is capable of. If you're reading an article about this - its because the US government wants you to know about it. The US has been doing this kind of stuff well before the Chinese or anyone else... And yes, totally agree that this is probably fairly common place. Some comments have shown that this happens with run-of-the-mill hardware like credit card scanners.

This stuff also reminds me of the Snowden revelations of hard-disk firmware hacking. Very similar conceptually to what is described in this article - albeit without custom hardware.

Every major major nation can do this stuff one way or the other. But somehow I am used to software hacks by now. That someone is basically managing aphysical supply chain (with suppliers, production and all of that) within an existing one is a first for me. The only thing similar are stories of the Italian mafia getting stuff by customs by duplicating shipping containers, still what the Chinese did is different level. But the, being a supply chain guy, I have an easier time understanding this as compared to what NSA and GCHQ are doing.

if they really are widely seeded (and it’s general knowledge/easily assumed that the pla leans on other manufacturers), this gives me a few ideas about what could happen in the event of conflict between china and the US. turnkey shutdown of the entire economy.

It reminds me of that scene on Armageddon. "Russian parts, American parts, they all come from Taiwan." Now it's China. It is easier to spy when basically everything runs on hardware made in your country I guess.

I’d expect that this type of parasite hardware attack isn’t unique. Securing the supply chain is even more important than ever.

Apple’s response isn’t great IMHO.

Just as a sidenote: X-raying PCBs and then diffing them against clean PCBs is a worthwhile thing to do if you're concerned about hardware backdoors, or 'interdiction' of hardware in a supply chain. I do this sometimes when ordering super-critical equipment like Thinkpads from the U.S as you never know what lurks on the motherboard (keyloggers, etc).

I have a clean Thinkpad that I use to compare against potential backdoored devices. So far I haven't spotted any differences in the PCBs. I guess the intelligence agencies have not marked me as important enough to target. That being said, I imagine there are people working in the cryptocurrency space who have a lot to hide (if you own their boxes, you could be looking at thefts worth millions of dollars, or whatever the equivalent is in the cryptocurrency they are developing).

How do you know that your clean Thinkpad is clean?

Sometimes the difference is inside of a chip.

> X-raying PCBs and then diffing them against clean PCBs is a worthwhile thing to do if you're concerned about hardware backdoors, or 'interdiction' of hardware in a supply chain. I do this sometimes when ordering super-critical equipment like Thinkpads from the U.S as you never know what lurks on the motherboard (keyloggers, etc).

I'm curious - how does one go about buying or getting access to an x-ray machine (and how much does that cost)?

I would go to a local SMD assembly house and ask to use theirs.

Once you do a thorough sweep and decide one device is clean all it takes is a small PCB revision and you have to start over. It becomes a very expensive exercise after a point (time and money) that is reserved for the most critical of situations.

The problem is a good hack doesn't need the most critical of situations to succeed. It can creep in through the most innocent cracks and climb its way up the chain.

And some hardware hacks are beyond what a regular Xray PCB check would outline, like a modified piece of silicon in a chip. Xrays are great for more obvious changes, not necessarily for something that was inserted at the very start of the supply chain.

"In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips."

Wow.

> That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest ... “You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.”

Once the critical mass of electronics manufacturing moved outside the US and manufacturers outsourced the bulk of work, there was no longer any alternative.

Economists say this is ok or even really good for global welfare.

But do any economic models accurately account for this type of economic damage, this type of cost to government, consumers, companies like Supermicro, Apple, Amazon, etc?

And this is one of the major reasons why I support that the US is going after China.

It's a disgrace that previous politicians from both the EU and the US have just let China steal from the west through technology transfer and not actually enforcing IP.

Side note: this is why private blockchains for supply chains don't have any sense because there is no way to probe that you are delivering what you say.

This sounds like something out of a movie. I would like some technical details how this is supposed to work with only a 6 lead chip.

Serial bus magic.

It seems surreal to me that this could be real. And yet I'm not surprised.

Most likely it is using SPI https://en.wikipedia.org/wiki/Serial_Peripheral_Interface, that requires four pins and the two remaining ones are power and ground. SPI what is used to access EEPROMs and flash memory, so an attack that you can do is daisy chain such a device in the path to the EEPROM the board management controller uses as its firmware storage. Then you can very easily insert your own instructions and get the board management controller to execute whatever you want.

So it’s sounding like the Chinese government may have invested a large portion of its war chest on subsidizing computer manufacturing in order to gain this type of strategic position. It feels like chess, and seems brilliantly scary.

I wonder if SuperMicro being delisted from Stock market has anything to do with this? Where their "Accounting Errata" were merely cover up.

I'm not sure exactly what "delisting" entails, but if my billion dollar company was about to be in the news for having its supply chain compromised by a foreign state actor, I would be happy to not see the stock crash triggered by a public order book.

Stories like these make me think there is a real need for electronic devices made entirely in the US (or your preferred trusted country).

Products like the Yubikey are apparently made in the US / Sweden, but I wonder how many components are actually made elsewhere and just assembled in these countries. Are there any good examples of consumer devices actually made entirely in the US down to the microchip?

Like data breaches, this will be big news that some devices were compromised at some time. Then followed by a slow dribble here and there, that, oh yeah, well more than we initially thought. Then a few months later it will be, "Yeah, it's really a lot more than we thought." Maybe a year from now it will basically turn out to be everything. But numbers get so big so fast people don't really know how to effectively parse the difference between 20 million devices and 20 billion devices. So it won't matter to most people.

And really, it shouldn't matter to most people. There's nothing the vast majority of people can do about this. It's not like we can just go buy stuff made in the U.S.

Which companies performed the security audits?

Are those chips still sitting in functioning DoD servers?

How have they mitigated this issue, and where will they source new hardware from?

China is not a friend to the US alone, many countries hate them for their unethical practices

The more I speak to scientist and engineers from sub-Saharan Africa, the more hate I hear for China. It’s hard to comprehend.

Building in China may be cheaper, but that’s a stupid consideration. After all the Greeks charged the Trojans absolutely nothing for a fabulous wooden horse.

what they describe isn't possible. which is perhaps why Apple et al. are easily able to refute it.

what actually occurred must be something a bit different, that they didn't understand and/or aren't conveying accurately or didn't receive accurate info due to it being classified. or you know, because the general press is still at movie technology ... the hacker is in the building!

A chip the size of a pencil point can't be inserted anywhere useful or do anything meaningful. Something that small amounts to a discrete component, not a "chip", if packaged. If bare die, it still can't be vastly complex and needs to be mounted in a way that is very obvious (epoxy blob).

So, it must be an additional discrete that inhibits the burning of a W/O fuse in the BMC or some other management function. Thus preventing the disabling of some debug function.

There is no way it has communication capability, etc, on its own.

EDIT: ah, @baybal2 perhaps figured it out. it's likely flash memory and the bmc already has provision to read it as the "recovery" flash.

Seems like a bit of Karma to me. Outsource to try to get cheap labor rather than pay workers a living wage, and this is what you get.

The Bloomberg article might well be the better explanation for the recent problems of Supermicro: https://www.theregister.co.uk/2018/03/14/supermicro_praying_...

  The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.”

Essentially China ils saying "it was not me". Plausible

I Read that as "The US is also attacking our hardware supply chains". That is, the statement concerned supply chain attacks in general, not this specific one.

Yeah it seemed to imply a ‘tit for tat’ attitude to me

China is sourcing virtually nothing from US. If US is attacking the supply chain of China, it's doing it on Chinese soil. Which brings us back to my interpretation

This has been posted several times and there are tons of comments:

[1]: https://news.ycombinator.com/item?id=18146438 [2]: https://news.ycombinator.com/item?id=18138328 [3]: https://news.ycombinator.com/item?id=18145645 [4]: https://news.ycombinator.com/item?id=18138990 [5]: https://news.ycombinator.com/item?id=18141328

If you scroll all the way to the bottom of the article you will find the disclaimer

Bloomberg LP has been a Supermicro customer. According to a Bloomberg LP spokesperson, the company has found no evidence to suggest that it has been affected by the hardware issues raised in the article.

Why did Bloomberg not buy a bunch of SuperMicros (new and used) and find the chip? That would be difficult but would turn this into a HUGE story. Even if they didn't have the technology to do so in-house, there are many companies they could hire to do a forensic investigation. The weakness of all bugging is that it has to communicate to the "outside world" at some point to be useful and that communications is discoverable. Even Stuxnet, which was much more narrowly targeted than this, was eventually discovered.

It’s describing events from 2015. Presumably this issue has been resolved.

There are tons of old SuperMicro computers on eBay for very reasonable prices. They could have bought lots of them of the appropriate vintage (2015 and prior).

> There are tons of old SuperMicro computers on eBay for very reasonable prices. They could have bought lots of them of the appropriate vintage (2015 and prior).

I think buying a bunch of old servers and looking in them for implants is so far outside Bloomberg News' core competencies that it's unsurprising they didn't do it.

Based on my reading of the article, the allegation is that only boards going to certain designated customers were targeted. Not all boards.

I will certainly show this to everybody who tells me "google won't get hacked" when privacy comes up :)

Talking about things like this https://news.ycombinator.com/item?id=18074097

I take a different view of this. Some 30 companies got this hardware, two are named as having the resources to find the offending hardware. By this logic you’re still safer using a company such as Amazon.

My point is that you should care about privacy, even if you're using, for example, apple services. Thinking that nobody has the resources to hack a company that big isn't an argument and has just been disproven.

In other words, if you have plans to ever be a politician that china may not want in any position of power, don't store your nudes on icloud/dropbox/google drive, or they may suddenly get leaked when you least expect it and ruin your career in favor of a more... Shall we say "convenient"? ...alternative.

Disproven on speculation?

I’m not saying nobody has the resources to hack a company. I’m saying few companies have the resources to detect such a sophisticated hack. Therefore my infra is probably safer with, say, Amazon vs my local colo.

You are correct that individuals seeking personal infosec against state actors must be eternally vigilant.

>Supermicro subcontractor

There is only one I know of: Soyo motherboard

> In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe.

So this is still possible ? And this is slipped so lightly in the article.

"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

Amused to see some element of humor added to an article on a subject that will have extremely serious consequences, the likes and extent of which are hard to estimate at the moment

Where is the _actual_ explanation of how the chip works? I assume it's been dissected to bits, and that's way more interesting than just "it's been found on lots of boards". What does it actually contain that apparently lets it do things that apparently no one else has managed to achieve at that scale?

Interesting that Apple didn't give the FBI access to its datacenters for the investigation. Didn't trust them?

Do we know how this attack actually worked? From TFA it seems to involve the BMC (which afaik everyone already assumed was untrustable), and also involved the capability to "phone home" (also notable since in security-conscious deployments the BMC NIC would never be allowed public Internet connectivity).

This reminds me of Anne McCaffery's PartnerShip: https://www.goodreads.com/book/show/410927.PartnerShip

One of the plot lines involves a similar attack vector.

In retrospect it seems that the decision to move all manufacturing to China was ill-advised.

Western companies ultimately will have no choice than to move it all back. (And Trump will want to take credit for that.)

I can understand all the big guys denying this. It's very hard to fix and very bad for business.

So, what was the SMT part supposed to be, and what was put in there instead? I would love identifying markings, or you know, a datasheet.

Nothing so far I've read includes part numbers. Sure would like to go through my supply to see if I have any of the offending parts.

something like this http://www.littelfuse.com/products/tvs-diode-arrays/ultra-lo... not only looks the part, but actually belongs on the data lines

"The resulting software dramatically reduced the time it took to process large video files. Elemental then loaded the software onto custom-built servers emblazoned with its leprechaun-green logos."

I didn't realize Pied Piper was based on a real company...

I'm very curious at what is in that chip. I wonder how much we can reverse engineer it.

If I wanted to buy devices (laptops / cellphones) that, which are manufactured and assembled in America, where would I even look?

(Yes I know that this does not necessarily make the devices any more secure, better the enemy you know, etc etc)

It's very odd that this was released the same day as claims against Russia.

https://www.bbc.co.uk/news/world-europe-45746837

Is this as big as it sounds?

Probably much bigger. It's naive to assume Supermicro was the only vendor compromised in this way. Similar implants probably exist in equipment from every major vendor of information-processing equipment. Routers, mobile network equipment, you name it.

This is the leverage you give another nation-state when you let them control your supply chain.

Why didn’t they ask any of their sources to point out one of these chips on a specific (common) Supermicro board? That’d allow independent verification by everyone and way less vague speculation.

Well it's nice to the see the specifics on how they did it, but anyone was really surprised by this? Most electronics in the World are made there what people are expecting gonna happen?

Where are Vcc and ground for the chip? These leads would show up on a simple micrograph of the board. How does this chip make connections that enable spying on data in the server?

Something is fishy.

As long as you are powerful, you can get away with almost anything. This goes from individuals to countries. Had China been weaker, it would have been invaded already .

Maybe it's worth talking about how we can better secure the chain of custody in shipping? I asked about this a year or two ago and my question got the thumbs-down.

I wonder, given the fact the C&C were identified, wouldn't be external firewall configuration much cheaper mitigation method than motherboard replacement?

Maybe it's worth talking about how we can better secure the chain of custody in shipping? I asked about this a year or two ago, here, and it got squelched.

Isn't this or alike chip put all the Chinese hardware other people and enterprises buy too? Or at least in all the SuperMicro mainboards (I use these too!)?

As a nation, we need to unite against the external threat of China. Russia and Israel are fine. China is the number one problem sowing discord in this country.

Ok why not all of them?

You know the answer to that. Russian bots target Hacker news as well as other sites.

Any evidence of that?

https://www.justice.gov/opa/pr/us-charges-russian-gru-office...

Have fun googling what the rest of the GRU does

Maybe it's worth talking about how we can better secure the chain of custody in shipping? I asked about this a year or two ago and it got squelched.

The image at the top says it's a Supermicro B1DRi. I wonder if this is an actual infected board or if it's just for illustration purposes.

We can agree this news is suitable for Hacker News.

The graphics - the opening animation and the penny are some of the most informative illustrations i've seen in an article.

> The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

Obviously this would still be possible without IntelME, but having an always on, highly privileged and remotely accessible baseband definitely makes the modifications easier and smaller...

What's the old saying about sleeping with dogs?

Yeah. That.

So. What now. Any consequences? I mean, will this trigger big changes, like some production moving somewhere else?

>like some production moving somewhere else?

Sure, to US, UK, New Zealand, Australia, Canada, because we trust these countries not to backdoor software and hardware.

Elemental servers sold for as much as $100,000 each, at profit margins of as high as 70 percent, according to a former adviser to the company. "Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

Church would certainly be more interesting if they did, though.

So what is the alternative to SuperMicro if you aren't large enough design your own board?

Other motherboard vendors (e.g. Tyan, Asrock, Gigabyte make server boards), or buying entire servers from Dell, HP, ... And hoping they don't have issues like this, despite probably all manufacturing partly in China.

None of those guys offer the same variety of form factors and features that Super Micro does. If you are an OEM that needs an Intel Xeon-D board in mini-itx form factor for example.

Define partly ? Almost all components are coming from China

I feel like there is a really large opportunity here for some other international supplier...

Holy molly! This sounds similar to what I thought a couple of days ago: China is the major electronics supplier to most technological companies, what if China added some sort of sniffing or malware on the chips to spy on their customers?

Is anything around the Intel Management Engine made in China?

As something I heard: In China, business a fight for survival that is brutal to the extreme. You will be cut down, cheated, extorted, and broken with zero hesitation.

Doing business in the West is comparatively like a walk in the park.

Couldn't the same be possible on routes?

the presentation is very good, at least. white text on black background, very dramatic.

... huh. For the first time, I'm considering the possibility that the Trump administration's tariffs on China might actually result in a positive outcome: forcing US companies to look elsewhere for hardware, possibly creating new supply chains with other (hopefully more trustworthy) partners out of sheer necessity.

Loose chips sink ships

1) I wonder how wide spread this is in electronics produced in China. Say, in those cheap usb blutooth radios, for example. 2) I wonder if this is spurring the current trade war push, besides just Trump. 3) Seems time for more transparency on supply chain integrity and security.

Deleted comment

Anyone have a mirror of the article? I can't read it behind Bloomberg's computational paywall.

This is crazy

Statements from Amazon, Apple, Supermicro and Chinese government.

https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...

From Apple:

"Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple."

Assuming Bloomberg's story is true, I wonder what reason Apple has to hide. Not wanting to upset relations with the PRC govt?

NSL letter, under active investigation

NSLs require secrecy not lying.

Couldn't an NSL have been served to datacenter operators, along with the notification of the attack, and the organisation's management simply be unaware?

They might volunteer to lie, though.

The snowden leaks among others show that most companies aware of PRISM ended up flat out lying about it. Either it's a type of NSL we haven't seen before or employees receive death threats, etc.

No, none of the companies lied about it. The companies worked with the FBI's Data Intercept Technology Unit. They would obviously have no knowledge of a dowstream data processing system like PRISM.

they have literally every reason to deny and literally no reason to say it's true

Except, you know, to avoid committing securities fraud by making a material misrepresentation.

There is no way that the intelligence community would allow that fraud case to go ahead.

That assumes that 1) the intelligence community has the power to stop it and 2) that Apple believes this to be the case and 3) that Apple is confident that the intel community would use that power to protect them. That seems like a reach to me.

#3 isn't the intel community protecting Apple, it would be protecting themselves, which is a lot more plausible. They don't want detailed information about the techniques coming out. Odds are good that the Bloomberg story is still incomplete in some critical way, and decent that even if the story as a whole is broadly-speaking "true" there's still an outright lie contained in it. My guess would be the way in which it was discovered.

I work for a company that sells network appliances, and I've been questioned by customers as to why I'm doing an SRV DNS lookup instead of a standard A DNS record lookup in some software I wrote, and had every detail of how I use TLS picked over by some customers. (More power to them. Not a complaint.) Some people run really tight networks. I wouldn't be surprised the real discovery mechanism was someone noticing the packets heading out that had implausible source-dest pairs ("why is my internal network that barely knows the internet exists trying to send packets to $RANDOM_LOCATION?"). If the people discovering this were actually the intel agencies themselves, for instance, they'd find another story to tell rather than reveal that. I am absolutely, positively not claiming this is true; I have no more evidence of it than anyone else. I'm just giving an example of the sort of thing I mean. It's also possible the intel agencies slipped a hint to someone about what to look for; again, I have no info to that effect, just an example of why they might not want something to go to court.

> Odds are good that the Bloomberg story is still incomplete in some critical way

You mean like this?

https://news.ycombinator.com/item?id=18139371

They will absolutely step in when Apple decides to fight the case in court risking the exposure of sensitive info.

I'm reminded of Matrix-Churchill: https://en.wikipedia.org/wiki/Arms-to-Iraq

Although that nearly went in the other direction. The people involved were nearly sent to jail for shipping arms to Iraq which they had been doing at the behest and with the complicity of the UK security services.

The power of government agencies to turn up to people's offices and tell them "you need to stop doing what you're doing, and it's illegal to mention this meeting" should not be underestimated.

I think it's extremely foolish to think that the SEC would have the ability to overrule the CIA/NSA/other TLA when it comes to disclosure of this.

Not at all. It would be quite damaging to their reputation if it came out later that they were affected by this, knew it, and lied about it. Especially since the privacy of customer data is a key part of their marketing message these days.

Damaging how? People will stop buying iPhones? Be realistic.

Assuming the story is true, you lie lie lie until something bigger in the news happens then you quietly relent.

Personally I buy Apple because they are more private than android. I would look at alternatives if Apple was caught lying here.

Well, what was Apple's response to the PRISM revelations?

Apple: "We have never heard of PRISM. We do not provide any government agency direct access to our servers, and any government agency requesting customer data must get a court order."

PRISM: "Collection directly from the servers of these US service providers: Microsoft, Yahoo, Google, Facebook, PalTalk, AOL, Skype, YouTube, Apple" A later slide claims that Apple joined the program in October 2012.

http://www.washingtonpost.com/wp-srv/special/politics/prism-... for those documents.

So, was Apple lying about PRISM? The Snowden documents certainly seem to support that position[1]. If they lied about PRISM, why would you trust them now?

[1]: Obviously there are possibilities like Snowden documents were wrong, or deliberate hoaxes, etc. Spy games are fun!

What percentage of stories like this do you think never come out publicly? 50%? 90%?

This article is more or less total bullshit. At _best_ that device might be a mechanism to cause failure intentionally. And there are tons of ways to detect it with commodity technology, and plenty of vendors who implement that technology for assembly manufactures commercially.

That’s what I thought but then it says it’s hooked to the BMC bus. It’s basically a small IME device with no java bloatware to run. I’d think it’s reasonably credible

My issue isn't whether or not it's possible for hardware to be insecure or whether or not it's possible for exploits to exist.

My issue is this Chinese undetectable super chip creating unpreventable wide-scale vulnerabilities.

For what it's worth, I've worked in hardware security and I own a hardware quality control startup.

> I wonder what reason Apple has to hide.

The perception is that Apple is perfect and worth paying 3x the cost?

EDIT: Curious if all of these Apple comments are going to disappear. I believe they have a strong marketing team to hide dissent.

Or people recognize that statements like "The perception is that Apple is perfect and worth paying 3x the cost?" are hyperbolic nonsense.

This breaks the site guideline that asks you not to insinuate astroturfing or shillage without evidence. Please don't do that—it's a toxic trope that leads to dumber threads.

https://news.ycombinator.com/newsguidelines.html

Edit: looks like we've already warned you about this more than once. If you keep doing it we're going to have to ban you, so please don't post like this again. Ditto for unsubstantive comments in general.

What evidence do you want? They only have 5 accounts with the ability to downvote. This is why comments always have -4 with anything critical of apple.

HN's software puts a floor of -4 on downvoted comments.

Please stop this now.

I can think of a trillion reasons...

What liars. Apple has done this before as well, when they said they had "never heard" of PRISM, despite a Snowden leak showing the exact opposite.

https://www.theguardian.com/world/2013/jun/06/us-tech-giants...

I mean what are they gonna say? "Yes, we have been aware that an unknown but possibly huge number of our servers have been compromised, but decided to keep our customers in the dark"?

Your point being ?

I think the point is that actual honesty from these megacorps would be so surprising that even raising the possibility of it happening is so absurd it feels like parody writing.

Last week Facebook was reasonably transparent about a hack affecting tens of millions of users.

This may be the first time in the history of the internet a statement from Facebook has ever been held up as an example of honesty and transparency from a corporation in America.

The GDPR has already called out Facebook for lack of info in its response to the breach: https://www.cnbc.com/2018/10/02/facebooks-muddy-account-brea...

Not sure why you'd pick that example.

Mostly because I believe that, while it's important to maintain healthy skepticism around privacy and security issues, I also believe that we don't benefit from cynical hyperbole.

That's because they have been under tremendous scrutiny recently.

The trick seems to be having sufficiently uninformed people in all positions that might get to write that kind of response. No need to feign ignorance when you can have the real thing.

At least in the case of Amazon, the denial is published under the name of the CISO, Steve Schmidt, who previously worked for the FBI for a decade.

https://aws.amazon.com/blogs/security/setting-the-record-str...

Not saying he's not lying but it definitely raises the stakes.

Qualification is no hindrance to being or of the loop though, deliberately or not. It just makes it more embarrassing. Ignorance itself is still easily attainable, just "shoot the messenger" (this will make you seem tough and thorough to your higher ups) a few times when bad internal news escalate to your level and you will remain in blissful ignorance for the rest of your tenure.

Regarding both cases:

https://en.wikipedia.org/wiki/Gag_order

I wonder how legit it would be for an US company to set up an internal gag order? Running on compromised hardware is practically a nuclear meltdown (pun intended) and publicly admitting to it as well.

The event was probably classified as a national security incident which would compel the affected parties to not disclose the event.

How does that actually work? How far down the chain of related facts to the national security incident are parties allowed/required to lie? If facts can be used to triangulate the secret, that can't be disclosed, right? Are incidents like this like a little fact-bomb which can be used to legally hide other institutional facts under its cover?

There most likely are classified legal constructs that compel speech. You see that with the PRISM denials by Apple and Google.

s/legal/extralegal/

"Secret law" is an oxymoron.

I assume it’s like national security letter. Only people in the company that has knowledge would be the ceo, general counsel, and people working directly to mitigate the issue. PR and corporate communication wouldn’t have any knowledge on the incident. I wonder how you collect insurance for these types of incidents if you can’t disclose them.

Weren't they uner a gag order? Even if not that's a risky thing to acknowledge.

The Snowden leak confirmed Apple's statement. It showed that Apple worked with the FBI on a system to process court ordered user data requests. Apple would obviously have no knowledge of the code name of a downstream NSA system that processed that data.

>Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them.

An uncharitable reading, but this statement does not exclude the possibility of investigations by 3rd parties hired by Apple.

Neither does it exclude the possibility of internal investigations of which Apple's press office is unaware. If Tim Cook simply said this never happened I'd believe him. This elaborate denial suggests otherwise.

I mean: > The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation.

The article says they shipped the boards to a company in Ontario, lawyerly:

>…At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.

They have to deny it. If the allegation is true, it means the Chinese CCP Gov knows which computer parts are produced specially for US gov or certain big companies and target precisely. There has to be some deep link for information flow to allow that. Anyway, worth further digging.

That seems plausible, given what we know about Amazon employees with ties to China disclosing private sales info or changing reviews in exchange for cash bribes. https://abcnews.go.com/Business/amazon-probes-report-workers...

Main discussion of that post: https://news.ycombinator.com/item?id=18138990.

I'm actually kind of sympathetic to Apple here.

One of the company I worked for once received request from a news outlet about potential rumor around us, and bullied our CEO into an interview to disprove that rumor.

Then the journalist picked several quotes out of context as proof that rumor being true.

While I don't trust megacorps, I don't know if I can trust journalists more when a major breaking news is on the line.

This reminds me of that old story about the Xerox copy machines that the Soviet Union bought.

Where each unit was planted with a image recorder. And for years, the American spy agencies had a great laugh, that they were able to intercept all the documents that the Russians made a copy of.

Back then, this was an off-network infiltration. Where the copied images, were retrieved during regular servicing intervals by a Xerox technician.

Huh crazy, I hadn't heard about this before! Just found a couple posts on it [0][1], for anyone who's interested.

0: https://electricalstrategies.com/about/in-the-news/spies-in-...

1: https://discover.cobbtechnologies.com/blog/the-soviet-union-...

I had never heard of this...but now I can only imagine how exciting this must have been as one of the engineers-- working on a top secret project for the CIA in an abandoned bowling alley: https://electricalstrategies.com/about/in-the-news/spies-in-...

edit: whoops, looks like amatecha beat me to posting more info

Exciting, but not so ethical. We owe society to put our knowledge towards making it better for all people, not just "our team".

It's important though to make sure that your team keeps tactical advantage so that it can continue existing. Maybe someday mankind will find world peace but in our current world there are a lot of nations that hate each other still and wouldn't hesitate to take advantage of weaknesses of other nations for personal gain.

If our team is resorting to unethical and immoral ways to gain that advantage, then we can't take the moral high ground and also can't complain when the other team also does "whatever it takes" to gain an advantage.

Also, security through obscurity is, as we know, an illusion. Information always finds a way out.

I understand your point, but there should be limits.

As I tell my small human, The good guy must do only good, or he is also the bad guy.

Good on you, man. The world needs more good guys.

What do you tell your small human about the current behavior of many prominent adults, or if the small one is too young to see what's happening, what will you tell them?

But once you create the technology and hand it over to someone else, you have no assurance that it's only going to be used against the "bad guys":

Judging by the number of parts ordered from Xerox, Zoppoth believes that spy cameras may have been installed in photocopiers all over the world, to keep an eye on U.S. allies as well as enemies

It's equally important to question your "team".

In the developed world, hatred is often manufactured to gain power... The leaders of a nation can cause more harm in their quest for glory than the average citizen of an opposing state.

> Exciting, but not so ethical. We owe society to put our knowledge towards making it better for all people, not just "our team".

Our team promoted liberty and democracy worldwide (see below). The other promoted totalitarian dictatorship, labor camps, etc. If your team is liberty and democracy - self-determination for all - then it's not 'our team', it's everyone's team.

To address the elephant: Yes, there were many, many exceptions to the West's support of democracy, and many of them were awful (Indonesia, the Mideast, Zaire/Congo, and many others come to mind) but beyond a doubt, during the Cold War and after democracy and liberty exploded - stop and compare it to any other time in human history. Lots of bad things happened, but compare that with Soviet-dominated areas such as the USSR itself and Eastern Europe.

+1, couldn't agree more. I wish more nations/Governments would think like this, but too many are motivated by more wealth and power. At some point, you would think, the animal would satiate itself. It appears that is not the case.

Or the IBM Selectric Typewriter implant.

http://www.cryptomuseum.com/covert/bugs/selectric/

One of my personal favorites.

wow. what goes around comes around.

Like the IBM typewriters with keyloggers that the Russians planted on US diplomats during the 80s?

https://www.google.com/amp/s/arstechnica.com/information-tec...

I was born raised, and went to CS undergrad program in China before come to US. Yes. I can testify that the "Xerox Copy Machine" is the first thing they teach you about information security :p

I think Russia became much more aware of that kind of security threats. CPUs, motherboards etc are produced internally, i.e. https://en.wikipedia.org/wiki/Elbrus-2S%2B

I don't know which is more disturbing here.

That the Chinese military is technically competent enough to pull off such a thing.

Or that they are incompetent enough, to not secure their own back doors and networks, and allowed the FBI, NSA, and other American government organizations, the ability to counter-hack them, and monitor all their internal communications.

The truth is somewhere in between.

So, this article is basically saying: China got clever, and spied on us. We spied back on them, and we also hacked all of their internal communications too, so basically, we own them.

Well, at least America is finally admitting, that they too, actively spy on other countries. And can easily infiltrate any other country that they want.

> America is finally admitting, that they too, actively spy on other countries.

The Snowden revelations made that very clear to all years ago, no?

One doesn't even need Snowden to know this. Given even a little reading of the IC's (intelligence community), and more specifically CIA's, history. What else do you think they do all day, if not for spying on other countries?

As mercutio2 mentioned in a reply to your same parent, the controversy w/r/t Snowden is domestic spying.

> As mercutio2 mentioned in a reply to your same parent, the controversy w/r/t Snowden is domestic spying.

In the US maybe, but most of the US's supposed "allies", and their citizens, would probably disagree with that interpretation.

> we also hacked all of their internal communications too

my take on the article was that US counterintelligence sort of cobbled together some hacked phone communications with some info from human agents working in or around or near Chinese factories. to say that "we own them" maybe goes too far?

The US has never denied that it spies on other countries’ citizens and state apparatus. Every nation does.

The only controversy has been about whether the US intelligence community spies on US citizens without warrants, which used to be considered out of bounds, but is now awash in murkiness.

Not surprising, when you consider that every intelligence agency spends at least 10x more money on offense than defense. I first became aware of this when I worked at a company where we had the NSA "defense" as a customer. We tried and tried to get contacts on the offense, because it was a much bigger market opportunity, but the company went under before we succeeded. So it's our Ferrari offense against their Fiat defense, and vice versa. (Not at all a comment on skills or motivation BTW. There might be an element of having one's pick of a larger talent pool, but mostly it's about resources.) It's practically inevitable that we'll both score lots of points off each other.

> Or that they are incompetent enough, to not secure their own back doors and networks, and allowed the FBI, NSA, and other American government organizations, the ability to counter-hack them, and monitor all their internal communications.

Wait.. did I miss something in the article? The closest thing to that, I remember happening, had the roles reversed [0], and should be an important lesson about not underestimating the opposition.

[0] https://foreignpolicy.com/2018/08/15/botched-cia-communicati...

Who else found the design of the page made the article difficult to read? I know darkness, spies and hacking go hand-in-hand but it's a bit too much imho.

It was perfect on a mobile device.

Good point. Definitely seems designed for mobile (only).

I know this is going off the main topic. But it is very strange that I discover Black background, as in Dark mode in macOS, doesn't work at all despite what I have always thought it would be cool. But Black background on mobile devices work absolutely great!.

And I have no idea why, I searched on Google and couldn't find any decent answer. All results were either OLED being battery friendly with switched off pixels or other mobile UX for Dark mode. None of them describe the difference felt in Desktop and Mobile. Any link or explanation would be much appreciated.

As a fan of dark color themes and displays, I've shared a bunch of thoughts related to this on my personal blog. But I'd like to share a few quick thoughts below. I am not an expect on these matters, so these are just my opinions:

1. Glare. When reading on a small device, the amount of glare reflected in the screen space occupied by black pixels may be fairly minimal depending on your reading environment and positioning. Also, although mobile devices use glossy display surfaces, they tend to have several anti-glare layers in the display stack. You said your computer is macOS, so I suspect you're unlucky enough to be reading on a glossy laptop display. With "dark mode" color schemes, your eyes can more easily see the reflected scene (maybe your office lights, an exterior window, or even your own face). And the focal length of that reflected scene is 2x your reading distance to the screen. That reflected scene at an extended focal length is more relaxing for your eyes to focus on. So in order to read, you need to fight your natural tendency to relax and look at the reflected scene.

If you are lucky enough to be reading on a matte desktop display (typically a professional or prosumer monitor, such as a Dell UltraSharp or LG 43MU79-B), the glare will be minimal and it should be much easier to read.

2. Pixel density. I contend that one reason dark themes have become more prevalent in recent history is thanks to wider adoption of high pixel density displays. At a legacy density of approximately 75 dots per inch, the stroke weight of bright text on a dark background appears too faint if the strokes render as just one pixel in width. Higher pixel density allows for the strokes of letters to be wider than a single pixel, allowing for greater clarity. If you ever designed a dark theme in the days of ~75 dpi displays, you might naturally favor bold text as the default because it was considerably easier to read. (Interesting sidebar: many console oriented bitmap fixed-width fonts historically used two pixels for stroke weight, especially in the horizontal dimension, presumably because they were designed to be used bright-on-dark.)

Now, you did again say you were reading on macOS, so your display's pixel density is probably higher than ~75 dpi. But a MacBook Pro is still only ~220 dpi. A Surface Book is ~260 dpi. A Dell XPS laptop with a 4K display will be a little higher still (maybe ~300 dpi). But many phones are using 450+ dpi displays. The stroke weight of a character on a mobile device is several pixels wide, so it's highly defined and clear.

3. Font selection (related to above). Bloomberg has selected a serif font, presumably because they are a media organization and serif fonts are typically used for article bodies. However, combined with pixel density, the serifs will lose a lot of their definition and (in my opinion) reduce readability versus a sans-serif font. As an experiment, pull up the dev tools and change article[data-brand="businessweek"] .body-copy p to use sans-serif and see what you think. It may be marginally easier to read.

4. Don't discount OLED. The contrast that Bloomberg selected is maximal (pure black background and pure white text) and that works well for OLED since the background vanishes entirely. However, since most desktop and laptop monitors are not OLED, you're still getting backlight bleed, so the contrast is imperfect. Especially combined with the glare factor above, my experience is that given LCD backlight bleed, it is better to use a dark gray background instead of stark black. This makes the backlight bleed less distracting, for lack of a better word. The background ends up looking more uniform.

As with above, try adding "background-color: rgb(40,40,40)" or similar to .body-copy and see what you think.

yep I have blurry vision coming back to HN for the comments...

I lasted about 3 seconds before I toggled reader view.

I found it perfect on my OLED mobile screen.

OLED works better with darkness than with bright backgrounds.

This is because, contrary to what a lot of people say about dark themes (even though it's mostly a meme at this point), dark letters on white background are better to read than white letters on black background.

"People with astigmatism (approximately 50% of the population) find it harder to read white text on black than black text on white. Part of this has to do with light levels: with a bright display (white background) the iris closes a bit more, decreasing the effect of the "deformed" lens; with a dark display (black background) the iris opens to receive more light and the deformation of the lens creates a much fuzzier focus at the eye." - Jason Harrison – Post Doctoral Fellow, Imager Lab Manager – Sensory Perception and Interaction Research Group, University of British Columbia

This quote describes the situation well I think, why the opinions differ and why neither dominates UIs really.

I actually have astigmatism so that might be why I think it's even harder to read on black/dark themes. So yeah, definitely true in my case.

It's interesting to finally have some explanation as to why I have always found dark themes worse, also having astigmatism.

How common is it for a seemingly standard security audit to inspect motherboards with such level of detail or at all?

They likely needed to have the exact official schematic of the motherboard to compare every single detail of the hardware with.

While you may find a particular attack if you're looking for, in general, it's impossible for even the most thorough audits to check for the whole class of such attacks. You're not going to look into the chips. Well, you can, but that's prohibitively expensive and destructive - even if you could check that this chip was okay, then you still have to throw it out after analysis and plug in a different one.

The only feasible thing to do is thorough audits of all the supply chain for every component in your system, ensuring that your supply chain does not include even a single chip from an "untrustworthy" supplier, and even then it reduces the chances of an attack but does not eliminate it.

Zilog Z80, 1976

> Faggin: Yes, we were concerned about others copying the Z80. So I was trying to figure what we could do that that would be effective, and that’s when I came across an idea that if we use the depletion load the mask that doesn’t leave any trace, then I could create depletion load devices that look like enhancement mode devices. And by doing that we could trick the customer into believing that a certain logic was implemented, when it was not. Then I told Shima, “Shima, this is the idea how to implement traps. Put traps, you know, figure out how to do the worst possible traps that you can imagine,” and then Shima with his mind, that was steel mind, was able to actually figure out a bunch of traps that he could talk about.

> Slater: You want to tell us a little about that Shima?

> Shima: I didn’t count [on] talking about that mostly. I placed six traps for stopping the copy of the layout by the copy maker. And one transistor was added to existing enhancement transistors. And I added a transistor looks like an enhancement transistor. But if transistors are set to be always on state by the ion implantations, it has a drastic effect on very much. I heard from NEC later the copy maker delayed the announcement of Z80 compatible product for about six months.

http://archive.computerhistory.org/resources/text/Oral_Histo...

There are vendors in this space. See: https://www.harris.com/sites/default/files/supplier_counterf...

Good starting point.

The logical extension of this is to embed a sleeper chip between board layers that can be activated by a radio signal, then try to disperse it widely within the DND.

Any serious conflict with China would then look like the first day of the second Cylon war...

Of course, they would probably have to get a Chinese boat really close to a US warship a few times beforehand to test the system. All they would need to do is to receive some sort of a ping back to confirm receipt of the signal. Could be simple as an innocuous visit to a particular page of a website.

It would probably look like brinksmanship or posturing, or a navigation error of some sort.

Doesn't the printed circuit board also need to be redesigned, to allow for the spy chip to be inserted?

So, this means that China forced multiple companies to modify their processes, in order to pull this off.

1. The PCB designer

2. The PCB printer, if it's a separate company

3. The company assembling the final product

And if they forced the PCB to be redesigned, then wouldn't this be a immediate red flag? But this means that the customer, Supermicro, would have to audit the PCB results as well.

They wouldn't need the PCB designer. They would need to suborn (employees of) the PCB manufacturer and the assembly house. These will be supplied with the PCB layout (in Gerber or ODB format, for making the PCB) and the bill of materials and pick and place file (for assembling it). With these files you can reverse engineer the schematic and carry out the modifications described. The PCB manufacturer might well be a sub-contractor to the assembly house.

Most electronic engineers don't have the software tools for or experience of sophisticated reverse engineering but there aren't any major conceptual barriers. They would have to go from the geometry of the tracks and pads to a connectivity graph (very automateable), then collect the pads into footprints for components (probably partly automated), then identify those components and the functions of their pins (easy with a complete BOM) and then workout the circuit function (should be straightforward for standard parts and circuits).

There might not be automated tools for making the desired changes, in which case they will have to manually draw the new track geometry on the Gerbers, add the parts to the BOM and pick and place files and change or nobble the test criteria / files. Hard work but quite straightforward.

These skills will be developed by people doing legitimate industrial reverse engineering as well as espionage / intelligence. I would think there are also unfortunate cases where firms have to reverse engineer their own products after losing the original files.

"Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not."

Why does this keep getting quoted in the comments. Yes we read the article too.

Because it's so awesome. This must be one of this journalist's career highlights, to be able to put something like this in a mainstream serious reporting piece.

So damn true! It's one of those things which you wait all your life to slip into an article/ convo.

That line in the article made me LOL. Douglas Adams would be proud.

> Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.

Looks like SuperMicro was delisted by Nasdaq just over a month ago. Wonder if it has anything to do with this under their pretty poor surface excuses for not being in compliance.

This article frightens me considering that you can almost always identify the endusers for a lot of the personal devices shipping directly from China.

white text on 0,0,0 black background... ugh my eyes do not thank thee

In my OLED phone screen, my eyes do thank me (although I have anyway the black mode in Kiwi browser all the time)

Imagine if this had happened the other way around. Death threats and calls of oppression and foul play would only have been the start.

Wow, the Obama administration knew this was happening, even before it happened, and did nothing to stop it?

This story could easily be interpreted as anti-China propaganda. Could you think of any other sovereign power who would want a backdoor into servers used by billions of people across the internet? Hardware comes from China. This doesn't mean that the Chinese government orchestrated the attack. The United States government is having a trade war with China. This article's publication isn't just coincidence.

Further, those with insight about how to use the backdoor definitely made use of them. This is one of the central issues with government-mandated backdoors. They are introduced for one group but used by others.

It could be interpreted that way, but this sort of activity isnt novel or unknown.

The Feds run key hardware through third parties to detect counterfeit hardware. The “new” thing is the targeting of devices headed for commercial customers.

Anti-China propaganda? These chips were designed by the Chinese military and inserted by PLA agents. Do you have to start calling things propaganda to make it seem like this is baseless criticism of China to those who have not read the article?

These chips were REPORTED TO BE designed by the Chinese military and inserted by PLA agents.

Big difference, which is parent's point.

> Could you think of any other sovereign power who would want a backdoor into servers used by billions of people across the internet?

I cannot think of any sovereign power that wouldn't.

This comment could easily be interpreted as pro-China propaganda.

> Could you think of any other sovereign power who would want a backdoor into servers used by billions of people across the internet?

The US's NSA would want and probably does have such a thing.

your comment looks like an pro-china propaganda. In china the state has heavy control on all the companies. The hardware is manufactured in china. I don't think something like this is happen without the state's knowledge.

It seems like many people take communist-like propaganda at face value.

The Chinese government would like to have almost-complete control. The Chinese government publicly says it has high levels of control.

But when they can't effectively regulate their medical (mass HIV infection from blood plasma needle reuse), food (tainted milk), or chemical (unlicensed mass CFC production) industries... reality seems to differ.

So, suppose you're Supermicro. When some CPC official comes around to tell you to make your technology a little easier for the intelligence department to access, you're going to do it.

Companies in China (especially those in the tech sector) have to keep close ties to the government, and most of their leaders are members are of the party. You don't GET to be a multi-billion dollar tech company* in China without toeing the party line [1] [2] [3]

The issues around regulating food safety and vaccines that you mentioned are irrelevant.

* Foreign companies must operate Chinese subsidiaries to run their operations in China.

[1] https://www.wsj.com/articles/beijing-pushes-for-a-direct-han...

[2] https://qz.com/1102948/chinas-communist-party-is-all-in-on-t...

[3] http://chinamediaproject.org/2018/05/02/tech-firms-tilt-towa...

I see your *, so I assume you do know SuperMicro is a US company? And that the breaches reportedly happened at local Chinese subcontractors?

Everything you say seems to be valid for Chinese companies. Which SuperMicro... isn't.

If they want to start auditing their incoming supply from China more closely, or even shift production elsewhere, there's nothing except cost stopping them from doing so.

And the rub is that any competitor also using Chinese supply (for cost savings) is vulnerable to the same attack. SuperMicro was presumably targeted because of their size and global customers.

Apple is not Apple Inc. in China, it is Apple Computer Trading (Shanghai) Co., Ltd. SuperMicro will have its own subsidiary/subcontractors to handle their operations in China.

As having worked for a subsidiary of a large multinational company in China, I can tell you that a lot goes on that you might find surprising. Middle management and those involved with establishing the deals with subcontractors would often have arrangements to make money through various means, such as through IP transfer or property theft. The subcontractors themselves are Chinese companies.

At our place a lot of shady things were going on and were, thankfully, found after a number of years. The global HQ had to step in and fire around 1/3 of employees working at the China branch.

Now, there is no reason to suggest that this was related to any government policy or request. But it should be clear by now that the CPC are not exactly opposed to shady things happening to foreign companies. My point is that this is an an environment where something like this can happen quite easily, especially when a lot of technology companies have close ties to the state.

The United States is having a trade war with China partially because of this kind of thing.

> Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design.

> During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.

The only interesting thing about this is left out. Who planted it is clear (someone told to do so) but not a single time is it questioned who they planted it for. Smells like false flag to me.

We think China does X Y and Z but we know the US does X Y Z and the rest of the alphabet. So unless something specific is leaked that shows who actually ordered this, logic would point at the US.

> logic would point at the US

But then, since the logic points to the US it only stands to reason that is a false false flag! Why those wiley spies in Beijing really are clever. Discredit The USA and plant spy chips in important computers!

> ...but not a single time is it questioned who they planted it for...

The article and some accompanying reporting on Bloomberg audio/video segments says the attack seems targeted relatively specifically towards nearly 30 organizations (only US-based organizations were mentioned as targets, unknown if the list included organizations based in other nations). One known vector was through four subcontractors in China that built the boards for Supermicro's main Shanghai factory, specifically by bribing and/or coercing managers of those subcontractors' factories to go along with accepting the chip shipments and to make changes to the plant floor from the design to perform the chip insertions.

Designing and building a chip like this and then mounting the logistical effort to performing the insertions costing some non-trivial funds, coupled with the known targets, (Amazon didn't seem specifically targeted, Elemental a company they acquired was, who notably has US national security clients), form the circumstantial allegation that a PLA spy unit was behind the attack. You are correct that this doesn't entirely rule out a false flag possibility, but until we get more details about this, we're operating in the dark.

A false flag is an interesting supposition, but how would the US benefit from successfully convincing the world of the false flag's cover story?

The US has a few things to gain from this story: Economically, because Chinese products are perceived as compromised. Politically: because the Chinese government is seen in the offense.

The US has something to loose too: Being perceived as dependent on Chinese manufacturing and potentially compromised down to military hardware. (The first everybody knows, the second would be devastating for trust.)

All in all it would be a weird angle for a false-flag attack.

The article specifically mentions the PLA as the bad guy. So that part was not left out. The whole story is hearsay so far. It's plausible, but best read with a good dose of salt.

If we believe the story up to the point where subcontractors are forced into planting the chips, we must accept that it's easiest and least risk for Chinese government actors to force them into it. Mounting a false-flag attack at Chinese subcontractors would be exceedingly difficult for foreign agents. They'd likely blow their cover when trying to represent Chinese officials.

If you want an alternate version, I suggest you start with the easiest: The whole thing didn't happen. And Bloomberg is a victim of propaganda.

looks like no one noticed the word "Troll" in OP's username

Reply on news.ycombinator.com