This is a brown M&Ms problem: https://conversableeconomist.blogspot.com/2020/10/the-no-bro...
If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.
- no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension.
- a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement.
"No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc.
Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms.
If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
This would be more like finding a container coming from dockahub.com and then ultimately finding out that it was a legitimate typo.
It mostly affects low complexity components that are easy to clone so a BMC would be unlikely, but even that is not safe as sometimes used components de-soldered from other products make it back into the supply chain too.
(And I understand it they did this knowing the effect it would have. It wasn't some accident.)
More specifically: many of FTDI's devices, including the FT232RL, can store configuration data in an EEPROM. EEPROMs are conventionally organized as an array of bytes, but FTDI devices always access the EEPROM 16 bits at a time. The FTDI devices include some commands to read/write the EEPROM. For convenience, these commands use an address counting in bytes, but the official drivers and tools only ever use even addresses (so that the EEPROM addresses accessed never go "across" fields).
The FT232RL is one of FTDI's parts which has an internal EEPROM. An implementation quirk means that any attempt to perform a EEPROM write on this part with an odd address will be ignored. (This quirk is specific to this one part -- most of FTDI's other parts will perform this operation as expected.) This fact was apparently not known by the developers of the FT232RL clone; it will perform the write.
FTDI released a driver update which would, upon detecting a FT232RL, attempt to write zeroes to several odd addresses near the locations used for the device's VID/PID (vendor ID and product ID). These writes were ignored by the genuine devices, but were interpreted by clones as writing zeroes to these fields. This caused the devices to fail to be recognized when next connected to a host.
What's funny is that, as far as documented functionality is concerned, the clones work perfectly. In some regards, they actually work better than the originals!
Usually, when programming the EEPROM in these devices (e.g. with FTDI's tools or open source ones), you write the whole thing at once, so those pairs are always written sequentially and it works anyway. However, if you try to write to some words at even addresses without following up with the next one, the FT232RL misbehaves and ignores the write.
Specifically, what happened seems to be that the FT232RL uses an internal 32-bit EEPROM. So when you write to even addresses, those writes are staged to a buffer in anticipation that you will subsequently write to the next odd address. At that point all 32 bits are written. The bricking code only ever wrote to even addresses, which involved a preimage trick to keep the existing checksum valid, since the checksum is at an odd address they couldn't write to. Instead they calculate what the value for the word before the checksum should be to make the existing checksum valid, and write that instead. Genuine chips would just stage all these writes in a holding register and never get to commit them.
Basically, FTDI decided to use an internal 32-bit EEPROM macro when developing this chip, and came up with this hack to shoehorn it into the 16-bit protocol (in what is arguably a buggy way). The clones just implemented 16-bit writes properly, which is the same thing all other FTDI chips (with external 16-bit EEPROMs) do.
(Source: I'm the guy who first figured all this stuff out back when this happened, including reverse engineering the FTDI bricking code and writing an unbricker).
The code was blatantly deliberate; to pull this off they had to perform a preimage attack on the EEPROM checksum, etc.
No, I don't think that was the thought process. I think they wanted to send a message and I think they wanted to test the waters to see if anyone would hold them accountable for deploying first party malware.
The miraculous thing is that FTDI escaped criminal prosecution for this.
The vandalism though, that's definitely illegal, and outright criminal.
What FTDI did, on the other hand, is malware and clearly illegal. It is destroying private property and I guarantee violated multiple laws in many countries.
Of course this kind of vigilante justice by FDTI is illegal, but who is going to press charges if that means they will get their devices taken from them and destroyed?
There's this:
https://twitter.com/marcan42/status/695292366639378433
And also, the way FTDI bricked the clones (only) was by exploiting the fact that they also implement the EEPROM write command in the sane way other FTDI chips do, instead of having the write staging quirk of the "genuine" FT232RL that they introduced when they wired the internal 32-bit EEPROM block to a 16-bit interface.
So all around, the clones work better than the originals in several ways.
Yeah, nightmares. It doesn’t matter if they’re crap if it’s what you have to work with.
I really never understood why the open source world went with FTDI when the CP210X was perfectly available (I’ve been using them for ~12 years now), in fact I found about the FTDI chips later and I was very confused of why would I even pay more for an inferior part.
But whatever appears in hobby boards is bound to appear in some products.
> it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
The notion that someone would pause a line over this (even if we were not in the middle of unprecedented component/manufacturing/shipping disruptions) is beyond fucking absurd, much less that anyone would do so until a "re-check of the supply chain" is completed.
Production schedules are tight as hell.
You miss your deadline for getting the board assembled, they don't make it to the line or factory putting the boards into the chassis on time.
That means they've started on another job and now you wait until they have free time on the line.
That means you don't get your container to the port on time.
That means you miss the space you had paid for on the ship.
That means you miss your product launch date. Possibly by months; especially right now, shipping is severely constrained.
That means your competitor takes your lunch money.
Are you familiar with what happened with the counterfeit capacitor plague?
https://www.google.com/search?client=firefox-b-d&q=capacitor...
Keep on cranking out that production line with your suspicious/manufacturer-source-unknown parts with improper labeling on them and end up in a situation with hundreds of millions of dollars of financial damages due to burst capacitors.
Not every manufacturer has its absolute and highest goal set as massive quantity/cheap and shoddy QA/lowest price/highest volume possible.
For something like a 100Gb ethernet switch, standards should be much higher than the PCBs of a bunch of $40 802.11ac wifi routers to put in a shiny box and sell at Best Buy.
The reason I had them available is that they were supposed to go into a machine that we sold, but since someone in Receiving missed a step in their inspection process, they all had to be marked as Non-Conforming and could not be used in the machine, even though there was absolutely nothing wrong with them. Since they were a special order, the vendor wasn't about to take them back.
So it was either toss tens of thousands of dollars worth of hardware in the dumpster or try to find a use for them internally.
If I saw a misspelling like that it would indeed be cause for hitting the Big Red Button.
Quite frankly, it doesn't matter how much process there is to avoid this sort of thing. It is a big industry and we should expect an error as trivial as a typo to slip through eventually. The reader would have been much better served if the author was upfront about that, discussed why the error was significant, and offered some insight into how the industry normally avoids this type of error. Maybe they did get to that point eventually, but this reader will never know since this reader gave up when the article felt self-serving rather than informative.
There's no proof that tampering has happened but there is proof that not only it might have happened but also that there is no way to tell if it did happen short of, idk, forensically auditing the network traffic of your $10,000 switch with the help of another $10,000 switch and someone skilled enough to pick apart the output over a long enough period to determine if tampering happened.
I'm not paranoid to do something like that but if the US Government isn't paranoid enough to do something like that then something has gone horribly away in government cybersecurity.
Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.
Stopping the production run and delaying shipments would have a palpable impact on shipment numbers. Most likely it was noticed, corrected for later batches, and they made the call to run this round anyway because it had zero effect on function.
(Source: If you work in hardware long enough, you’ll have to make similar calls like this eventually)
No idea how it works in electronics, in aerospace so you would do thorough testing on the affected units. And if they meet the spec, someone would make a call regarding relabeling. Considering how expensive relabeling tends to be, it isn't unreasonable to assume the parts just got a concession for the labels. I wouldn't read to much into it.
</s>
It's actually so well know that http://www.grauniad.co.uk redirects to the correct sight
It would not be the first time someone finds exploitable firmware bugs and vulnerable BMCs through Shodan.
Some politicians make a point of displaying this so that donors know they aren't going to be somebody smart, ambitious and dangerous like Ralph Nader.
One example was Karen Bass, who bent over backwards to praise Scientology once.
I'm pretty sure "they" that design the stickers are very different from "they" that design the chip... Similarly chip verification has rather a different skill set and attention to detail than someone doing "sticker QA".
Also I doubt anyone was doing sticker QA, I'd guess that this was probably an outsourced mistake (I used to work in printing when I was younger - the mistakes you wouldn't believe).