Dell and HPE switches come with 'American Megatrands' stickers
servethehome.com
servethehome.com
1. Article author discovers "American MegaTrAnds" sticker on a chip in high-end switches.
2. Author fears supply chain tampering.
3. Author contacts American Megatrends (AMI). Hijinks ensue. AMI eventually confirms that it's a typo.
4. Therefore, there is no supply chain tampering.
5. But author is concerned about what this means for the state of supply chain verification.
This is a brown M&Ms problem: https://conversableeconomist.blogspot.com/2020/10/the-no-bro...
If they missed something as obvious as this, who knows what other problems are going on in supply chain security or total lack of QA.
- no brown m&m's specifically calls out no brown m&ms in a list of requirements, and uses it as a canary for reading comprehension.
- a misspelling is an "obvious" problem, but I suspect not called out anywhere as a specific requirement.
"No brown m&ms" catches when people aren't paying detailed attention to your (presumably reasonably scoped) requirements doc.
Asking people to catch all "obvious" problems holds them accountable to an unbounded guessing game, and you're far more likely to catch people out, simply because of differences in where they choose to focus.
Those who have worked in electronics manufacturing will immediately see it as a possibly scary sign of counterfeit components making their way into the supply chain. Same as what happened with counterfeit capacitors in east asia. Much like the early 1980s Van Halen tour example linked above, it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
Article didn't fully explain why the sticker matters so much, so that left me scratching my head. (my gut reaction was "well, wouldn't a genuine sticker still be easy to counterfeit?). But based on your explanation this is more of a smell that everyone in the electronics manufacturing space is culturally attuned to. So, the fact that it slipped by so many people does indicate a slippage of norms.
If I draw a comparison to software to bring it closer to something familiar to me... would this be like inconsistent variable name formatting? CamelCase in some places and snake_case in others? To an outsider, arguably inconsequential, so insisting on consistency here might seem OCD to them, but to someone who's worked in the space it's actually a useful marker of general detail orientation.
I’d say it’s more like misspelling your own company name in the JPEG logo in the HTML welcome email sent out to new users, and no one notices for a year.
This would be more like finding a container coming from dockahub.com and then ultimately finding out that it was a legitimate typo.
It mostly affects low complexity components that are easy to clone so a BMC would be unlikely, but even that is not safe as sometimes used components de-soldered from other products make it back into the supply chain too.
(And I understand it they did this knowing the effect it would have. It wasn't some accident.)
More specifically: many of FTDI's devices, including the FT232RL, can store configuration data in an EEPROM. EEPROMs are conventionally organized as an array of bytes, but FTDI devices always access the EEPROM 16 bits at a time. The FTDI devices include some commands to read/write the EEPROM. For convenience, these commands use an address counting in bytes, but the official drivers and tools only ever use even addresses (so that the EEPROM addresses accessed never go "across" fields).
The FT232RL is one of FTDI's parts which has an internal EEPROM. An implementation quirk means that any attempt to perform a EEPROM write on this part with an odd address will be ignored. (This quirk is specific to this one part -- most of FTDI's other parts will perform this operation as expected.) This fact was apparently not known by the developers of the FT232RL clone; it will perform the write.
FTDI released a driver update which would, upon detecting a FT232RL, attempt to write zeroes to several odd addresses near the locations used for the device's VID/PID (vendor ID and product ID). These writes were ignored by the genuine devices, but were interpreted by clones as writing zeroes to these fields. This caused the devices to fail to be recognized when next connected to a host.
What's funny is that, as far as documented functionality is concerned, the clones work perfectly. In some regards, they actually work better than the originals!
Usually, when programming the EEPROM in these devices (e.g. with FTDI's tools or open source ones), you write the whole thing at once, so those pairs are always written sequentially and it works anyway. However, if you try to write to some words at even addresses without following up with the next one, the FT232RL misbehaves and ignores the write.
Specifically, what happened seems to be that the FT232RL uses an internal 32-bit EEPROM. So when you write to even addresses, those writes are staged to a buffer in anticipation that you will subsequently write to the next odd address. At that point all 32 bits are written. The bricking code only ever wrote to even addresses, which involved a preimage trick to keep the existing checksum valid, since the checksum is at an odd address they couldn't write to. Instead they calculate what the value for the word before the checksum should be to make the existing checksum valid, and write that instead. Genuine chips would just stage all these writes in a holding register and never get to commit them.
Basically, FTDI decided to use an internal 32-bit EEPROM macro when developing this chip, and came up with this hack to shoehorn it into the 16-bit protocol (in what is arguably a buggy way). The clones just implemented 16-bit writes properly, which is the same thing all other FTDI chips (with external 16-bit EEPROMs) do.
(Source: I'm the guy who first figured all this stuff out back when this happened, including reverse engineering the FTDI bricking code and writing an unbricker).
The code was blatantly deliberate; to pull this off they had to perform a preimage attack on the EEPROM checksum, etc.
No, I don't think that was the thought process. I think they wanted to send a message and I think they wanted to test the waters to see if anyone would hold them accountable for deploying first party malware.
The miraculous thing is that FTDI escaped criminal prosecution for this.
The vandalism though, that's definitely illegal, and outright criminal.
What FTDI did, on the other hand, is malware and clearly illegal. It is destroying private property and I guarantee violated multiple laws in many countries.
Of course this kind of vigilante justice by FDTI is illegal, but who is going to press charges if that means they will get their devices taken from them and destroyed?
There's this:
https://twitter.com/marcan42/status/695292366639378433
And also, the way FTDI bricked the clones (only) was by exploiting the fact that they also implement the EEPROM write command in the sane way other FTDI chips do, instead of having the write staging quirk of the "genuine" FT232RL that they introduced when they wired the internal 32-bit EEPROM block to a 16-bit interface.
So all around, the clones work better than the originals in several ways.
Yeah, nightmares. It doesn’t matter if they’re crap if it’s what you have to work with.
I really never understood why the open source world went with FTDI when the CP210X was perfectly available (I’ve been using them for ~12 years now), in fact I found about the FTDI chips later and I was very confused of why would I even pay more for an inferior part.
But whatever appears in hobby boards is bound to appear in some products.
> it's a reason for hitting the big red "OKAY, STOP EVERYTHING" button and re-check of all of the other components and supply chain going into the product.
The notion that someone would pause a line over this (even if we were not in the middle of unprecedented component/manufacturing/shipping disruptions) is beyond fucking absurd, much less that anyone would do so until a "re-check of the supply chain" is completed.
Production schedules are tight as hell.
You miss your deadline for getting the board assembled, they don't make it to the line or factory putting the boards into the chassis on time.
That means they've started on another job and now you wait until they have free time on the line.
That means you don't get your container to the port on time.
That means you miss the space you had paid for on the ship.
That means you miss your product launch date. Possibly by months; especially right now, shipping is severely constrained.
That means your competitor takes your lunch money.
Are you familiar with what happened with the counterfeit capacitor plague?
https://www.google.com/search?client=firefox-b-d&q=capacitor...
Keep on cranking out that production line with your suspicious/manufacturer-source-unknown parts with improper labeling on them and end up in a situation with hundreds of millions of dollars of financial damages due to burst capacitors.
Not every manufacturer has its absolute and highest goal set as massive quantity/cheap and shoddy QA/lowest price/highest volume possible.
For something like a 100Gb ethernet switch, standards should be much higher than the PCBs of a bunch of $40 802.11ac wifi routers to put in a shiny box and sell at Best Buy.
The reason I had them available is that they were supposed to go into a machine that we sold, but since someone in Receiving missed a step in their inspection process, they all had to be marked as Non-Conforming and could not be used in the machine, even though there was absolutely nothing wrong with them. Since they were a special order, the vendor wasn't about to take them back.
So it was either toss tens of thousands of dollars worth of hardware in the dumpster or try to find a use for them internally.
If I saw a misspelling like that it would indeed be cause for hitting the Big Red Button.
Quite frankly, it doesn't matter how much process there is to avoid this sort of thing. It is a big industry and we should expect an error as trivial as a typo to slip through eventually. The reader would have been much better served if the author was upfront about that, discussed why the error was significant, and offered some insight into how the industry normally avoids this type of error. Maybe they did get to that point eventually, but this reader will never know since this reader gave up when the article felt self-serving rather than informative.
There's no proof that tampering has happened but there is proof that not only it might have happened but also that there is no way to tell if it did happen short of, idk, forensically auditing the network traffic of your $10,000 switch with the help of another $10,000 switch and someone skilled enough to pick apart the output over a long enough period to determine if tampering happened.
I'm not paranoid to do something like that but if the US Government isn't paranoid enough to do something like that then something has gone horribly away in government cybersecurity.
Will QC be improved after that? Potentially. Will there be more scrutiny around that particular supplier? I would think so. Is it a general supply chain problem? No.
It would not be the first time someone finds exploitable firmware bugs and vulnerable BMCs through Shodan.
</s>
It's actually so well know that http://www.grauniad.co.uk redirects to the correct sight
Some politicians make a point of displaying this so that donors know they aren't going to be somebody smart, ambitious and dangerous like Ralph Nader.
One example was Karen Bass, who bent over backwards to praise Scientology once.
Stopping the production run and delaying shipments would have a palpable impact on shipment numbers. Most likely it was noticed, corrected for later batches, and they made the call to run this round anyway because it had zero effect on function.
(Source: If you work in hardware long enough, you’ll have to make similar calls like this eventually)
No idea how it works in electronics, in aerospace so you would do thorough testing on the affected units. And if they meet the spec, someone would make a call regarding relabeling. Considering how expensive relabeling tends to be, it isn't unreasonable to assume the parts just got a concession for the labels. I wouldn't read to much into it.
I'm pretty sure "they" that design the stickers are very different from "they" that design the chip... Similarly chip verification has rather a different skill set and attention to detail than someone doing "sticker QA".
Also I doubt anyone was doing sticker QA, I'd guess that this was probably an outsourced mistake (I used to work in printing when I was younger - the mistakes you wouldn't believe).
Gamer troubles. Sticker troubles. But what does -gate even mean?
Later, one of Nixon's former speechwriters, William Safire, propagated the use of -gate as a generic suffix for any type of scandal, notably including very minor ones. It's likely that part of why he did this was to retrospectively diminish the perceived seriousness of the Watergate scandal.
https://www.poynter.org/reporting-editing/2015/gategate-its-...
Huh? How much cultural relevance do you expect words to have? The only word in your comment you could even make a case for is the "meta-".
One of my favorite customers actually took a calorimeter to the LEDs that were in one of our deliveries of router chassis. They felt that the Amber wasn’t quite Amber enough and so they measured it. They were right while it was amber it wasn’t spec amber. So they shipped it all back. We were very embarrassed and supply chain was given a dressing down. They missed a parts quality issue.
A typo like this is extremely indicative of a sloppy organization.
So you could say something like RAL color # 123 +- 5% in XYZ space. Or you can just specify: That color needs to be closer to RAL #123 than to any other RAL number, so then you'd also have an implicit definition of the valid color range.
This is for interface LEDs on a network device. This was one batch of a few hundred chassis, and they ended up being _just_ _slightly_ more yellow than the rest. Back they came.
Or a communication issue? How do the people looking inside the assembled product tell the people who design the product that they made a typo?
A few years back, the company I worked for created a landing page with an image of a beautiful young woman who was happy about our deals. It wasn’t my job, I was looking at the design out of curiosity and I noticed that one eye of her was looking in one direction and the other eye to the other direction. I tried to raise the issue with a few people higher up but they didn’t understand or didn’t care as they were excited for the release or busy with other stuff.
To this day, I wonder, was it intentional? Maybe it was a joke or something I didn’t get. The campaign run fine, no one talked about it. Maybe I don’t get graphic design, maybe the eye situation was a marketing message about how the lady was having eye on the numerous amazing deals of ours.
I could write a book about customers being extremely picky about "visual" quality issues. Usually as a way of either returning inventory they don't want or to drive prices down. Dressing down someone about that, and indicating that the whole org is sloppy, is not the usual reaction I would expect.
Naive question, I don't know much about this industry. But if someone from China or the NSA wanted to make counterfeit chips, why would they risk putting something different on the label? It seems like a weird place to draw the line. Would the NSA really say "oh no, we can't violate anyone's trademark!" and misspell the label and hope their entire operation isn't exposed?
The military does the same thing on badges for secure areas. They will have a misspelling of the base name or two shades of the same color on the front/back so guards can visually verify them as legitimate.
The STH guy can't spell to save his life and his grammar is terrible. His sentence structure and general writing skills are about what I would expect from a fourth-grade child.
He also is an incredible drama-llama, making mountains out of molehills; I've seen him do this time and time again.
He sees a misprinted sticker and sees supply chain attacks? Dude. This gear is assembled by people in third world countries making wages that amount to a few dollars a day or less. They don't speak English. They may not even read roman letters. I'd challenge him to do QA on any non-roman alphabet...
I bet someone did notice the stickers, but getting them reprinted (assuming it was caught before assembly started) may have meant a delay. Even a minor delay can be a major, major problem since this stuff is scheduled practically down to the hour in the factories; ditto for shipping deadlines. Or if they were already on assembled boards (or worse, inside assembled equipment) the cost to replace the sticker would be astronomical, with exactly zero value to the vendor or their customers. It's cosmetic.
I mean, he acknowledges this:
> I make a lot of typos, so I know how it goes. As someone who is dyslexic, this piece is going to have a few typos when it goes live, so I empathize with this. At the same time, I tend to spell my name correctly on official documents.
In the electronics industry, a typo on a label, and especially in the brand name, is a very strong indicator that the product is counterfeit (AFAIK such mispellings are often deliberate to avoid using a trademark). For instance, fake Apple products are often "Designed by Apole in Colifornia". And while it's true that the products are most likely assembled by non-native English speakers, this sticker appears to include some licensing/serial number information and was probably designed by someone at corporate headquarters for the express purpose of certifying that the chip is genuine.
And the author places the bulk of the responsibility on Dell and AMI for not catching the error, rather than the workers at Foxconn or wherever. Counterfeit parts, supply chain issues, and cut corners are common, so engineers in the US closely inspect a random sample of boards received from the factories to make sure everything was built according to spec, which is where this should have been caught.
In a case like this, it seems like a reasonable course of action. Assuming otherwise is borderline paranoiac.
There's something wrong here. Do we have a second instance?
It's like a choreographed dance with millions of people simultaneously, somebody is bound to trip over their laces.
I have zero faith in their supply chain.
Ordered a few hundred laptops with a office 2016 license key. They key came on a printed card with each laptop. About 33% of the cards were printed in the US. Every one of them worked. The other 2/3rds were printed in China. Half of those were already registered under random email addresses (as each had a randomly generated email that appeared to be at yahoo). It was a massive fight with Dell sales to rectify the situation.
And this is why the people shouting that "the direct cause of the log4j bug is our collective failure to give open source maintainers bags of money" are wrong. Don't get me wrong, you should give people bags of money if you want them to drop everything and go fix a critical bug after it's been discovered.
But our collective ability to discover bugs is abysmal.
The distinction is that even with a magic 100% infallible bug finding service, if the fixes don't get the attention they need, even with this magic service is, the problem's still going to be around. (While this perfect infallible service doesn't actually exist, fuzzers do find bugs in a semi-automated fashion. Still, getting someone to pay attention to the reports is an uphill battle due to false positives.)
IIRC it was found and first exploited by Minecraft trolls.
Lesson learned: the hardest opponent for any IT security team are online-game cheaters and trolls.
Sure, something like 8 years after it was put in place.
1. If one is going to counterfeit the chip, counterfeiting the sticker wouldn't give you any pause - so it's probably just an error, however counterfeiters wouldn't have the review process that would catch typos.
2. Maybe it is a sneaky warranty workaround like a bank calls itself Banq on paperwork - https://en.wikipedia.org/wiki/Banq_(term)
So in the end, there's no real way for the end-user to know what's what and I agree with the author that the typo deserves an answer.
If someone manages to do a good job of replicating the labels, hopefully they did a good job of replicating everything else.
On the other hand, if QA didn't notice the label is misspelled, what other problems did they miss (some of which may be obvious to them if they look, but not obvious to me even)
Shanghai Wicresoft Co.,Ltd
This is a recipe for uselessness. If they can’t force the devs to do things properly, it won’t get done.
Then you can use the Windows OS storage pool commands to figure out the internal mapping.
However, you do have a point -- the disk WWN or GUID or something properly unique ought be available via the Portal or the ARM REST API! That's far easier to automate robustly.
Will give it a proper visit, the last time I tried to downgrade some ssd to hdd resulted in windows boot loop. fun times, had to take a snapshot then fiddle it's registry in another vm so it'd use the last known working registry to get it back up.
I believed it then, and I still believe it now. An "evil" BMC (like the ASPEED AST2600 mentioned in this article) is game over security-wise. BMCs are capable of flashing the BIOS/UEFI, capable of inserting arbitrary disk images as virtual CD/etc drives, capable of arbitrary keyboard/mouse input equivalent to having a hardware keyboard/mouse attached, like a remote evil-maid. If you had to pick one single thing to "pwn" in a server it would be the BMC. There's no way it's just a typo.
I do run DHCP/NTP/etc on that subnet, but those services are running in FreeBSD Jails connected to the host OS with vnet(9) "epair" interfaces given RFC3021-style /31 addresses (and/or the IPv6 equivalent) and firewalled with PF on the host-OS side such that none of them have the ability to explore any other parts of my network if they get popped :)
Whether or not that was the exploit being referred to by Bloomberg is unknown, but suspicious.
Perhaps there was a Chinese whispers (no pun intended) style miscommunication and while the original source meant "software component", it became "hardware component" somewhere along. Or the attack was actually developed as proof-of-concept but never applied in the wild. The attack is feasible as other security researchers have shown:
I myself don’t have much of an opinion on this
That is a useful signal to avoid HPE.
* American Megatrends Inc. (AMI) was founded in 1985 by Subramonian Shankar and Pat Sarma with funds from a previous consulting venture, Access Methods Inc. (also AMI). Access Methods was a company run by Pat Sarma and his partner. After Access Methods successfully launched the AMIBIOS, there were legal issues among the owners of the company, resulting in Sarma buying out his partners. Access Methods still owned the rights to the AMIBIOS. Sarma had already started a company called Quintessential Consultants Inc. (QCI), and later set up an equal partnership with Shankar.
By this time the AMIBIOS had become established and there was a need to keep the initials AMI. The partners renamed QCI as American Megatrends Inc., with the same initials as Access Methods Inc.; the renamed company then purchased AMIBIOS from Access Methods. Shankar became the president and Sarma the executive vice-president of this company. This partnership continued until 2001, when LSI Logic purchased the RAID Division of American Megatrends; American Megatrends then purchased all shares of the company owned by Sarma, making Shankar the majority owner. *
Many chips have been stable for decades without any serious modification with today's version being effectively identical to those of 25 years ago.
It's not defacto secure, but age is not a reliable indicator without more information
No, virtually nothing made 18 years ago is secure. In fact virtually nothing made a day ago is secure, but when something has been sitting around stagnating for 18 years that means the world has had ~6575x as many opportunities to find a vulnerability.
Any random chip, let's take a max232, from 1987. It hasn't changed since 1987. They are widely used.
There's countless simple chips there that have no "security issues" such as fram and pram chips that just store data and that's it.
Many programmers are only accustomed to seeing the modern JavaScript systems with 5GB of dependencies to print hello world and sure, fine; a virtual machine on a browser, on a userland, on a kernel, there's lots of stuff there to go wrong.
That's not what this necessarily is.
Systems can be simple. You use 1970s era zilog-80 with code that hasn't changed in 30 years every day when you do things like wait at a stoplight, use municipal water when you turn on your faucet, put food in your microwave, turn on your light switch ... You probably interact with and certainly depend on more 1970s and 80s era hardware and software that's been faithfully reordered and reproduced for decades then you do any modern system.
Motorola 6800 (not a typo) MOS 6502, and Intel 8080 clones are in every car, jet airplane, bus and train made in the past 30 years or so and they are still being used; oftentimes with no real modifications, for decades.
They are used exactly because there's no surprises and everything is known and accounted for. Newness, the thing you advocate for is the vulnerability.
Society entrusts things like interbank transfers (trillions a day) and international carrier switching on these systems.
This strategy works so well that these systems are utterly invisible.
I was incredibly impressed at the speed that Compaq escalated the issue from a first line support person to having a couple engineers drive out to the credit union to see it with their own eyes to determine a fix (I had no pull or special access, and we're only talking 150 computers). It was convenient that this happened in Houston so the engineers were in close proximity.
This article makes me skeptical that HP would handle it the same way today that Compaq did at the time based on HPE's response to STH. I hate the buying process with both Dell and HP, but they are the brands I buy from when it is a server or workstation, because I love the build quality for both brands on their enterprise servers and high end workstations. I've never owned a consumer device from either brand, but I would not have the same expectations on a consumer device from them.
the problem is the all in one is not all in one. the back of all in one is missing a PC! the all in one is now a 27" monitor with a empty lump where the PC supposed to be. Dell screw up. Dell end up selling these all in one PC monitors to Corporation for cheap.
Persons intending to use it as a monitor would have to go scrounge ebay for something like an HDMI-to-LVDS interface board and wire it up themselves.
sorry if i wasn't clear.
They might be eDP if they're new enough, since LVDS doesn't provide enough bandwidth for modern high-res/high-refresh/high-DPI panels: https://www.embedded.com/edp-a-better-embedded-display-ecosy...
In fact, what's the point of putting a sticker on if nobody is going to look at it?
> John Etulain of Seattle Washington registered those 2 domain names, and it is being served HTTP using STH’s SSL cert.
> Staffer of yours, Pat?
Huh.
And, someone at that OEM ordered a bunch of misspelled stickers. Easy mistake to make, if the latin alphabet is literally foreign to you.
And if you think that sticker is bad? Wait until you see the actual firmware, oh boy... (I had some fun Edgecore LACP bugs take down an pretty sizable network. Things got slightly better once they moved to Linux-based firmware, but never to the point that their kit was, like, entirely reliable...)
The article makes this out to be a major supply chain security issue, and that only makes sense if branding stickers are actually reliable for validation purposes. But that seems... nonsensical? Wouldn't stickers be very easy to forge?
But, I don't work in supply chains. Anyone with better expertise in this area able to chime in?
I will admit I skimmed the article, because it is long and overly-detailed for my level of interest, and because it lacks summary sections.
Almost any corporate or institutional counterfeit or supply chain security program will explicitly teach you: if anything is misspelled or shows other obvious mistakes, hold the part as a suspected counterfeit. It's a pretty good quality indication.
So of course manufacturers do genuinely make spelling mistakes sometimes, but this context makes it a pretty embarrassing and serious thing to do. It's like your bank misspelling their name in an account notification: sure, in some extremely theoretical sense it doesn't mean anything, but in practice they're giving you exactly the signal that everyone tells you to check for to identify phishing, and it raises questions about their processes that they let it slip through.
Turns out it is coming from AMI, but AMI Taiwan.
>AMI Taiwan needed to get license stickers for the local market. Instead of using the “American Megatrends” MegaRAC PM sticker template, it decided to make its own that had the misspelling.
they're generally a competitor of companies like compal, clevo, quanta. All well known in Taiwan if you're in the business of having 3rd parties manufacture your stuff.
it's very interconnected now.
there's a fascinating yearly trade show of taiwanese manufacturers: https://www.computextaipei.com.tw/en/index.html
https://en.wikipedia.org/wiki/Compal_Electronics
US $26 billion revenues. Most people have never heard of it, only its consumer facing brands like Ignitenet.
> Final Words
> All told, there is probably nothing to see here.
Want some Amtech flux made in Colifomia? https://sudonull.com/post/100244-Amtech-fluxes-hoax-on-a-glo... https://ultrakeet.com.au/write-ups/fluxInfo
It would not be the first time someone sends a device for RMA to be told it’s out of warranty because it has been tampered, when in reality it was a manufacturing mistake.