Report: Apple building its own servers to prevent snooping | Hacker News Reader

Report: Apple building its own servers to prevent snooping

9to5mac.com

231 pointsdankohn1107 comments

Apple might want to talk to IBM about OpenPOWER. The POWER8 chips and chipsets lack known internal micro controllers that have their own flash unlike Intel's chips that have the Intel management engine. Consequently, there is nothing there to flash with malware. Also, all of the system firmware is open source. That is the motivation behind the Talos workstation board, provided that they get enough interest for an initial run:

https://raptorengineeringinc.com/TALOS/prerelease.php

If Apple is concerned about tampering enroute, they could have the flash chips for the system firmware provided separately by a trusted party and transported like a bank shipment. Then flash and install them at the datacenter. That should thwart adversaries who cannot do their own manufacturing runs of modified versions of the chips, which is just about everyone. I guess the manufacturer/fabrication plant could do a custom compromised chip, but given that the costs involved are prohibitive, I doubt that would happen.

Apple could do the same with the firmware and flash for every other component in their datacenter that has a microprocessor such as the hard drives, the NICs, etcetera. They are large enough that part manufacturers would likely turn over the source code for their firmware in order to secure their business along with anything else that they need/want.

>> "...transported like a bank shipment..."

If you think the NSA/state level adversaries have not penetrated bank shipments, you are dangerously naive. Instead, maybe transport it like the CIA moves the communication hardware for its quiet rooms, or like the Russians move the bugs destined for US embassies.

Apple is interesting because it's one of the few organizations that could plausibly do this.

I said bank shipment because I did not know of a more secure way of transporting a high value target. :/

ICBM-deliveries, guaranteed untampered with if on time!

What is a quiet room?

I think OP is making a reference to TEMPEST:

https://en.m.wikipedia.org/wiki/Tempest_(codename)

I think Apple would probably just use ARM chips; they already have the license and they have lots of experience designing their own.

OpenPOWER is awesome though. I'm hoping Talos will get enough interest; I'm ready to throw money at them if it succeeds. POWER chips are super good, I don't know how IBM has managed to keep up with Intel's R&D but they're definitely competitive in performance.

1. http://0pf.org/j-core.html

I doubt Apple would go so far as designing a low volume server processor for their datacenters. It is probably not cost effective. Apple reusing their iDevice SoCs in microservers could work though.

The A9 has PCI Express lanes for NVMe flash that could be used to add wired networking and storage while the secure boot and code signing would allow them to build a microserver that only runs their code. This arrangement would also be incredibly cost effective for them.

They would want to give their SoCs have properly protected internal data paths and the necessary logic for ECC RAM before using them in micro-servers if those chip features are not already there though. If they are, I doubt that the ECC is being used in their iDevices.

> I doubt Apple would go so far as designing a low volume server processor for their datacenters. It is probably not cost effective.

They have the luxury of having enough cash, and enough profitability, to do things that have negative ROI. The recently concluded lawsuit comes to mind — there was no business case for not settling it at the earliest. They didn't settle at all, and took it to the US Supreme Court.

And if a secure architecture requires a custom-designed backend server farm, they can do it, and even probably make it revenue-neutral, by re-introducing the Xserve product line.

I don't think they will do a full-blown custom design, and the advantage of having their own hardware stack is that they can leverage existing technology, and make a few tweaks. And they must have been running at least designs on this possibility, because they do want to take notebook/desktop processor manufacture in-house in the medium term, and end reliance on Intel.

A secure architecture is possible with IBM POWER. Unless they can save money building their own without hurting iDevice development, there is just no reason to make their own server chips.

> The recently concluded lawsuit comes to mind — there was no business case for not settling it at the earliest. They didn't settle at all, and took it to the US Supreme Court.

I'm inclined to disagree. Apple is a global company, and many of its customers would not look favorably upon the US government being able to force Apple to decrypt their data. I actually think that Apple would have gone as far as to move a good portion of their operations outside the US and create separate legal entities to protect against this had they lost the case.

And while globally people tend to have mixed opinions of Americans, the US government and its three-letter agencies are almost universally despised around the world. Apple being seen as a willing cooperator with the US government (especially on the issue of privacy / espionage) would be a huge blow to their global brand.

The e-books case. You cannot very well settle an FBI request or subpoena.

http://www.bloomberg.com/politics/articles/2016-03-07/apple-...

I don't know the Apple ecosystem well enough but does apple even have a decent virtualization software stack running on Macs? Or would they just run linux on apple machines?

Probably run Linux. Xserve is dead, I don't think Apple uses OS X on servers.

Linux on non-apple machines.

HP machines to be specific.

http://blog.fosketts.net/2011/06/06/datacenter-equipment-app...

They expose a virtualization API that allows this on Mac App Store: http://veertu.com/

Don't know how robust it is compared to e.g. VMWare or VirtualBox, but it seems to be working well enough in my simple tests.

That same Hypervisor.framework is also what powers https://github.com/mist64/xhyve

I was really excited about that project but it doesn't appear to be getting much attention.

iCloud uses Linux according to their job postings.

Doesn't the management engine exist as part of the BIOS on the same NAND? Couldn't Apple simply have Intel give them a custom BIOS that is signed?

Groups more powerful than Apple tried to convince Intel to give them the opportunity to run only their own custom code on those Intel-signed areas of the processors they buy. What seems possible is running code on top of and in addition to the existing system, see CompuTrace.

This isn't "simple" except perhaps for some US (and US only, by virtue of Intel being head-quartered there) agency with huge procurement budgets and persuasive legal instruments, and I'd expect push-back even then.

Why would they need to convince Intel when they could infiltrate them for what they want?

If you infiltrate, with luck you get a half-assed data sheet.

If you coerce them (with those persuasive legal instruments) you get minimum compliance to keep people out of jail.

If you actually get some cooperation established, I'll guess that gives a much better support experience.

If you are willing to throw resources at deciphering things, infiltration could be all you need.

I have been told by the CoreBoot developers that it has its own separate flash inside the chipset. There is definitely something loaded by the BIOS though.

ME firmware is on the same SPI flash part as the host firmware[0]

The BIOS doesn't need to load the ME firmware - that wouldn't be very useful since the ME is what gets the CPU out of reset these days.

However, there are more binary components on a contemporary Intel device (listing the variants used with coreboot. other firmware may look slightly different, but the parts are pretty much the same internally): CPU microcode updates (useful because they can't manage to ship CPUs that work correctly on the first try); FSP (memory initialization code that's totally boring but guarded like a state secret); graphics initialization code (VGA BIOS or GOP driver. likewise boring-but-secret); sometimes some more hardware initialization, but we try to get most of that opened up (with varying success).

And then there are some more signed binary components that aren't directly related to host firmware: TXT and SGX require intel-signed code, and - newest addition, I think - GPU firmware binaries. Because they were so well received with AMD and nVidia hardware.

source: am a coreboot developer.

[0] It's also a NOR part, not NAND, and it may be multiple SPI devices that the firmware hub exposes like a single one, but either detail is beyond nit-picking for the current discussion.

I had talked to tpearson about the ME. He left me with the impression that there is flash memory inside the chipset that is separate from the BIOS chip.

Maybe I had misunderstood as I can find no reference to substantiate that.

This is an interesting talk about Intel's management engine:

https://media.ccc.de/v/32c3-7352-towards_reasonably_trustwor...

"Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration, according to a person familiar with the matter."

I feel like the entire world has gone insane, and every boundary is being pushed to its limits ... and then pushed beyond those limits. Where does this end?

The world was that way long before you realized it.

No, this is post-9/11. It only started with the PATRIOT Act. I vividly remember the Clipper chip debates in Congress. This is that on steroids, with no disclosure what-so-ever. In the communist block, that was different, of course. But in the West, you used to laugh and sneer at Stasi and Soviet "papers, please!".

The only reason this wasn't happening in the days of J Edgar Hoover, was due to technological limitations.

Echelon dates to the late 1960s.

COINTELPRO went on for two decades starting from 1956.

Privacy and rights abuses have been rampant among the three letter agencies since their originations. Today, they can scale the privacy invasion. That's the sole difference from their side. Previously they'd just violate someone's rights and go about their business, good luck proving it or fighting it at a small scale as a specific target of one of these agencies.

https://en.wikipedia.org/wiki/ECHELON

https://en.wikipedia.org/wiki/COINTELPRO

I feel you but... McCarthyism. Watergate

The list of rationalized extensions of the US Government into what becomes in retrospect a blatant abuse of power would be an interesting one.

McCarthy was right, and the criticism based on his conjuring daemons that weren't there, as popular as it has been, is misguided. If anything, as the archives open after all these years, they show the Soviet threat and the extent of infiltration was far greater than anybody even imagined.

Watergate was a quaint bespoke little operation. Much wider unconstrained eavesdropping capability is available to grunts like Snowden on tap. And all the perpetrators were indicted or jailed[0]. A president even was made to resign.

If anything, the magnitude of the threat the Un-American Activities Committee dealt with, and the constrains Nixon faced (both technical during, and legal after the fact) provide a contrast to the post-PATRIOT-Act reality of threats and constrains, all imaginary at best.

[0] http://watergate.info/analysis/casualties-and-convictions

>No, this is post-9/11. It only started with the PATRIOT Act.

you must be kidding. https://en.wikipedia.org/wiki/Crypto_AG#Compromised_machines

The Crypto AG case is a stronger argument for his point than yours: the targets were foreign, which is to say legal for the NSA to spy on, and it was done with the involvement of the company.

In the current era, the targets are often domestic and they're willing to compromise U.S. companies on a large scale to do it.

we were never this powerful, though

I'm sorry to say it, but I read this and think "plausible deniability for future-Apple when they open the (back) doors".

Wow, so this is very interesting given, pretty much, everything that has been going on.

Will apple actually be the bastion of freedom (both in market and privacy) that the US supposedly stands for??

Google makes its own machines, as does facebook (actually more interested in FB's fiber switches/routers, but thats beside the point)....

But Apple has made "servers" for years... I guess they didnt consume them in their own DCs?? So I basically take it that they are effectively joining their take on Open Compute (mobos that can be mounted in controlled environments, where controlled now also means they can ID if anything was modded/changed in shipping?)

EDIT: I would really like to know how long "long" is from "apple long suspected"....

I was informed of NSA back-doors in Cisco gear in 1997 - WTF is Cisco's stance on any of this -- I haven't heard anything from them at all (or I missed anything they said)

Xserves were replaced a few years ago. The current "replacement" is the Mac Pro. I highly doubt that all of iCloud runs on racks filled with the little cylinders.

https://en.wikipedia.org/wiki/Xserve http://readwrite.com/2014/08/26/apple-icloud-amazon-web-serv...

If they were really paranoid they could build a new Xserve powered by an offshoot of their A series ARM chips. To help defray costs, they could actually sell the thing.

That'd be an awesome extension of the resources they invested, but it runs the risk of designs serving too many masters (like the F35 Joint Strike Fighter trying to add VTOL, making the non-VTOL thing way too heavy), and the A-series probably doesn't have enough cores for the unit price to work well for these services. What Intel is doing with Xeon Phi and the 18-core Xeons is the way things are going I think.

I think they could handle a separate line of chips. Plus, a lot of folks are going the ARM route in server-land.

If their A-series chips have protected internal data paths and ECC, they could build microservers around them. The A9 already has PCI-E lanes that could be used for networking and storage.

If for every F35 they made three of each, with different capabilities, and ceremonially dumped the two without the desired capabilities out in the ocean, it would still be cheaper overall. Which is hopefully what Apple would do. It's not like they lack talent or organizational ability to employ a few more chip designers. Or a few dozen.

The A9 does have respectable performance and great thermal characteristics. Rather than compete head to head with Intel, who are very very good a general purpose CPUs, it'd be interesting to see a cloud-optimized “server” which is actually a 1U cluster of storage processors – tweaked A9, as much flash as they can fit, etc. – so you'd have capacity to do things like encrypt everything at rest and in transit and apply distributed database techniques to everything because you'd have a relatively huge amount of processing capacity very close to each block of storage and they control the entire software stack.

Basically like what Oracle does with Exadata without fitting a yacht into the purchase price.

Maybe not Apple data centers, but it is possible to have a data center of Mac pros: http://photos.imgix.com/racking-mac-pros

Sounds like NSA's Tailored Access Operations[0], and other equivalent agencies. When you're a big enough target, state actors bring the full force of the state's espionage tools to bear.

[0] http://www.pbs.org/wgbh/frontline/article/how-the-nsas-secre...

Exactly!

How many machines have you purchased in your career? I've purchased THOUSANDS - which of these have been intercepted and compromised?

I have no clue.

I don't know about spy agency compromise, but I have definately caught suppliers "accidentally" sticking counterfeit memory into devices.

>> "...which of these have been...compromised...?

~All of them, though likely without bothering to intercept your specific hardware, since you were relatively unimportant. Instead, the breaches were built into commodity hardware far upstream of you, for fully general attacks.

It occurs to me if the company is that big, or that big of a target, it can manage to collect its servers directly from the manufacturer/supplier rather than rely on a third unrelated party for transportation.

So what? How do you know Dell/HP/Whoever has not been forced by NSL letters to implement backdoors.

How do we know if whoever doesn't think it's their patriotic duty to do so?

Historically, that's been the bigger issue. Or at least, that's what I get from Bamford's books.

> So what? How do you know Dell/HP/Whoever has not been forced by NSL letters to implement backdoors.

That is not necessary. Just get the shipping company to redirect the hardware to a facility that flashes the Intel ME with signed malware, reseals it and reships it without the recipient seeing anything different. That way fewer people need to know about it, which decreases the chance of detection:

http://www.pcworld.com/article/2083300/report-nsa-intercepts... https://libreboot.org/faq/#intelme

Not even Intel needs to know if they are compromised successfully, just like Google did not need to know about those fiber taps between their datacenters. That means countries like China that have things being shipped to possible targets can do this too if their hackers gain the necessary secrets.

> That is not necessary. Just get the shipping company to redirect the hardware to a facility that flashes the Intel ME with signed malware, reseals it and reships it without the recipient seeing anything different. That way fewer people need to know about it, which decreases the chance of detection:

@iam-TJ 's point was in reference to the suggestion that Apple could pick the servers up directly from the manufacturer to avoid the interception while the servers are in the shipping company's hands.

In that case, the attack would not need much cooperation on the part of the OEM/ODM. The Intel chipsets could be flashed enroute to their manufacturing plant. They might not want the malware to go many places other than the intended target, which is why they would likely want some cooperation to ensure that they only make it to the intended target, which is why I say not much cooperation rather than no cooperation. Anyone from Apple inspecting the factory might not realize it.

I probably should elaborate that this attack assumes that the chipsets have internal flash. So far, I have only heard of this from a coreboot developer and upon doing Google searches, I cannot find any sources for that information. :/

it always helps to be paranoid!

> it can manage to collect its servers directly from the manufacturer/supplier rather than rely on a third unrelated party for transportation.

Until it hits customs.

Or the Chinese equivalent. Or both.

Intercepted during shipping?

How long would an operation like that take? And can't a company with Apple's clout ask the supplier for a spec sheet that lists all the chips and a diagram showing the location of said chips?

Of course they can, which is why perhaps apple "long suspected"... maybe shipping times were fucked and apple noticed - but my question is WHEN did apple notice???

Long ago, like anyone else who was paying attention. See among (many) others the PC mag links given elsewhere in this thread.

No later than Snowden.

You sound incredulous. It's a thing. http://www.pcworld.com/article/2083300/report-nsa-intercepts...

The Intel chipsets even have an embedded device with complete access to everything, its own firmware and its own integrated flash where malware can be flashed by anyone with Intel's internal documentation and the private RSA key corresponding to the public key that is etched into the hardware:

https://libreboot.org/faq/#intelme

A bit.

So are they going to run these in another country? Because Congress is pretty idiotic when it comes to technology and they can pass laws to force Apple (and others) to allow snooping.

Good question. For now, Apple seems to have the legislative branch on its side. Some Congresspeople such as Darrell Issa and Lindsay Graham have actually been educated on the technological issues.

This doesn't make much sense. Even if apple made their own, unless they will be shipping with their own fleet of delivery trucks with armed guards not on the take, how would this differ from shipping dell machines, or if they do go this route, why not use their own guarded shipping for Dells and netapp stuff?

Not only that, but they can confirm the firmware after receiving the equipment, unless they suspect the CPUs machine code, but then who will they get their CPUs from?

And besides, it's not like they'll be tearing out all the netapp, dells and Cisco --everyone even amazon have them all, from what I hear.

It's a lot easier to diff if you invented the originals.

They could do stuff like this for their own servers:

https://www.kickstarter.com/projects/designshift/orwl-the-fi...

But they may want to build their own chips, too, because they probably shouldn't trust Intel (they do some of that already, but they may need more powerful chips for their datacenters).

I have a buddy who was super high up in a telecom back in the late 90s/early 00s and he mentioned that all the big telecoms basically voluntarily were creating traffic backdoors for the government.

When he mentioned it, he didn't even blink. It was eerie.

Echelon.

It is real.

It has been around forever.

Is Apple saying anything about what they found? This is a big deal. Who put something in their servers? NSA? The PLA? Samsung? Did they have any strange chips analyzed? There are companies that can take an IC apart and see what's inside.[1][2] It's not cheap, but Apple could afford it.

[1] http://www.siliconinvestigations.com/ [2] http://www.istgroup.com/english/3_service/03_01_detail.php?M...

The PLA?

People's Liberation Army. China's military.

Why not just contribute a few designs or some open source software and join OCP?

http://opencompute.org/

Last time I checked, Apple was not in the business of making data center equipment, so it's not like they would give up IP that is central to their business model.

When will we get secure PC architecture for the consumer market?

When it's in someone's financial interest to build it. Which is to say, never.

Never.

When consumers start paying for things like Talos:

https://raptorengineeringinc.com/TALOS/prerelease.php

https://www.crowdsupply.com/sutajio-kosagi/novena as close as we have, at least in terms of verifiabily uncompromised.

No company is static, what will Apple look like in 10 years? What would happen with my data they recorded today should company policy take a shift?

I'm quite uneasy regardless of who holds my data.

I don't know how Apple's servers security fares now, but I took a quick look 4 years ago and it was notoriously bad. They were responsive and solved the exploitable bugs soon after I notified them. No bounties though.

If their software and network security is similar now.. then they should spend resources there rather than care too much about modified hardware by a governmental agency.

Wouldn't it be completely separate teams ? I can't imagine the hardware team's efforts taking away engineers from teams handling software security.

An issue at very large companies like Apple is that while they can be superb in one area or in one core product does not translate into being adept in other areas under different teams with different management styles, demands budgets, etc.

So unless Apple are going to put their best and poach other industry bests, their result will likely not deliver on premise or promise.

Judging by their rhetoric and their recent win (can it be called that?), there is certainly widespread interest in them locking themselves down. They would look a little foolish if not.

Certainly they are different teams.

My comment just meant that I needed 1 hour to find a way to get a shell in a couple of their servers. That is more worrisome to me than the NSA snooping around. And I'm not a great pentester, I probably wouldn't find a bug in Google even if I spent a couple weeks.

They are HP built Intel systems running Linux.

In theory, does a backdoored firmware run slower? If yes, then can you detect a backdoor by building one yourself and benchmarking?

Or are the margins of error on repeated benchmarks larger than any performance hit due to a backdoor, or can something be backdoored without any performance hit?

What about power consumption?

Practically, backdoored firmware doesn't necessarily run services and communicates to the mothership all the time, nor does it communicate using conventional protocols. It creates invisible hole in the system beyond OS's reach to detect and repair it. Furthermore, the hardware (or chip based) backdoor is way more sophisticated, typically have been in use in the servers of foreign governments of interest. This chips can be used to take full control of a server, and the whole network when places strategically.

In another note, hard RTOS are not used in servers, so performance cannot be compared with benchmarking tools.

>> In another note, hard RTOS are not used in servers, so performance cannot be compared with benchmarking tools.

It may seem surprising, but performance is easily determined by side channel attacks.

https://www.google.com/search?q=blackhat+side+channel+web+ti...

It runs at hardware speeds, and only some of the time, so not really. You can perform timing attacks, and see if you're running under a modified hypervisor, for example. But if you have a sniffer chip on the PCI bus or Northbridge, you won't be able to see it. Power consumption has been done as well, but you really need to know what it is you're looking for, and if it is just dormant for most of the time, how are you going to test for that?

If this is a trend I guess HP wishes it would never have so readily worked with the NSA bugging servers.

“[23:06] <ATPtipster> we've gotten Cisco equipment, Supermicro servers, and Seagate hard drives that have been tampered with [by the NSA]”

This is one of the faultier parts of having to answer to share holders. Apple is finding it more difficult to justify having their own entire supply chain rather than relying on other companies (i.e. Intel, IBM, Samsung, etc...)

If they didn't make that horrible mistake of their recent multi-billion dollar purchases, they'd have extra billion atleast towards a fab plant.

I'm laughing at Apple right now. Idiots. You Apple board... are stupid.

What? Why would they care about a billion dollars? They're sitting on around $200 billion cash.

They are seen as throwing away cash, and throwing away 1% of cash holdings is very bad.

if the NSA is doing this in the US against a US company isn't this illegal?

"Legality", in this context, went out the window long ago. Power relentlessly weaponises "legality". Power uses it to attack its enemies and to shield the unethical activities it wishes to protect.

So Apple is supposed to safeguard our data now? If you really want privacy don't expect big corp to keep it for you.

Url changed from http://daringfireball.net/linked/2016/03/23/apple-data-cente..., which points to this.

Like the FBI vs Apple case, this looks like marketing hype to me. Apple is increasingly selling on "security" they can't deliver.

(I mean, I'm sure they'll try to action something, but we all know security is a compromise not a guarantee).

Reply on news.ycombinator.com