Apple to Power AI Features with M2 Ultra Servers | Hacker News Reader

Apple to Power AI Features with M2 Ultra Servers

macrumors.com

38 pointsmikhael60 comments

Waiting for Apple to make MK Ultra servers

underrated comment :) bold of you to assume they don't have them already only 6 countries have a higher GDP than apple lol

if anyone from Apple is in these threads please sign me up for the beta although i think the first round pretty much knocked me & my family out

You are comparing market cap to GDP. You should compare revenue to GDP. Still not a perfect comparison, but much much better. Apple's revenue: 380B. Iran sits at position 40 with 380B GDP in 2022.

ahhh makes sense thank you for catching that Iran keeps popping up all over HN past few days haha

still kind of crazy to think Iran and it's entire oil reserves that power idk how many households produce the same amount of revenue as 175k people

That oil is sold unprocessed and won't be powering anything unless it goes through a fairly expensive processing and logistics chain. Iran is also extremely inefficient and unproductive - it's a state after all.

Similarly, African raw materials form the core of many computers and yet Africa is still poor - because the raw material itself isn't that useful.

Eat your own dog food. Whatever they learn / develop there could pay dividend elsewhere. Seems logical. Got some questions or ideas, you know a guy who knows those chips.

If Apple is going to build servers, it sure would be nice if they went the extra half-step to also sell them so people could stop with the weird rack-mounted mac minis. I guess https://www.apple.com/shop/buy-mac/mac-pro/rack is a step towards that, but "real" server hardware would be better.

The fascinating decision someone at Apple has to make is how to allocate the production of chips. Every M2 Ultra chip that goes into a server is a chip that can't go into a consumer device, and vice versa. They're the only source for M2 Ultra chips, so they can't go somewhere else to get more, so they have to decide which division gets what.

Running what operating system?

Maybe baremetal BSD 4.4? That's the codebase for the Unix personality layer of OSX (which is Mach as ukernel + a Unix layer on top). Or FreeBSD like Netflix uses for their first-party servers?

Out of curiosity why cant it run macOS?

Speaking in general, one reason not to use macOS for servers is that the macOS kernel does not provide all of the necessary APIs for containerization.

A container is a combination of a restricted filesystem (e.g. chroot), separate namespaces (e.g. pids, network, ipc), and resource limits (e.g. cgroups for max RAM and CPU usage). It is a big undertaking to modify a kernel to provide these capabilities.

A few operating systems have these APIs (or something similar). The ones that I know about off the top of my head are: Solaris Zones, FreeBSD Jails, Linux containers, and Windows Server containers in Process Isolation mode.

The macOS kernel simply doesn't provide these APIs and I doubt that Apple is really interested in putting in the substantial effort to develop them.

They don't need Linux style containers, they control the OS. The Linux container situation is just a hack around the general API instability of the Linux userspace, a problem macOS doesn't have.

For inferencing workloads they also don't need to control max RAM or CPU usage as they can just dedicate the entire machine to handling requests.

And for sandboxing, Apple's sandboxing infrastructure is actually the best of any OS (but mostly private unfortunately).

If you don't mind me asking do you know if they intentionally removed support for containers? The closest cousin FreeBSD seems a lot more friendly in terms of support

Plus a lot of devs use mac is it not a large enough addressable market for apple to care about the servers

I'm honestly not really sure how it went down. It's possible that XNU (the macOS kernel) was simply developed before FreeBSD Jails were developed, and they never put in the engineering effort to port over the feature. I don't think there's a fundamental reason why it wouldn't work.

For devs, I think that most devs are okay with the current practice of running a virtualized Linux (or other) guest via Docker and deploying to Linux (or other) servers. macOS does support virtualization. The difference between virtualization and containerization is that a virtualized guest uses its own kernel whereas a containerized guest shares the host kernel.

I'll also point you to two comments in this old HN thread which seem to have good information about Apple's use of server operating systems:

The possibility of Jails in XNU: https://news.ycombinator.com/item?id=31384534

Apple's use of Linux in the Services division: https://news.ycombinator.com/item?id=31383746

IIRC Apple has had a history of using NetBSD for server/infra systems, especially those demanding high network throughput and stability. Examples:

• their ObjC WebObjects-based web backends (such as the iTunes Music Store backend — which might still be ObjC-based to this day? Anyone know?) ran on NetBSD servers

• the firmware in Airport routers was NetBSD

I wouldn't be surprised if there’s an Apple Silicon build of NetBSD, created for internal use at Apple. (Though I also wouldn't be surprised, if they've tried to "converge efforts" since then, and have somehow stuck a Darwin userland on top of a NetBSD kernel.)

I would be genuinely surprised, though, if the infra folk at Apple trust the XNU kernel's network stack enough at this point to want to use plain XNU/Darwin for their servers!

---

As an aside, I've always been fascinated by Apple's approach to backend/server technologies, compared to other bigcorps. Despite not being a "UNIX shop" per se, Apple seems to have acquired some engineers somewhere along the way who have a deep understanding of the "UNIX way" to build stuff, and who carry what might even be called "legacy ideas" about system architecture.

You can get a peek into this approach, by prying into the insides of the old Server.app. Sort of like how XCode.app has a BSD buildroot inside it, with the base OS shipping with stub binaries that call inside it when it's available; Server.app does the same, but with an extended BSD userland containing regular old BSD daemons.

Besides being a wrapper for this BSD-userland payload, Server.app itself was just a configuration wizard and state-converger for a set of plain-old config files, living in a virtual /etc dir (regenerated from canonial plist files), that enabled and drove these venerable daemons to do their thing. The "Websites" feature was just Apache, and the "Wiki", "Calendar/Contacts", etc features were just symlink-managed PHP-FCGI plugins for that Apache instance. The DNS was just BIND. The VPN feature was mostly racoond[1]. It's exactly what you'd expect from e.g. cPanel on Linux, with no sense of anything "Apple-y" going on.

[1] https://www.netbsd.org/docs/network/ipsec/rasvpn.html

This UNIX-y approach also seemingly extends to Apple's use of networking protocols. Even while Apple was giving users a proprietary SMB-alike protocol (AFP) to use for local-network file sharing, their internal approach was different: macOS, to this day, ships with a pre-configured "auto-home" mobile user profile feature, that expects to talk not to an AFP server, or even an SMB server, but rather to an NFS(!!) server. (You could even set this up for yourself, in the Server.app days, given sufficient understanding of Apple's OpenDirectory + how it integrates with BSD Kerberos.)

Those AirPort routers were incredibly solid and unfussy. I wish Apple would get back into that product category, like they did with monitors.

I wish they'd get into widescreen monitors.

I feel like with any tech, the folks who really understand how things work + the history / can get the most out of technology. The folks who understand the history, the whys, and accordingly they know where the bottlenecks are, where efficiencies are and etc.

Accordingly they know where in the "old stuff" are really good ideas.

Asahi Linux, obviously

LOL, you can get 5x - 10x computational power (including AI if needed) for the same price from other vendors.

Unless they overshot production and have stock of unsold M2s they're willing to sell at 90% discount.

Apple doesn’t pay consumer prices for their own chips.

Sure but they also don't have a profit margin of like 800%. And there's power consumption to think about.

I expect it's about control and Apple's philosophy that they have to do everything themselves.

They don't pay retail for their own vertically integrated products, they likely would have inventory surplus as new retail products move to other chips, and they mature production pipelines primed to efficiently squeeze out additional volume for a new application like this.

Your comment doesn't make much sense to me.

I wondered about this too. One thing that did make sense to me was that these are for supporting features as-is on older devices by offloading on-device workloads to the cloud.

If you're releasing a bunch of new ML-powered features that are setup to run on-device, you have to have a cutoff for what generation of device is required. Apple has indicated with recent OpenELM model releases that they're targeting different levels of device capability. But you can only go so far.

For older devices, you can offload those tasks to the cloud, but then you have to maintain 2 different software stacks and there are potential compatibility issues.

If you have a bunch of M2Us in the cloud though, you have the exact same hardware that you're developing on-device for. If you package up the core software components as frameworks, you can run on device or in the cloud, and it MAYBE simplifies keeping the results and output in sync between the two.

Otherwise, it makes no sense from a pure performance perspective vs. just deploying racks of H100s...

Apple is not becoming a third party cloud service provider. This is hardware for their own internal uses

I am still amazed that I can run a local instance of LLama3 on my 3-year old M1 MacBook Air. It is a bit slower than ChatGPT online, but nice to have a GPT-3.5 level LLM running locally!

I kinda checked out of LLM news as the hype cycle had got annoying and I had no idea this is possible. I have an M1 Pro and am excited to try it.

ECC

I wonder if that means Apple will bring ECC to Apple Silicon.

Summarisation & SDXL run just fine on M1 & above Macs. Probably the video editing features like Google/Pixel might need cloud compute.

Wonder what kind of tech they’ll be using on the server side to develop all those services.

Some likely candidates are the existing open source tools their ML team has released, most of which either use PyTorch or are usable via PyTorch:

- CoreNet, a training toolchain developed by their ML teams: https://github.com/apple/corenet. It's built on PyTorch, but uses MLX

- MLX, Apples internal array processing framework which is C/C++/Swift (https://pypi.org/project/mlx/).

- CoreML tools, their python package for accessing the lower-level CoreML APIs (https://github.com/apple/coremltools)

Tensorflow and PyTorch have been ported to use the accelerators on the M series of chips and they could just be deploying Mac Pros that are rack mounted. So Python.

It's been a few years but when I was there, Apple had an almost frothingly EXTREME policy against using ANYTHING from Google, most especially Tensorflow. Things may have changed in the last few years but I would not be surprised if that were still the case.

https://developer.apple.com/metal/tensorflow-plugin/

Will they be using the ports that have been painstakingly made by open-source enthusiasts without having access to official Apple documentation?

I’m pretty sure you’re thinking about the work on the open source GPU driver. AFAIK Apple contributed the MPS backend for pytorch and tensorflow-metal PluggableDevice.

They often work with the community in helping the ports for a lot of these projects, it's just not publicized as much (and often their engineers aren't allowed to talk about it).

E.g. when Apple Silicon was released there were people in various FOSS projects who worked with Apple assistance for porting. Here's an example that comes to mind:

"In collaboration with the Metal engineering team at Apple, we are excited to announce support for GPU-accelerated PyTorch training on Mac."

https://pytorch.org/blog/introducing-accelerated-pytorch-tra...

Swift perhaps

iAI

Pronounced for the poor team working on it "Aye yai yai".

Coincidentally I just installed Windows 7 on an old ThinkPad from 2009 today. Nice to see some jokes from that era too!

Probably as a stop gap solution until they make a true AI-centric chip.

Get's all the moving parts into place for when they have a competing solution to Nvidia / Google Tensor.

The real problem is the amount of Ram they include in the processors, Apple's insistence on keeping 8gb as the base is going to hold them back here.

The amount of ram on their lower end configurations really has nothing to do with this.

I'm not sure that a MacBook Pro at £1,899.00 should be a 'lower end configuration', if we are talking bout running AI locally and Apple want's to get in this game, they can't expect everyone to buy an extremely high end machine because their Macbook 'PRO' only has 8gb of RAM.

Why would a low end config that they don't have to choose limit them?

What does the base 8GB RAM option have to do with how Apple will spec their servers?

The M2 Ultra supports up to 196GB of RAM in the products Apple offers it in.

I was responding to a comment that directly said this was a stop gap before they made a real AI centric chip, in that context, the chips will need more ram, until they do, they will be offloading a lot of stuff to the server.

The Ultra may support 192gb but that also comes at an insane cost, if AI is going to become mainstream and Apple want to do this on device, the base models are going to need more Ram, processing power on these systems is not the bottleneck, the RAM is.

The whole context of this article and post is Apple using their own chips on the server for AI tasks.

The “stopgap” is using the existing M2 Ultra for this, versus a chip specifically designed for AI server duty.

I don’t think the person you responded to is talking about on-device processing. Apple definitely wants to do as much of this on-device as they can of course, that’s just kind of tangential to this discussion on which server chips they are using.

M2 Ultra machines don’t come in an 8GB variant

It's also £4,199.00 for the M2 Ultra studio.

I'm talking about more normal machines, all have 8gb including 2 out of the 3 MacBook Pro models, meaning you have to drop two grand for a MacBook Pro with 16gb memory and even that is low for AI.

But those aren’t the subject of the thread or this article

Original comment I was responding too 'Probably as a stop gap solution until they make a true AI-centric chip.'

They are using M2 Ultra on the servers because that has enough power / memory, the issue is there baseline machines don't and only have 8gb, hence needing to offload to a server and it's going to be a stop gap until they put more ram in the standard machines.

The base on the M2 Ultra was 64GB.

Running on Snow Leopard Server?

those were the days..

I feel Snow Leopard was peak OS X. Clean, lean, no bullshit. It seemed magical how well everything worked.

To be fair though, current versions are probably 'better' overall, I'm just annoyed with all the new stuff and crappy apps haha

It was great, but when I use it there’s little bits of accumulated polish here and there that I feel the absence of (e.g. inertial scrolling, which as it turns out is something I use heavily).

Would love it if someone were to build a desktop that uses 10.4/10.5/10.6 as its design basis and then adds that missing polish.

I concur. Apple really needs to get back to their roots and come out with a bugfixed macOS release without all the extra gunk. Perhaps even make it an LTS release as a last hurrah for Intel Macs.

Now they will realize 8gb won’t cut it

Reply on news.ycombinator.com