CoreNet: A library for training deep neural networks
github.com
github.com
> CoreNet evolved from CVNets, to encompass a broader range of applications beyond computer vision. Its expansion facilitated the training of foundational models, including LLMs.
We can expect it to have grown from here: https://apple.github.io/ml-cvnets/index.html
It looks like a mid-level implementations of training and inference. You can see in their "default_trainer.py"[1] that the engine uses Tensors from torch but implements its own training method. They implement their own LR scheduler and optimizer; the caller can optionally use Adam from torch.
It's an interesting (maybe very Apple) choice to build from the ground up instead of partnering with existing frameworks to provide first class support in them.
The MLX examples seem to be inference only at this point. It does look like this might be a landing ground for more MLX specific implementations: e.g. https://github.com/apple/corenet/blob/5b50eca42bc97f6146b812...
It will be interesting to see how it tracks over the next year; especially with their recent acquisitions:
Datakalab https://news.ycombinator.com/item?id=40114350
DarwinAI https://news.ycombinator.com/item?id=39709835
1: https://github.com/apple/corenet/blob/main/corenet/engine/de...
I’m not familiar with how any of this works but what does state of the art training look like? Almost no models release their training source code or data sets or pre processing or evaluation code. So is it known what the high level implementation even is?
This is probably a good baseline to start thinking about LLM training at scale.
One example: https://github.com/apple/corenet/tree/main/projects/clip#tra...
It smells of a somewhat panicked attempt to prepare for WWDC to me. Apple has really dropped the ball on AI and now they're trying to catch up.
So because their first party AI products are so non-existent, I think WWDC is a desperate attempt by Apple to get third party developers to build compelling AI products. I say desperate because they're already a year behind the competition in this space.
(I can imagine they'll be trying to get developers to build Vision Pro software too, though I hear sales there have collapsed so again, way too little, too late)
What AI products are present in other ecosystems (eg. Android, Samsung, whatever) and missing from Apple?
Honest question: I find the platform distinction largely meaningless in most cases apart from “what your phone looks like” and “can you side load apps”…
I think it’s important to remember that there are a bunch of actual useful AI-driven features out there that aren’t just GenAI chatbots.
They haven’t put an LLM assistant out there. But they don’t make their own search engine either so I don’t think “online LLM assistant” is something they’ll ever put much effort into unless it’s part of a bigger effort to launch their own AI-based search engine as well.
As for generative AI I don’t think the quality is up to a level that would be reasonable for Apple.
The only area where i would expect Apple to keep up is the kind of Copilot integration Microsoft is working on. And we know Apple is working on on-device AI assistant, and probably have for a long time. It’ll be launched when they can get good quality results on-device. Something nobody else has achieved anyway, so we can’t say that they’re behind anyone yet.
That's the public perception. Maybe due to them not getting in on a quick cash grab off the LLM hype wave?
1) Siri
2) Dearth of published AI research
You could argue that publishing research is not the same thing as doing it. But they don't seem to have done much of it until fairly recently.
I agree that Apple does less research than other big tech companies. But they do it where they think it matters. Their M-series CPUs are more than just integration and polishing. And they have been doing some research in health AI as well, I think.
https://machinelearning.apple.com/research seems to have too many publications to be considered a "dearth" IMO.
But I think part of the problem is that Apple simply hasn't focused on the tasks and the methods and the people that have now turned out to be so impactful.
They have clearly been course correcting for a while now as some of the more recent papers show, and they have done successful research in areas such as health AI.
If that's all it takes to stay ahead of the curve, then Rockchip and Qualcomm are arguably right up there alongside them. Tons of vendors shipped their own AI silicon, and of those vendors, it seems like Nvidia is the only one that shipped anything truly usable. Medium-sized LLMs, Stable Diffusion and probably even stuff like OAI Whisper is faster run on Apple's GPUs than their AI coprocessor.
Be careful not to have NVIDIA-shaped tunnel vision. Performance isn't the whole story. It's very telling that approximately everybody making SoCs for battery powered devices (phones, tablets, laptops) has implemented an AI coprocessor that's separate from the GPU. NVIDIA may take exception, but the industry consensus is that GPUs aren't always the right solution to every AI/ML-related problem.
There's not nothing you can run on that Neural Engine, but it's absolutely being misunderstood relative to the AI applications people are excited for today. Again; if chucking a few TOPS of optimized AI compute onto a mobile chipset is all we needed, then everyone would be running float16 Llama on their smartphone already. Very clearly, something must change.
Curious then, why they keep recruiting search engineers[1]. And why they run a web crawler[2]. And why typing "Taylor Swift" into safari offers a Siri Suggested website before Google.
I guess what people mean by search engine is "show ads alongside web search to as many people as possible"?
1: https://jobs.apple.com/en-us/details/200548043/aiml-senior-s... 2: https://support.apple.com/en-us/HT204683
Apple put a neural engine on-die in the A11 back in 2017:
* https://en.wikipedia.org/wiki/Apple_A11#Neural_Engine
The A-derived M-series chips had them from the beginning in 2020:
* https://en.wikipedia.org/wiki/Apple_M1#Other_features
Seems like they've been doing machine learning for a while now.
> CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data
This is the first I’m hearing of that, and the link seems broken.
curious to know how fast catlip is. the above using openai clip is already fast.
Is this meant for training MLX models in a distributed manner? Or what is its purpose?
> mlx_example/clip: ... an example to convert CoreNet's CLIP model implementation to MLX's CLIP example with some customized modification.
- FP16 Base variant: 60% speedup over PyTorch
- FP16 Huge variant: 12% speedup
Seems like an advantage is extra speedups thanks to specialization for Apple Silicon. This might be the most power-efficient DNN training framework (for small models) out there. But we won't really know until someone benchmarks it.
https://github.com/CarperAI/OpenELM (ELM = Evolution through Large Models)
This repo has a lot of handy utilities but also a bunch of clean implementations of common models, metrics, etc.
In other words, this is more for writing new models rather than inference.
If your product runs on an iPhone or iPad, I’m sure this is great.
If you only ever want to run on 4090s or other server stuff, yeah this probably isn’t that interesting.
Maybe it’s a good design for the tools or something, I have no experience to know. Maybe someone else can build off it.
But it makes sense Apple is releasing tools to make stuff that works better on Apple platforms.
Id be SHOCKED if so. Its been 15 years, but I was there when xserve died. Priorities were iphone > other mobile devices >>> laptops > displays & desktops >>> literally anything else. When xserve died we still needed osx for OD & similar. Teams moved on to 3P rack mount trays of mac minis as a stop gap. Any internal support/preference for server style hardware was a lolwut response. Externally I see no reason to suspect thats changed.
Apple is pushing for open information on LLM training? World is changing...
I guess there's nothing stopping me from moving to Homebrew other than familiarity.
Other taps, like homebrew-ffmpeg, offer a ton of options.
The other issue I experienced with homebrew around that time were related to having different versions of openssl installed because I had some old codebase I had to run (and for performance reasons didn't want to use docker). But that's definitely a edge case.
I still use MacPorts for that reason and it is easy enough to create a local portfile for whatever isn't in Macports.
I find this to be the easy way to manage networked development computers.
It's funny because a multi-trillion dollar company can't be bothered to release a native package manager or an official binary repository for their OS after decades of pleading from developers.
It has become a core OS feature. Historically, you see the set of core OS features expand tremendously. Back in the 80’s drawing lines and circles wasn’t even a core OS feature (not on many home computers, and certainly not on early PCs), bit-mapped fonts were third part add-ons for a while, vector-based fonts were an Adobe add-on (https://en.wikipedia.org/wiki/Adobe_Type_Manager), printer drivers were third party, etc.
I think that’s natural. As lower layers become commodities (try making money selling an OS that only manages memory and processes), OS sellers have to add higher layer stuff to their products to make money on them.
As to Sherlocking, big companies cannot do well there in the eyes of “the angry internet”:
- don’t release feature F: “They don’t even support F out of the box. On the competitor’s product, you get that for free”
- release a minimal implementation: “They have F, but it doesn’t do F1, F2, or F3”
- release a fairly full implementation: “Sherlocking!” and/or nitpicking about their engineering choices.
I love Nix but it probably has too many rough edges for the typical homebrew user.
My inner trademark troll demands a bucket of popcorn.
I think these had their initial releases even before .NET Framework 1.0 (and so even longer before .NET Core) so Apple could probably claim "prior art" or whatever this would be called (IANAL).