Benchmarking Pocket-Scale Inference | Hacker News Reader

Benchmarking Pocket-Scale Inference

artificialanalysis.ai

84 pointssys4259018 comments

Apple has been stingy with RAM in consumer hardware. RAM prices will continue escalating, some analysts say into 2030, and this will make it more difficult to build next generation phones with sufficient memory for meaningful ML workloads.

I hope there's some kind of inversion in the current chip economics, because I love distributed/democratized/private compute, but currently cloud based LLM inference seems to be much more viable. I don't see local llms meaningfully viable for the general usecase in the near future.

I'm not sure about that. I can run Qwen 3.8 27B at acceptable speed on an M1 MacBook Pro from 5 years ago. Thats Opus 4.6 Quality, on 5 year old consumer hardware. I think the trend is the opposite - smaller models that can run on hardware people already have are getting better and better.

How much RAM does your system have?

There’s something I’d don’t understand about this. Apple’s RAM is on their own die. Is the price of RAM an indicator for the cost of die space in general?

No - DRAM is made on a different process than compute. Apple uses dram made by the usual dram manufacturers.

The dram shortage is a combination of high demand + shifting manufacturing to HBM, which consumes more wafer capacity than DDR5

SoC RAM is on a separate die in the same package. With modern chiplet architectures even the CPU itself isn't on a single die.

Note that the Model Intelligence score shown here is not the same as the Artificial Analysis Intelligence Index score they use for full size models, the top models on this Pocket-Scale chart have a score of 63, but that translates to just an 11 on the normal scale.

Can someone give me a breakdown on how good these are vs say GPT-4 or GPT-4o? Curious if the frontier from a few years ago now runs on a phone.

Qwen 3.5 9B scores 2-3x higher than 4o (depending on the 4o version), on the benchmarks.

Whether it's actually better for the kind of things people actually use it for... the benchmarks don't really tell you that. (In my experience, no.)

I often have funny experiences where models do great on benchmarks and are awful, or do poorly and are great for my use cases.

And different people use them in different ways, which probably explains why some people think one models is great and others think it sucks.

In my experience even small local models are now surprisingly good at programming and using a computer (bash), i.e. completing agentic tasks, but fall apart quickly in conversation (especially knowledge and understanding).

I’ve been running this level of models through a personal benchmark on a RX6750XT.

I am hoping to get to the point of being able to run a personal assistant agent with tools at its disposal.

The results are largely the same, with some outliers.

Results can be seen here: https://0xff.nu/hexbench/

Tangent question: what about NPU performance? I always read about CPU vs GPU, but nobody talks about NPUs; I don't even know if llama supports them.

Why do phones and laptops get equipped with NPUs if there's no use for them?

There are uses for them but as a sibling comment mentions they're currently extra work to target per vendor. If you squint a bit they're all basically stripped down GPU compute elements with everything except the matrix multiplication hardware removed, a few MB of cache, no RAM of their own, and a nonstandard API.

OpenCL and Vulkan are supposed to gain support eventually but given their extremely constrained instruction sets and the wide variance in capabilities between different hardware models I don't imagine they're the sort of thing you'd want to target manually.

NPUs, at least for apple, are built for relatively small scale efficient background processing. Theyre taking care of extracting structured metadata from freetext, recognizing people & text in photos, indexing file contents, etc. they do this really well with crazy low power draw. Theyre generally not even a target for current inference frameworks ala ollama, pytorch, etc.

I did see one prototype someone built using private ios APIs a few months back. They were able to run a reasonable SLM about an order of magnitude slower than GPU but with a tiny power draw.

I dont think llama does. There has been some work. They were originally designed for image processing (remove background from calls) and many exist because Microsoft mandated them for "AI laptops". Often they don't have much memory bandwidth, and are limited in compute and they are all incompatible with each other.

Edge and local inference completely strip away the infinite scaling safety net of cloud backends. When compute and memory are heavily constrained, runtime governance cannot afford any heavy control-plane overhead.

The benchmark is nice but it's very much biased towards flagships i.e. not very useful in practice if you are trying to ship production mobile apps. Apple historically is extremely stingy when it comes to RAM and they never bothered giving iPads and iPhones more ram until fairly recently (most likely because of ML demands). Your covid era 10th Gen iPads only have 4GB of RAM for the base models.

The Android ecosystem is much more liberal when it comes to RAM because their Dalvik VM JIT (their Java Android Runtime, partially AOT compiled and partially JITed) design is not particularly memory efficient. But the main issue with Android is that their mid/low end (think the Samsung Galaxy A series, the OnePlus Nords, the Motorola Gs etc.) are very inefficient when it comes to single core compute performance compared to iPhones, and it gets worse once you factor in power efficiency. The high end Android flagships running the Snapdragons elites (especially post Oryon acquisition) have no problems matching if not exceeding Apple hardware performance in terms of raw power but they are much more power hungry.

At the end of the day, the current gen of "pocket scale" LLMs are still far from being able to be deployed at scale on mobile. Maybe in another year or two once RAM prices have fallen enough and mobile manufacturers build a lot more matmul and memory circuits into their SoCs instead of a tiny mostly useless "NPU/tensor processor" that doesn't have enough RAM to run anything useful. Current gen production mobile ML still heavily rely on traditional transfer learning like Apple's Core.ML (tiny pre trained models, with perhaps the final layer inference head swapped out).

If you want a good example of production mobile ML, take a look at FUTO keyboard's transformer model for their keystroke prediction/autocorrect which operates in near real-time.

https://github.com/futo-org/android-keyboard/blob/7ebbf5c8b5...

I would say right now 100M-300M params is about the maximum you can do if you want any reasonable level of performance on your "average" non flagship mobile. I would keep to sub 100M for power efficiency sake.

s-tier info / thanks for posting!

I need this version for embedded devices.

Reply on news.ycombinator.com