Raspberry Pi's New AI Hat Adds 8GB of RAM for Local LLMs | Hacker News Reader

Raspberry Pi's New AI Hat Adds 8GB of RAM for Local LLMs

jeffgeerling.com

256 pointsingve209 comments

Within minutes of each other I can read a bunch of comments about how an 8GB AI hat for RPi would be grand for LLMs, while another contingent points out that my M2 MAX 96GB MacBook is completely useless for LLMs. All I can say is at least the latter is also a great laptop.

For server use, the Pi is cheaper, better-supported and lower power. I really cannot recommend that anyone spend $2,500+ to run inference on a laptop that is slower than $300 GPUs.

I think Raspberry lost the magic of the older Pis, they lost that sense of purpose. They basically created a niche with the first Pis, now they're just jumping into segments that others created and are already filled to the brim with perhaps even more qualified competition.

Are they seeing a worthwhile niche for the tinkerers (or businesses?) who want to run local LLMs with middling performance but still need full set of GPIOs in a small package? Maybe. But maybe this is just Raspberry jumping on the bandwagon.

I don't blame them for looking to expand into new segments, the business needs to survive. But these efforts just look a bit aimless to me. I "blame" them for not having another "Raspberry Pi moment".

P.S. I can maybe see Frigate and similar solutions driving the adoption for these, like they boosted Coral TPU sales. Not sure if that's enough of a push to make it successful. The hat just doesn't have any of the unique value proposition that kickstarted the Raspberry wave.

That niche was long taken over by cheap Chinese SBCs, so they have to innovate somehow. Their only advantage that remains is the community.

The thing with the Chinese SBCs is that (like every other player in the embedded world) they don't give a flying fuck about upstreaming their code to the Linux kernel.

Of course, Raspberry Pi just like everyone else has their custom patches, but at least to my knowledge you can use a straight Linux kernel and still have a running system.

Or just updates in general. Need documentation? Good luck.

Yep. RPi foundation lost the plot a long time ago. The original RPi was in a league of its own when it launched since nothing like it existed and it was cheap.

But now if I want some low power linux PC replacement with display output, for the price of the latest RPi 5, I can buy on the used market a ~2018 laptop with a 15W quad core CPU, 8GB RAM, 256 NVME and 1080p IPS display, that's orders of magnitude more capable. And if I want a battery powered embedded ARM device for GPIO over WIFI, I can get an ESP32 clone, that's orders of magnitude cheaper.

Now RPi at sticker price is only good for commercial users since it's still cheaper than the dedicated industrial embedded boards, which I think is the new market the RPI company caters to. I haven't seen any embedded product company that hasn't incorporate RPis in its products they ship, or at least in their lab/dev/testing stage, so if you can sell your entire production stock to industrial users who will pay top dollar, why bother making less money selling to consumers, just thank them for all the fish. Jensen Huang would approve.

Not just laptops but the used enterprise micro PCs from Dell, HP, and Lenovo. All the same small form factor with very low TDP You can have up to 32 or 64 GB RAMs depending on the CPU, dual or even triple disks if you want a NAS etc.

yeah, depends on what the used market looks like where you live. Here I see way more laptops for sale for cheap than those enterprise thin clients.

And the thin clients when they are for sale tend to have their SSDs ripped out by IT for data security, so then it's a hassle to go out and buy and extra SSD, compared to just buying a used laptop that already comes with display , keyboard, etc.

I have seen quite a few the size of a Mac Studio / Intel Nuc, what are the device called that are the size of a pi?

>I can buy on the used market a ~2018 laptop with a 15W quad core CPU, 8GB RAM, 256 NVME and 1080p IPS display, that's orders of magnitude more capable..

But it won't be as reliable, mostly motherboards won't last long.

Don't know what your source is for that, but that's not my experience, and i've had dozens of laptops through my hands due to my hobby.

The ticking timebomb lemons with reliability or design issues, will just die in the first 2-4 years like clockwork, but if they've already survived 6+ years without any faults, they'll most likely be reliable from then on as well.

Bathtub curve is extremely common https://en.wikipedia.org/wiki/Bathtub_curve

>survived 6+ years without any faults, they'll most likely be reliable from then on as well

Ok, let us say they ll last 4 more years, so 10 years total lifespan.

A PI would last a lot longer.

What makes you think so? Just a feeling? A Vibe?

About what exactly..

That if a laptop is 6 years old it will only last 4 more years. Or that a Pi will last more than 10.

If it is a generic laptop, yes. 10 years is a stretch. Components used in the motherboard are probably not high quality enough to last more than 10 years. A manufacturer does not have an incentive to put high quality stuff (that is probably costlier) in a laptop who's only selling point is cheap for the "features", and not reliability or longevity..

One might get lucky with such a laptop, but I won't count on it.

Again, is that just a feeling, or do you have some data to actually show this. In my experience even old basic Acer laptops easily last more than 10 years, probably without the battery and married to the charger forever now, but they will work fine. But I don't go on the internet and tell everyone laptops last most than 10 years just because I know of a few Acers lasting that. Likewise, do you have any statistics on longevity of Raspberry Pis.

What makes you assume that the Raspberry Pi is using higher quality components?

One thing is that Raspberry PI have a fewer of them. So less chance of one becoming faulty.

Regarding higher quality components, I think the for the usecase (I mean the kinds of thing it is supposed to be used for) of Raspberry PI, reliability is more important.

This also matches with my experience.

> Regarding higher quality components, I think the for the usecase (I mean the kinds of thing it is supposed to be used for) of Raspberry PI, reliability is more important.

That you think that reliability is more important for a Raspberry Pi usecase than a laptop doesn't somehow magically make it a fact that its components are of higher quality than your average laptop. You only speculate and then speculate further on the basis of your original speculation. That's not how you arrive at a basis for a factual claim or an estimate.

Sure, but $200 on eBay will get you something along the lines of a Dell Latitude, with decent build quality, cost-optimized more than a flagship workstation-class laptop, but certainly not designed to squeeze out the last penny at the expense of reliability or repairability like the cheapest consumer models.

And if you buy a 5-year-old corporate laptop in very good condition with minimal visible wear on the keyboard and touchpad, it was likely only used as a desktop replacement connected to a dock, so unlikely to have suffered abuse not apparent from visual inspection alone.

If you're planning to use it as an actual laptop, price out a replacement battery before purchase, as battery capacity will degrade over time, even if the laptop is exclusively used on AC, so will always be something of a crapshoot.

Otherwise, I'd expect the rate of component failure to be no higher than for any other lightly-used laptop of similar vintage, which is low.

Is it a fanless laptop? Fans aren't any different than filters in cars. They fail and need replaced.

>let us say they ll last 4 more years

Why not 50 more years if we're just making up numbers? I still have an IBM thinkpad from 2006 in my possession with everything working. I also see people with Macbooks from the era with the light up apple logo in the wild and at DJs.

>A PI would last a lot longer.

Because you say so? OK, sure.

In your comment you didn't say Apple computers or Thinkpads. Those are different. I was talking about plain old vanilla business class laptop (because we are talking about raspberry alternative).

You're contradicting yourself

I was referring to your original comment

>I can buy on the used market a ~2018 laptop with a 15W quad core CPU, 8GB RAM, 256 NVME and 1080p IPS display, that's orders of magnitude more capable..

I have computers that are ~20 years or even more and still work fine. My main computer which I just replaced is ~14 years old (with some components even older than that), was used every single day, and is now a perfectly functioning server. I have stacks of SFFs and minipcs from eBay going back to 2008 but most from 2012-2015, which have been running virtually uninterrupted for a decade, and still working fine. I have several laptops from different OEMs, business and consumer lines, that are as old as 2008 and have been used regularly for at least 10 years, all still fine.

I understand what you're saying but saying it isn't enough. There's nothing to support your claim.

3-5 years for a cheap laptop [0].

3-5 years of office use for a Pi. [1]

Sure, there's other numbers to find as well, but I'd suggest that they're pretty comparable in the way they handle environments. If one would fail, so would the other.

[0] https://pcpatching.com/2025/11/extend-your-pcs-life-how-long...

[1] https://raspberrypicase.com/how-long-does-a-raspberry-pi-las...

I still use Pis in my 3d printers. Laptop would be too big, and a ESP could not run the software. "China clone" might work, but the nice part of the pi is the images available. It just works™

I'm also currently building a small device with 5" touchscreen that can control a midi fx padle of mine. It's just so easy to find images, code and documentation on how to use the GPIO pins.

Might be niche, but that is just what the Pi excels at. It's a board for tinkers and it works.

Yeah sure, for niche use cases it's the best and only choice, but that's why it now commands niche prices ;)

Yeah, Pi 5 2gb is ~20% more expensive compared to pi3b on release, factoring in inflation (Both in including VAT and local prices)

It's 10 bucks more. ¯\_(ツ)_/¯ Still half the price that I see intel NUCs for sale. Which of course are way more capable. But still, I don't mind the price that much.

I could go with a cheaper alternative, but then AFAIK you might have to fiddle with images, kernel and documentation. For me that is worth 10 bucks.

>Yeah, Pi 5 2gb is ~20% more expensive compared to pi3b on release, factoring in inflation (Both in including VAT and local prices)

I don't really care how it compares to past models or inflation to justify its price tag. I was just comparing to to what you can buy on the used market today for the same price and it gets absolutely dunked on in the value proposition by notebooks since the modern full spec RPi is designed to more of a ARM PC than an cheap embedded board.

60 Euros for 2GB and 100 for 8GB models is kind of a ripoff if you don't really need it for a specific niche use case.

I think an updated Pi-zero with 2GB RAM and better CPU stripped of other bells and whistles for 30 Euros max, would be amazing value, and more back to the original roots of cheap and simple server/embedded board that made the first pi sell well.

A used notebook was also better in price to performance 10 years ago, no?

Yeash, but not as good as an alternative to a PI back then, since 8 year old notebooks 10 years ago (so 18 year old notebooks today) were too bulky and power hungry to be a real alternative. Power bricks were all 90W and CPU TDW was 35-45W. But notebooks from the 2018 era (intel 8th gen) have quite low power chips that make a good PI alternatives nowadays.

The mobile and embedded X86 chips have closed the gap a lot in power consumption since the PI first launched.

Now you can even get laptops with broken screens for free, and just use their motherboard as a home server alternative to a PI. Power consumption will be a bit higher, but not enough to offset the money you just saved anytime soon.

Laptops are still pretty bulky and power hungry in comparison if you're looking for very SFF and passive cooling.

Pop open a cheap Celeron N laptop, and often, the motherboard inside is essentially a passively cooled SBC.

https://www.insidemylaptop.com/wp-content/uploads/2018/03/Le...

The big pain with using something like this would mostly be the IO and odd form factor.

You can passively cool a laptop CPU too if that's your jam. Plus, latest RPI 5 cases now feature fans to not thermal throttle :))

You can get a 5-year-old laptop with a perfectly working screen for free if you're on good terms with the owner of a company who has a stack of them sitting in a storage closet waiting for disposal. :)

Which is basically just cutting out the middlemen in a transaction that might cost $100 on eBay.

Used corporate laptops are particularly cost-effective if you're interested in running Windows, as unlike Intel NUCs and most SBC products, they typically include hardware-locked Windows 10 Pro licenses which can be upgraded to Windows 11 Pro for free.

5-year old laptops for free aren't really a thing in most of Europe unless maybe you're in Norway or some super rich country where 100$ is pocket change. In most large places I worked in Europe, laptops are leased from a service provider, not owned by the company. When they're obsolete they get sold in bulk locally or abroad. But never for free.

That comparison was true back in 2012 when the first version was released, too.

Things like used PCs and forgotten closet laptops were running circles around brand-new Raspberry Pi systems, in performance per dollar, for as long as we've had new Raspberry Pis to make that comparison with.

Those first Pis didn't even have wifi, and they were as picky about power supplies and stuff back then as a Pi 5 is today.

The primary aspects that are new are that the featureset of new models continues to improve, and the price of a bare board has increased by an inflation-adjusted ~$10.

(Meanwhile: A bare Pi 3B still costs $35 right now -- same as in 2016. When adjusted for inflation, it has become cheaper. $35 in 2016 is worth about $48 today.)

> Yeah, Pi 5 2gb is ~20% more expensive compared to pi3b

What prices are you using for the 3b and 5 to get this percentage? The lowest percentage I got from available data is a 57% increase ($35 -> $55)

I added inflation.

40 EUR form 2016 is now ~52 EUR.

Compared to 62 EUR for the current model.

Are you adding in the correct 'RAM' inflation being that it's costs are up dramatically?

I just looked up the current local prices I can buy a unit for.

The main things I use Pis for are

1. Testing images to be deployed on customer Pis.

2. Testing software on ARM64 Linux. Pis are still cheaper than used Apple Silicon Macs, and require less fiddling to run Linux. I currently have a free Oracle Cloud instance that would work just as well for this, but it could go away at any time and it's a PITA to reprovision.

3. Running Mathematica, because it's free on Pi, I only use it a few times a year, and a fully-loaded Pi 5 is cheaper than a single-year personal license to run it on any other platform.

4. Silly stuff like one Pi 3 I have set up to emulate a vintage IBM mainframe.

You can run Klipper on any Linux SBC with a USB port, RPi works but so does an old router that supports OpenWRT, a cheap Android TV box that could be flashed to run Linux, or any of the OrangePi/Banana Pi/Alliwinner H3 boards. You don't really need hardware UART because most of the printer boards you'd be using have either native USB or USB to UART converters. For that pedal, would an old Android tablet that supports USB OTG work? Because that's got to be much cheaper, and with much better SDK.

Correct. But when I looked into it a few years back fir OrangePi it was not as easy as downloading raspbian. All the images made for the pi would not work, you had to download a kernel from another place or something like that? Sorry I don't remember the details, but it was not as easy as a pi.

How much cheaper then 50 bucks can a tablet get? With the pi I can quickly in a hacky way connect rotary encoders with female-female dupon cables, use a python GPIO library made for raspberry pi.

https://media.discordapp.net/attachments/1461079634354639132...

I can also use it for Zynthian. And if I'm done with it, I can build a new printer :P

I think the part you're missing, is how expensive the Pi is now.

Buy one with a psu, and you're 100 to 150.

Just a few weeks ago I bought a pi5.

See my other comment, pi5 2 GB is about ~10% more expensive then when 3 or 3b got released when factoring in inflation. ~60 EUR including 25% VAT.

With PSU it's 77 EUR including 25% VAT right now.

4 GB version + case + 64GB SD + PSU = 135 EUR, but I don't need that much ram, disk space or the case. When I put it into a 3d printer I also don't need the PSU.

> When I put it into a 3d printer I also don't need the PSU

Unless you want your printer to power up on demand, then you need a separate PSU and an SSR (and you still need a buck converter because printers don't supply 5V at required amperage).

Sorry, correct. I meant I don't need THE official raspberry pi 5A USB C PSU. It still needs power.

I looked back in your post history a couple of pages, didn't see said comment.

However, there is no way it's 10% more expensive than the original Pi with inflation. It's easily double, if not more.

You can buy low end tablets significantly cheaper now. A tablet with a screen, more ram, storage, a battery.

A Pi is a rip off, for what it costs to build.

It's not like they make the OS or kernel either. They basically copy and paste Debian, and add some fluff modz, most of it OSS they didn't write.

https://www.amazon.ca/Expansion-Octa-core-Processor-Touchscr...

https://www.amazon.ca/Raspberry-Pi-8GB-2023-Processor/dp/B0C...

My mistake, it was indeed 20% percent, aka 10 bucks.

rPi3 40 EUR form 2016 is now ~52 EUR adjusted for inflation.

Compared to 62 EUR for the current model rPi 2Gb.

Cheapest tablet with larger then 1gb ram locally is around ~85 EUR. All prices including 25% VAT.

RPi will still have lower power consumption and is far more compact. And mechanically reliable.

I'm in the market to replace my aging Intel NUCs, but RPi is still cheaper.

I agree completely - the NUC segment has a gaping hole post 2023, and faster raspberry pis can probably fill a lot of it especially for small scale commercial stuff.

>the NUC segment has a gaping hole post 2023

There are dozens and dozens of NUC style / form factor machines available these days. Especially cheap ones from China. Not sure what you mean by gaping hole post 2023. I'm running 3 of them with N97 and N150 Cpus. All bought within the last 18 months.

There are very few which are suitable for integration into other products - I currently build a scientific instrument that needs a fairly powerful SBC to run. Intel NUCs were well supported and documented: all of their firmware was updatable on linux without any issues, they had data sheets with power specs so you could run them off DC predictably, and you could buy boards without a case. There are plenty of small NUC shaped mini-PCs but few that are suitable for integration (at the price point intel was at).

Cheap Chinese mini PCs just aren't well documented and don't have predictable supply.

> And mechanically reliable.

What moving parts do competitors have to be less mechanically reliable?

In fact, a NUC or used laptop would be even more reliable since you can replace NVME storage and RAM sticks. If your RPI ram goes bad you're shit out of luck.

>RPi will still have lower power consumption and is far more compact.

Not that big of on an issue in most home user cases as a home server, emulator or PC replacement. For industrial users where space, power usage and heat is limited, definitely.

>I'm in the market to replace my aging Intel NUCs, but RPi is still cheaper.

Cheaper if you ignore much lower performance and versatility vs a X86_X64 NUC as a home server.

It feels like you think that the parent hasn't really considered their options and don't know what they really want.

> Not that big of on an issue in most home user cases as a home server

I don't know what "most home users" want, but I can understand wanting something more compact and efficient (also easier to keep cool in tighter or closed spaces), even at home.

> Cheaper if you ignore much lower performance and versatility vs a X86_X64 NUC as a home server.

Or maybe they noticed they don't need all the performance and versatility. Been there. It's plenty versatile and can run everything I need.

I don't want a used laptop. I have my NUC mounted inside a small enclosure on a bracket, with PoE for power. It's a single-purpose device, only used for HomeAssistant and nothing else. It also has to be located centrally in my house for better ZigBee/ZWave networking.

Unfortunately, it's close to dying. The heat from the CPU disintegrated the plastic of the SATA cable header on the motherboard. I fixed it for now with a bit of glue, but it's not going to hold indefinitely. And NUCs were pretty pricey.

RPi with a SATA/M.2 disk and a PoE hat is not that much cheaper than Intel, but it uses much less power. They also tend to not have cables that are kept under mechanical strain. I have a single-purpose RPi that's been running since 2014, and it's doing just fine.

I got a fanless Celeron N4020 4GB RAM 64G Storage new on Amazon for under $150 in Jan 2025, and it has been running home assistant ever since.

I don't think I could a RPi as cheaply once parts and power supply etc are taken into account.

> I don't think I could a RPi as cheaply once parts and power supply etc are taken into account

The RPi Zero 2W costs $15 and runs HA just fine. One can splurge on a pricey case, microSD, and high-amp GAN charger, and still be under 50% of your spend. You don't have to buy the flagship RPi.

500MB is just not enough RAM for a good HA experience. I would go with at least 2GB. If you have a few add-ons running I would go with 4 or more.

If it's an option I would always go with an SSD for HA. It makes a big difference in usability. Writing often and a lot to SD cards, like HA does, kills them way too fast

> 500MB is just not enough RAM for a good HA experience

Strong disagree: my experience has been great, my HA has been running on a Zero 2W for more than 2 years! I have several HACS plugins enabled - just not any if the video or AI stuff. The same Pi concurrently runs PiHole. For a while, it also acted as a git mirror via iSCSI-backed Gitea, but I had to migrate Gitea off of it since it was memory-hungry. You can do a lot with 500MB in headless mode.

Its not

Go price out a used 1l form factor PC.

After you buy a case, and a real disk, the pi, cost savings is gone.

Meanwhile you can pick up a used 8th gen intel 1L form factor for about 100 bucks. You can pick up one that will take a PICE card for 150ish bucks, with remote management.

The 8th gen or better intel has all sorts of extra features that may make it worth while (transcoding/video support).

Do you have actual links for that? I looked at small form-factor PCs and I have not found anything cheaper than RPis.

There are also N100/N150 SBC offers which are superior to RPi

In The Netherlands the first generation RPi was only sold to users with a Chambers of Commerce registration, I figured this was always the typical end user for it. Like schools, universities, prototyping for companies. Was the RPi in the rest of the world targeted towards home users?

* https://tweakers.net/nieuws/80350/verkoop-goedkoop-arm-syste...

Yes, anyone can buy it.

What? I ordered the original Pi in May 2012 from Farnell/Element14 without a Chamber of Commerce registration (KvK nummer). A couple of my colleagues did too.

Same here. Also bought one in NL back then without issues.

That was only possible with a KvK registration back then. They changed it after a while so that end consumers could buy one directly.

Yes they changed it after a while so people without a CoC/KvK could also purchase them. But initially only companies and institutions could buy them.

I'd love to buy a 500+

https://www.raspberrypi.com/products/raspberry-pi-500-plus/

I can't justify it though as I've no use for it.

However I think it is way closer to their original vision than anything else, i.e. It is a lot like the 1980s computers, the magic they were trying to capture.

I love the idea, but the price/power ratio is just a teeeny bit on the expensive side for me.

For 100€ that would be something I'd buy for every niece and nephew to play with. For 200€ it's not even for me, I'd rather buy something like the uConsole RPI-CM4: https://www.clockworkpi.com/product-page/uconsole-kit-rpi-cm...

Clockwork Pi experience with the CM4 is not good. 10 months to ship. Horrible Wifi performance, can't hold a link, and it only has around 50 minutes of battery life. I regret my purchase and it's sitting in my rack next to a bunch of old ham radios.

It’s still more up my alley than the pi 500

Sell it on ebay to one of those people that have to wait 10 months, instead of letting it rot on a shelf?

I had just gotten into Arduinos when the first Raspberry Pi came out.

I noticed I can do 90% of the stuff I'd use an Arduino for with a RPi, except I had the full power of an internet connected Linux machine available. The Arduinos are still collecting dust somewhere =)

But now we have the ESP32 filling the same niche along with the Pi Zero W, so I don't really understand the purpose of RPi 4 and 5. They're not cheap compared to the price nor very powerful in any measure.

You don't even need a full laptop, any Chinese miniPC will blow the RPi5 out of the water AND some of them have expandable storage+RAM, while also having 5-20x more CPU/GPU oomph. They do consume a few watts more power, so there _might_ be a niche for the Raspberry Pi, but it's not a big one.

You don't buy the Pi for its price: performance as a desktop replacement, you buy it for the incredible stability of the platform as a target, the support, and the addon ecosystem. If you want to screw around with taking a motherboard out of a laptop, go right ahead. €160 for a 16GB Pi5 that I know for sure will be available, replaceable, and supported for the next decade is more than worth the small investment.

> I don't really understand the purpose of RPi 4 and 5. They're not cheap compared to the price nor very powerful in any measure.

They are good for commercial installations like smart displays in stores (think big screens with menus behind fast food counters) and information kiosks. The extra HDMI port lets you drive two screens with one pi and the extra processing power keeps the UI smooth on high resolution. They also have hardware acceleration video decoding for shops wishing to play hi res promo videos and hobbyists building media terminals.

Cost is not a major concern here because the installation volume is low and there are far bigger expenses anyway. Just take a look at how much commercial displays are. The Pi company’s future supply guarantee is also nice because you know that within a given number of years if something breaks or you need another screen, you can just buy another identical pi and be done with it. Good luck sourcing a Chinese mini pc with compatible footprints, port orientations etc five years down the road.

I'm not building commercial installations though, I just use them at home.

I do get the business case of a stable repeatable platform, but I'm just tinkering with crap, not spinning up a hardware startup =)

I went the opposite direction. When the RPI came out I realized just how many tasks could be done with an arduino and do not need the full power of an internet connected linux machine.

RPi supporting Linux is already a big benefit. Using a laptop with a battery isn't very convenient and dangerous (always being charged).

then remove the battery?

The Raspberry Pi probably still has the advantage of an actually robust firmware/software ecosystem? The problem with SBCs has always been that the software situation is awful. That was the Raspberry Pi's real innovation: Raspbian and a commitment to openness.

> The problem with SBCs has always been that the software situation is awful

Awful how? A SBC can take advantage of many software written from the dawn of x86.

x86 SBCs are niche, and ARM or RISC SBCs are too complex to figure out the boot process and the drivers for an average tinkerer.

Fragmentation in the non-x86 world really hurts adoption. RPi presents a very well documented configuration that can be used as a target for development.

RISC-V is going through this exact same problem right now. All of the current implementations have terrible documentation, and tailoring Linux for each of these is proving to be difficult. All of these vendors include on-board devices that have terrible doc and software support.

ARM has a mitigation for this called SystemReady. It's basically "does your board support UEFI enough to usefully boot a battery of generic ARM Linux images". The Raspberry Pi can be made SystemReady, and Radxa also makes SystemReady-compliant SBCs you can buy.

RISC-V would do well to adopt and promote a similar spec.

I wonder why we are not storing base DTB on the board, just enough to boot the board.

Not everything needs to be for everyone. I think this is super cool - I run a local transcription tool on my laptop, and the idea of miniaturising it is super cool.

It is neat, and at 32GB it might be useful.

Almost nothing useful runs in 8.

This is the problem with this gen of “external AI boards” floating around. 8, 16, even 24 is not really enough to run much useful, and even then (ie. offloading to disk) they're so impractically slow.

Forget running a serious foundation model, or any kind of realtime thing.

The blunt reality is fast high memory GPU systems you actually need to self host are really really expensive.

These devices are more optics and dreams (“itd be great if…”) than practical hacker toys.

Vision models are way smaller and run decently on these things.

> Not everything needs to be for everyone

I wouldn't dare suggest that. The RPi was never for everyone yet it turned out it was for many. It was small but powerful for the size, it was low power, it was extremely flexible, it had great software support, and last but not least, it was dirt cheap. There was nothing like that on the market.

They need to target a "minimum viable audience" with a unique value proposition otherwise they'll just Rube-Goldberg themselves into irrelevance. This hat is a convoluted way to change the parameters of an existing compromise and turn it into a different but equally difficult compromise. Worse performance, better efficiency, adds cost, and it doesn't differentiate itself from the competing Hailo-10H-based products that work with any system not just RPi (e.g. ASUS UGen300 USB AI Accelerator).

> the idea of miniaturising

If you aren't ditching the laptop you aren't miniaturizing, just splitting into discrete specialized components.

I love RPI4s for local homelab roles like DNS servers, even for NAS with USB attached external storage, VPN gateways (with Tailscale).

It's actually not great since Ethernet is over USB on the pi 4 (edit this is not true, confused with pi 3). Not that it doesn't work, but I'd rather have an N100 minipc.

OTOH with ram prices being where they are and no signs of coming back down in the foreseeable future a second hand pi 4 may be a very wise choice.

I think you mean the pi3? On the pi4 the ethernet is connected to the main SoC.

> It's actually not great since Ethernet is over USB on the pi 4.

Not true, you're thinking about earlier models.

yeah... sorry about that (note I do have an N100 box as a home server, though)

Nah, they released products better suited to what people were already using Pis for.

The Picos are great for the smaller stuff, new Pis are great for bigger stuff, and old Pis and Zeros are still available. They've innovated around their segment.

The AI stuff is just an expression of that. People are doing AI on Pi5s and this is just a way to make that better.

I don't know of any other ARM device that fulfills:

- I can boot it w/o having to learn about custom U-Boot implementations

- I, as a consumer or small business, can buy

- Can not only buy today but also still buy in 2 years

- Doesn't cost a small fortune

- Can be tugged away behind TVs and other small niches

Odroids!

Do they still need a custom kernel?

I understand wanting to run a local LLM for privacy, but on something more powerful. Yes it's costly but what use is an LLM on a cheap board?

I was mostly responding to

> [...] if I want some low power linux PC replacement with display output, for the price of the latest RPi 5, I can buy on the used market a ~2018 laptop

I guess. I don't care about the AI hat at all.

Depends entirely on the model you want to run.

There are some absolutely useful things you can do with TTS/STT/Diarization/etc on even really minimal specs.

Some of those will run fine on RPis even without this new hat.

The extra ram probably opens the door to a large number of vision/image models, which typically want a minimum of 16Gb, but do better with 24/32.

There are just a HUGE number of case specific models that do just fine on hardware at the RPi level, assuming you have the ram to load them.

Why does it need to be ARM?

Not ARM, but:

https://www.gmktec.com/products/nucbox-g3-plus-enhanced-perf...

If ARM is a requirement, then RPi is your only option that I know of.

But also seems to cost ~twice as much.

Whenever the Pi gets brought up people on HN will tell you you are wasting money if you are not buying a Chinese Pi clone with beefier specs instead. But at least for me, "Being able to boot into a Linux system without having to dig through outdated wikis and Chinese language support forums to hunt down a google drive link to an OS image from 2021 that has since then received zero updates" is definitely worth paying ten extra dollars for.

Exactly this.

really good point, but I actually don't think the Pi's biggest competitor is Chinese clones, but just regular laptops and mini-pcs that have dramatically lowered in price over the years. I do still think Pis have a purpose though, it's just harder to justify in certain cases.

It depends on what you use your Pi for. If you are self hosting anything beyond what a Pi Zero can handle then a mini PC would most likely be a better choice than a Pi 4 or 5. If you are building embedded devices or are a hobbyist then the Chinese ARM SBCs are definitely in the competition.

Yes, it getting a lot more competitive generally. But the brand and community around PI is still A1.

Some of these Chinese 'clones' have substantially more mainline Linux support than any of the Pi boards.

You don't need an ARM, and for the same price, before the RAM crisis, you could get many tiny PCs for the same amount of money, with proper SSDs and power supplies and in tiny form factors.

https://teampandory.com/2024/09/24/gmktec-g5-mini-pc-review-...

Something like this ($155) cost less than the Pi + case + power supply + nvme addon board + ssd, and it also runs windows and any other x86 OS.

The problem here is "something like this". I may buy one of these today. Then in 6 months, I have to buy a different one. Only to find out that it's firmware has a bug. Or it uses a different ethernet chips that is a problem for me for some reason. Or something like that.

The RPi I can depend upon to be shitty as well, but in the exact same way. So it stays fit for purpose.

People have for quite some time been using Googles Tensor chip to accelerate AI workloads on the Pi. I doubt that anyone runs Llms on Pis but stuff like security cameras with object detection...

I read somewhere that Google has largely abandoned support for their Coral TPUs? I still use a Coral TPU for Frigate NVR. But not sure how long they'll be supported.

The Coral isn't recommended by it anymore for most use cases. They're going to support it, but it shows it's age by never being updated for years

> I think Raspberry lost the magic of the older Pis, they lost that sense of purpose. They basically created a niche with the first Pis, now they're just jumping into segments that others created and are already filled to the brim with perhaps even more qualified competition.

I don't think you will find anything on the market enabling you to create your own audiophile quality AMP, DAC, or AMP+DAC for a pretty attractive price except a Pi 3/4/5 with a HifiBerry (https://www.hifiberry.com/) HAT.

There are some nice ESP-based solutions - https://sonocotta.com/esp-products/

HifiBerry is basically the only choice when you live in Europe and do not want to deal with tariffs.

AFAIK what I sent above is made and shipped from Poland.

IMO this is a consequence of Raspberry Pi going for-profit and IPOing. Now they are incentivised to chase the same hype trains as every other public tech company. I can't see them having another "Raspberry Pi moment", those are too risky now.

That said, more options at the (relatively speaking) low end of the AI hardware market probably isn't a bad thing. I'm not particularly an AI enthusiast generally, but if it is going to infest everything anyway, then at least I would like a decent ecosystem for running local models.

I think this is a miss on what the Pi is: an experiment. Sure, it stood on the shoulders of other SFF boards that came before it - but it broke into the general computing landscape targeting makers and builders. If the AI hat doesn't work out, so be it. The use cases for this type of hat may yet to be seen. On one hand it may feel shortsighted to bringing hardware to market with no explicit use case, but that's part of the Pi brand.

As someone else mentioned: if the hat could efficiently be leveraged with the YOLO models on Frigate for a low volume camera setup that could be a nice niche use case for it.

Either way I hope the RPi org keeps dropping things like this and letting the users sort out the use cases with their dollars.

On device processing for robots and drones

Everyone here is missing the fact that this board was made by a third party, not the Raspberry Pi organization.

What? Sure, its technically using technology by hailo, but it's definitely from the RPI organization

It's not "made by a third party" - it's sold as a first-party add-on, by Raspberry Pi on their own website, in a Raspberry Pi branded package.

https://www.raspberrypi.com/products/ai-hat-plus-2/

It's no more "made by a third party" than any other electronics device made by a contract manufacturer.

Oh, I did not realize it was getting that first class treatment, I thought (from only reading the article) that this was just a hat made by a third party and sold for the ecosystem.

Gotta get that investor cash.

In regard to their niche, their niche is a ridiculously well-documented ecosystem for SBCs. Want to do something with your RPi? You can find it on Google, and the LLM of your choice is probably trained to give you the answer on how to do it. If you're just tinkering or getting a POC ready, that's a big help.

Of course, if you're in the business of hardware prototyping, and have a set of libraries and programs you know you're going to work with, you don't need to care as much.

My biggest issue is the lack of really good cases. There are all those fancy peripherals you can buy, but it's really hard to find simple case that works without overheating and no cables sticking out on all four sides.

That magic now moved to ESP32.

This is why I keep buying 3B+ and Zero 2 W and not any of the newer versions...it's much more in keeping with the relatively low cost board with GPIO and reasonable compute. It's kind of the last one they made that does what I kind of expected out of a Raspberry Pi at a reasonable price point. If I needed more compute I would have skipped the travesty that is ARM and just bought an x86 system.

I feel like if RPI doubled down heavily into education, they would be in a much better spot. They really could never win on price in the long run. But having a bit of K-12 and university budgets going to RPIs every year, especially during the "teach the kids programming" era, would I think make them a much healthier business.

Chromebooks did what RPI should have done.

I still remember the email I got telling me that they were going to upgrade the RAM of the 256Mb Model B I ordered and that I would receive a brand new model with 512Mb for no extra cost. Hard to believe that was nearly 14 years ago.

Was very helpful in me learning Linux. The only alternative I had at the time was a few old Pentium 4 machines, which were very noisy and my parents didn't like me leaving turned on for a long time.

The original Pis are still for sale, are cheap, and still do everything you need. This doesn't conflict with an expanded product line. The whole reason for Pi is still GPIO plus general purpose computing. AI is now a part of general purpose computing, so it only makes sense to adopt it too.

The things you can do locally with AI now are amazing. For several years there's been multiple open source products that can do both audio and visual processing locally using AI models. Local-only Home Assistant is almost equivalent to Siri. The more things you throw at it, the more computing power it needs (especially for low latency), and that's where the dedicated GPUs/NPUs (previously ASICs) are needed. And consider the expanded use cases; drones and robots can now navigate the world autonomously using a $150 SoC and some software.

Do sales backup these claims that come up very often or not? Does anyone has any data?

Although the op is not wrong, maybe their decisions are data driven and pay off?

May have lost the magic with regards to price, but they're commodity hardware now.

My rpi3 (that's been running since 2019) died last year and I was able to buy another and just plug in the SD card.

> In practice, it's not as amazing as it sounds.

8GB RAM for AI on a Pi sounds underwhelming even from the headline

In the UK I've never seen the hailo hats (which are quite old BTW) advertised for LLMs. The presented usecase has been object detection from lots of video cameras in realtime.

They seem very fast and I certainly want to use that kind of thing in my house and garden - spotting when foxes and cats arrive and dig up my compost pit, or if people come over when I'm away to water the plants etc.

[edit: I've just seen the updated version in Pimonori and it does claim usefulness for LLMs but also for VLMs and I suspect this is the best way to use it].

8GB? What is this, an LLM for ants?

It's more about demonstrating what's possible on a Pi than expecting GPT-4 level performance. It's designed for LLMs that specialize in tiny, incredibly specific tasks. Like, "What's the weather in my ant farm?" ;)

The vision processing boost is notable, but not enough to justify the price over existing HATs. The lack of reliable mixed-mode functionality and sparse software support are significant red flags.

(This reply generated by an LLM smaller than 8GB, for ants, using the article and comment as context).

You can run some models pretty decently using CPU inference only, things like Gemma 3 that are built for exactly that use case or some tiny speech to text models via llama.cpp that I have tested out (not so good). Although not the best for "heavy" tasks, if you just need a decent text generator that can produce more or less sensible, generic output you are good to go.

Is there any usefulness with the small large language models, outside perhaps embeddings and learning?

I fail to see the use-case on a Pi. For learning you can have access to much better hardware for cheaper. Perhaps you can use it as a slow and expensive embedding machine, but why?

A natural language based smart home interface, perhaps?

Tiny LLMs are pretty much useless as general purpose workhorses, but where they shine is when you finetune them for a very specific application.

(In general this is applicable across the board, where if you have a single, specific usecase and can prepare appropriate training data, then you can often fine-tune a smaller model to match the performance of a general purpose model that is 10x its size.)

I think there's a lot of room to push this further. Of course there are LLMs being used for this case and I guess it's nice to be able to ask your house who the candidates were in the Venezuelan presidential election of 1936, but I'd be happy if I could just consistently control devices locally and a small language model definitely makes that easier.

A few years ago this product would have just been called an ML Accelerator and marketed as helping accelerate ML workloads like object detection in images

Hitching their wagon to the AI train comes with different expectations, leading to a mixed bag of reviews like this.

A good illustration of how “can run LLM” ≠ “makes sense to run LLM”. A prime example of how numbers in specs don’t translate into real UX.

What’s the current state of the art in low power wake word and speech to text? Has anyone written a blog post on this?

I was able to run a speech to text on my old Pixel 4 but it’s a bit flaky (the background process loses the audio device occasionally). I just want to take some wake word and then send everything to remote LLM and then get back text that I do TTS on.

Maybe not SOTA but the HA Voice Preview Edition [1] in tandem with a Pi 5 or some similar low-power host for the Piper / Whisper pipeline is pretty good. I don't use it but was able to get an Alexa/Google Home-like experience going with minimal effort.

I was only using it for local Home Assistant tasks, didn't try anything further like retrieving sports scores, managing TODO lists, or anything like that.

[1] https://www.home-assistant.io/voice-pe/

Wake word can be tiny. Like 10k weights and can run on an esp32 or similar with plenty of compute to spare.

TinyML is a book that goes through the process of building a wake word model for such constrained environments.

Wake word is not expensive, you can do it on esp32 https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3... (and then send audio to something more beefy as TTS will be marginal at best).

As an edge computing enthusiast, this feels like a meaningful leap for the Raspberry Pi ecosystem. Having a low-power inference accelerator baked into the platform opens up a lot of practical local AI use cases without dragging in the cloud. It’s still early, but this is the right direction for real edge workloads.

Which ones?

For example camera based object detection. People have been using usb Dongel Google Tensors for that previously

For example camera based object detection. People have been using usb Dongel Google Tensors for that previously.

A smarter home assistant would be one

Interesting idea. I think the Jetson Orin Nano is a better purchase for this application. The main downside is the RAM is shared so you lose about 1G from the OS overhead.

can't wait to not be able to buy it, and also for it to be more expensive than a mini-computer

I buy a raspberry pi because I need a small workhorse - I understand adding RAM for local LLMs, but it would be like a raspberry pi with a GPU, why do i need it when a normal mini machine will have more ram, more compute capacity and better specs for cheaper?

a lot of people buy rpis because they are the only reasonable option for connectivity with power. I'm not sure what other devices you can get that have gpio and mipi connectivity with the ability to (potentially) run vlms and llms on them.

I daresay they could charge more than a comparably specced computer (if they don't already) and they would still be a viable purchase.

Surely with this hat you don't have any access to any GPIO?

Unless i'm missing something - which is where i'm like why not just buy a NUC with similiar RAM for far less.

You do still have access to the GPIO. This HAT [1] stacks on top of the GPIO connector but passes through all the pins so you can still use them. This one is connected through PCIe so it shouldn't be blocking off any pins from use, unless you wanted an NVMe SSD hooked up!

[1] https://www.raspberrypi.com/news/introducing-raspberry-pi-ha...

Ah great! Yeah that makes the benefit a lot more clear. I'm used to hats that seem to lock you out of GPIO use

> The Pi's built-in CPU trounces the Hailo 10H.

Case closed. And that's extremely slow to begin with, the Pi 5 only gets what, a 32 bit bus? Laughable performance for a purpose built ASIC that costs more than the Pi itself.

> In my testing, Hailo's hailo-rpi5-examples were not yet updated for this new HAT, and even if I specified the Hailo 10H manually, model files would not load

Laughable levels of support too.

As another datapoint, I've recently managed to get the 8L working natively on Ubuntu 24 with ROS, but only after significant shenanigans involving recompiling the kernel module and building their library for python 3.12 that Hailo for some reason does not provide outside 3.11. They only support the Pi OS (like anyone would use that in prod) and even that is very spotty. Like, why would you not target the most popular robotics distro for an AI accelerator? Who else is gonna buy these things exactly?

Sounds like some PM just wanted to shove AI marketing where it doesn't make sense.

Hmm. Can this "AI" hardware - or any other "AI" hardware that isn't a GPU - be used for anything other than LLMs?

YOLO for example.

Yes. The Hailo chips are mainly for AI vision models. This is the first time I have seen them pushed for LLM. They are very finicky and difficult to setup outside of the examples. Documentation is inconsistent and the models have to be converted to a different format to run. It is possible to run a custom yolo8 model, but is challenging.

I'd expect that kind of thing to be the primary use case, tbh. Maybe even running whisper models?

If it could run whisper, it'd be a solid addition to a pi based home assistant setup.

8GB is really low.

That said, perhaps there is a niche for slow LLM inference for non-interactive use.

For example, if you use LLMs to triage your emails in the background, you don't care about latency. You just need the throughput to be high enough to handle the load.

This looks pretty nice for what it is. However, the RAM is a bit oversized for the vast majority of applications that will run on this, which is giving a misleading impression of what it is useful for.

I once tried to run a segmentation model based on a vision transformer on a PC and that model used somewhere around 1 GB for the parameters and several gigabytes for the KV cache and it was almost entirely compute bound. You couldn't run that type of model on previous AI accelerators because they only supported model sizes in the megabytes range.

I've seen the AI-8850 LLM Acceleration M.2 Module advertised as an alternative RPi accellorator (you need an M.2 hat for it).

That's also limited to 8Gb RAM so again you might be better off with a larger 16Gb Pi and using the CPU but at least the space is heating up.

With a lot of this stuff it seems to come down to how good the software support is. Raspberry Pis generally beat everything else for that.

Excited to see more hardware competition at this level. Models that can run on this amount of RAM are right in the sweet spot of small enough to train on consumer-grade GPUs (e.g. 4090) but big enough to do something interesting (simple audio/video/text processing).

The price point is still a little high for most tasks but I’m sure that will come down.

My primary concern with this offering is vendor of the enabling silicon. I think it’s important to consider who you do business with. The industrial uptake of RPi boards has poisoned their mandate and made them forget making a world where all children can discover the magic of computing, not just the special chosen ones.

It's useless for LLMs and it's actually slower than Hailo 8H for standard vision tasks, so, why ?

Are there significant usecases for the really small LLMs right now (<10b distills and such)?

My impression so far was that the resulting models are unusably stupid, but maybe there are some specific tasks where they still perform acceptably?

They're still very good for finetuned classification, often 10-100x cheaper to run at similar or higher accuracy as a large model - but I think most people just prompt the large model unless they have high volume needs or need to self host.

Can someone share a non-theoretical LLM application that demands local inference and pocketable hardware? The internet hype for these is sky high, but the actual results are disappointing.

Any chance you can split layers so that some run on the CPU and some run on this board to let you run bigger models and/or get better performance?

I wonder -- how does this thing compare against the Rubik Pi [1]?

[1] https://rubikpi.ai/

Is that affiliated with raspberry pis in some way, or are they just freeloading on the "pi" suffix to confuse people?

> are they just freeloading on the "pi" suffix to confuse people?

Yes, but that is normal I guess:

- https://banana-pi.org/

- http://www.orangepi.org/index.html

- https://radxa.com/products/rockpi

They are "freeloading" indeed but in this case it is a valuable indicator that it should be physically compatible with many add-ons/attachments that are compatible with the Raspberry Pi.

Have we not all given up on pis at this stage? I’ve got ESP32s for edge I/O, Nucs/VMs for the crunch. What’s the Pi doing?

There are definitely use cases. Pis have lower power consumption than NUCs. This is the main reason I went for one to run Home Assistant rather than a NUC.

I have a NAS/home server that I could put it on but I don't want my home automation going down when I tinker with it.

I think I’ve got too used to VM snapshotting to turn away from it for HA.

It does seem like this is a marketing gimmick. I doubt it will have worked in terms of sales numbers. "What else does it do?"

I wonder if this is the magic hardware for LiquidAI/LFM2.5-Audio-1.5B

Dont need more than 8gb. It'll be enough power. IT can do audio to audio.

"For example, the Hailo 10H is advertised as being used for a Fujitsu demo of automatic shrink detection for a self-checkout."

... why though? CV in software is good enough for this application and we've already been doing it forever (see also: Everseen). Now we're just wasting silicon.

What a pointless product to waste time making.

This AI 8GB HAT is $150, while better performing Pi5 8GB is $100. Makes no sense to buy the AI HAT.

What not use a USB Coral TPU? Seems to do mostly the same stuff and is half the price.

Coral is many times slower at this point (2 TOPS IIRC), but if it meets your needs, it's okay.

Yeah, I think it's 4 TOPS, so I see.

Glad Jeff was critical here they need a bit of a wake up call it seems.

i wonder how the Hailo 10H compares to Axera AX8850. add on boards seem to be cheaper and its a full SoC that can also draw much more power.

can You add gemma 3n to benchmark? this is less tahan normal 3 gemma but still good in small memory

Gigabytes?? In THIS economy?

Catalog TG 211, 1000 Hz.

At this moment my two questions for these things are:

1. Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc.

2. Can I run object/person detection on local video streams?

I want some AI stuff, but I want it local.

Looks like the answer for this one is: Meh. It can do point 2, but it's not the best option.

1. Probably, but not efficiently. But I'm just guessing I haven't tried local LLMs yet.

2. Has been possible in realtime since the first camera was released and has most likely improved since. I did this years ago on the pi zero and it was surprisingly good.

> Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc.

No. Get the larger PI recommended by the article.

Quote from the article:

> So power holds it back, but the 8 gigs of RAM holds back the LLM use case (vs just running on the Pi's CPU) the most. The Pi 5 can be bought in up to a 16 GB configuration. That's as much as you get in decent consumer graphics cards1.

> Because of that, many quantized medium-size models target 10-12 GB of RAM usage (leaving space for context, which eats up another 2+ GB of RAM).

…

> 8 GB of RAM is useful, but it's not quite enough to give this HAT an advantage over just paying for the bigger 16GB Pi with more RAM, which will be more flexible and run models faster.

The model specs shown for this device in the article are small, and not fit for purpose even for the relatively trivial use case you mentioned.

I mean, look, lots of people have lots of opinions about this (many of them wrong); it’s cheap, you can buy one and try… but, look. The OP really gave it a shot, and results were kind of shit. The article is pretty clear.

Don’t bother.

You want a device with more memory to mess around with for what you want to do.

That seems completely and utterly pointless.

A NPU that adds to price but underperforms a rasp cpu?

You get SBC with 32gb ram…

Nevermind the whole minipc ecosystem which will crush this

Are computers even for anything else anymore? Is it just AI until the bubble pops? This is exhausting.

I had a couple of Pis that I wanted to use as a Media center, I always had some small issues that created a suboptimal experience. Went for a regular 2nd hand amd64 with a small form factor and never looked back, much better userspace support and for my use case a much smoother experience, no lags no memory swapping and if needed I can just buy a different memory bank or a different component. I have no plans to use a raspberry pi any time soon. I am not sure these days if they really still have a niche to fill and if yes how large this niche is.

Reply on news.ycombinator.com