A PCIe Coral TPU Finally Works on Raspberry Pi 5 | Hacker News Reader

A PCIe Coral TPU Finally Works on Raspberry Pi 5

jeffgeerling.com

114 pointsmikece27 comments

Open articleView on HN

Coral is 4 years old, and it's both shocking & not that there aren't many competitors out there today.

Also a bit sad PyCoral requires python 3.9! Yikes!

Especially since it only has like 8 MB of memory I think?

Some SoCs even have competent built in NPUs, too, but software support is severely lacking.

PyCoral and Coral hardware development seems glacial lately :(

They have enjoyed a lot of momentum... I wish they could release a follow-up version and get some longstanding software issues resolved.

Orange Pi 5 has RK3588 with a neural processing unit (NPU).

SDKs for it

https://github.com/rockchip-linux/rknpu2

https://github.com/rockchip-linux/rknn-toolkit2

Quickstart and usage guide:

https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html

One of the most popular uses I've seen for Coral, Frigate for object detection for security cameras, recently added support for the RK3588 NPUs too: https://github.com/blakeblackshear/frigate/pull/8382 . I think it requiring a RK3588 makes it a bit annoying to have to upgrade your entire setup just for this though.

How many camera feeds can one Orange Pi 5 realistically monitor?

According to the author of that PR, they're using 10% of 1 NPU core on 3 cameras: https://github.com/blakeblackshear/frigate/pull/8382#issueco...

The bottleneck instead will probably be the video stream decoding speed, especially as the SoC's hardware decoder isn't being used yet.

I think companies prefer to develop AI stuff and then sell it, rather than sell the hardware.

However keeping the Python version up to date shouldn't be that hard though, should it?

3.9 isn’t that bad on my opinion. I work with other Python libraries, popular ones, that won’t work past 3.9. I think PyPDF is one of them? Something with PDF parsing.

The Hailo (https://hailo.ai) is the biggest competitor at the moment. It is a lot more powerful while not requiring much more power. It does, however, have some problems with Pyhon's GIL and documentation is not always perfect. Other than that it's like the Coral but much better.

Any alternative that has a simple USB connector?

It's also a bit more expensive :)

But I am going to test one of those out soon, too! Hopefully it's software stack is maintained a bit more actively than Coral

By "a bit more expensive", you mean $25,000 as opposed to Coral's $25? Because they will not even tell me the price without me filling out a big form.

The software is actively maintained yes. The price is between 100 to 200$.

>Also a bit sad PyCoral requires python 3.9! Yikes!

It's a bit on the old side, but hardly yikes territory. Ubuntu 20.04 ships with 3.8 [1] as the system python with support until April 2025, and AWS supports 3.7-3.11 for lambda runtimes [2]. 3.12 has only just been released

[1] https://packages.ubuntu.com/focal/python3

[2] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes...

It needs a whole bunch binary bindings. It is not really supported since Ubuntu 18 or pretty much any other operating system.

If you look at the github repo and bug tracker you will see it's largely abandoned. (I have a Coral USB and you can get it running with an old operating system version).

Well that's an entirely separate issue than the python version

Yes, that is my point. It's not really a python version issue.

An HBM3E HAT would or would not yet make TPUs more useful with a Raspberry Pi 5?

Jetson Nano (~$149)

Orin Nano (~$499, 32 tensor cores, 40 TOPS)

AGX Orin (200-275 TOPS)

NVIDIA Jetson > Origins: https://en.wikipedia.org/wiki/Nvidia_Jetson#Versions

TOPS for NVIDIA [Orin] Nano [AGX] https://connecttech.com/jetson/jetson-module-comparison/

Coral Mini-PCIe ($25; ? tensor cores, 4 TOPS (int8); 2 TOPS per watt)

TPUv5 (393 TOPS)

Tensor Processing Unit (TPU) https://en.wikipedia.org/wiki/Tensor_Processing_Unit

AI Accelerator > Nomenclature: https://en.wikipedia.org/wiki/AI_accelerator

NVIDIA DLSS > Architecture: https://en.wikipedia.org/wiki/Deep_learning_super_sampling#A... :

> DLSS is only available on GeForce RTX 20, GeForce RTX 30, GeForce RTX 40, and Quadro RTX series of video cards, using dedicated AI accelerators called Tensor Cores. [23][28] Tensor Cores are available since the Nvidia Volta GPU microarchitecture, which was first used on the Tesla V100 line of products.[29] They are used for doing fused multiply-add (FMA) operations that are used extensively in neural network calculations for applying a large series of multiplications on weights, followed by the addition of a bias. Tensor cores can operate on FP16, INT8, INT4, and INT1 data types.

Vision processing unit: https://en.wikipedia.org/wiki/Vision_processing_unit

Versatile Processor Unit (VPU)

Makes me wonder how Radxa Rock 5B's NPU can be used:

- Supporting INT4 / INT8 / INT16 / FP16 / BF16 and TF32 acceleration

- Computing power is up to 6TOPs

ive got some pcie corals if u want to buy one. lmk

What are your asking prices?

make me an offer.

I'm not sure exactly what pcie corals you're referring to. If you mean the m.2 ones, it'd have to be less than the $25 or so MSRP, but if you have full-size cards I'd be very curious to see what they even look like first.

My guess is they’re referring to the mini PCIe cards. I think Coral only comes in NVMe and mini PCIe (and technically you can buy just the chip itself I believe).

Obviously mini PCIe is deprecated and it’s unlikely many people have a device with a mini PCIe slot AND they don’t need that slot for WiFi/BT. That’s probably why they’re looking to sell.

The adapter for this would cost more than the device itself, so I don't think I can use it. I'd take an NVMe module in a heartbeat though.

Reply on news.ycombinator.com