Hacker News Reader

Comment by mynegation | Hacker News Reader

I wish there was some reference of what can I actually do with 4 or 8 TOPS and 4 or 8 or whatever GiB of accelerator memory. Can I run speech recognition? Run a model for object detection in images? In video? LLMs - probably not. Stable Diffusion also seems out of question. But really - what can I run?

View on HN

For scale, Microsoft requires 40 TOPS to certify a (consumer) computer as Copilot capable.

https://www.techpowerup.com/320933/microsoft-copilot-to-run-...

Agreed! What I really want to know from any hardware accelerator is:

* What can it accelerate?

* What software supports it (and what's the state of driver support)?

* How much faster or more power efficient is it vs CPU/GPU?

* How exactly do I use it? (Okay, I buy the thing, it arrives in a box, I plug it in... what exact steps do I take to get to running real workloads on it?)

There's about two 240p object detection demo videos with source code, one picture object detection demo, a bunch of forum posts with incomplete code about some audio processing, and they're all about a different SBC. Software wise, there's not much to it, you're on your own.

4 TOPS is last-gen smartphone territory - you aren't going to have a good time running big generative models, but it's a useful amount of performance for typical edge workloads like machine vision and robotics.

Efficientdet gives almost acceptable performance for security cams on an i5 if you only decode keyframes, I wonder how many NVR channels it could handle?

google coral tpu is 4Tops btw

These are INT8 TOPS are they not? That being the case, the Orin NX series are 50 TOPS I believe, the Orix AGX being 100 TOPS.

Orin AGX is advertised with 275 int8 Tops.

From 2019 though.

One thing that comes to mind is AI detection in Frigate https://docs.frigate.video. I’ve not played very much with AI but I’ve dabbled with Coral this past month and it would definitely be nice to have a fully integrated machine that could be dedicated to my security video feeds.

I was given a coral USB by a friend, and it is handling detection for 6 cameras without issues on frigate.

I have configured the cameras (had to go through an activX on IE6 via a MS Windows VM, yay) to serve two streams. 480x360 5fps for the AI detection, and 2592x1944 25fps for recording.

Only the low quality stream is decoded on the computer, for a very light load.

4 TOPS is the same as the Coral Accelerator. Look on YT for how people have used it. Another poster mentioned video object recognition with Frigate.

The bigger question for me is if it's easy to uses or add the accelerator memory to your application. For something like a virtual assistant or something with some basic image recognition, it can be really confusing about how to take advantage of the different hardware tools that are there. This is especially true in the maker space, when you're trying to leverage open source tool kits and don't necessarily have the time or knowledge to write a full, lower-level driver for open source tools kits.

Probably good enough to run openCV and turn it into a people and object recognizing brain for cameras but not sure.

Reply on news.ycombinator.com