YOLOv5 on FPGA with Hailo-8 and 4 Pi Cameras

Great question! I work for a computer vision company (Roboflow) and have seen computer vision used for everything from accident prevention on critical infrastructure to identifying defects on vehicle parts to detecting trading cards for use in video game applications.

Drawing bounding boxes is a common end point for demos, but for businesses using computer vision there is an entire world after that: on device deployment. This can be on devices like an NVIDIA Jetson (a very common choice), to Raspberry Pis to central CUDA GPU servers for processing large volumes of data (maybe connected to cameras over RTSP).

Note: There are many models that are faster and perform better than YOLOv5 (i.e. YOLOv8, YOLOv10, PaliGemma). Roboflow Inference that our ML team maintains has various guides on deploying models to the edge: https://inference.roboflow.com/#inference-pipeline

I use object detection to track tardigrades in my custom motorized microscope. It's very useful for making long observations in a field much larger than the scope's field of view.

The system works quite simply: I start with an existing object detector and train it with a small (<100) number of manually labelled images. Then during inference, I move the scope's field of view using motor commands to put the center of the tardigrade at the center of the field of view.

This technology is very useful for doing long-term observations of tardigrades (so, useful for science).

Frigate uses models like this one for NVR:

https://frigate.video/

This hailo 8l is extremely useful in mobile robotic systems particularly when paired with an rpi 5. The main board used in this demo is.... not particularly useful however. Generally speaking boards like fpga miss the forest for the trees. As a systems engineer, where am I supposed to put such big board? Also I can expect that, even if my team incorporated this board into a mobile system, it become vaporware well before we deploy anything due to low production numbers and we'd be paying ebay scalpers 2x as much because our distributors wouldn't carry them anymore lol.

As for the rpi 5 combination, the power draw is relatively low. The whole thing clocks in at about 14 watts on an rpi 5 which allows us to run this platform off of a battery. With 26 tops, this setup can contend with the jetson xavier nx (21 tops, ~$500) and the jetson orin nano (40 tops, ~$500) for for a cost of around $170. Furthermore the cpu on the rpi 5 is generally more performant than the xavier nx.

Specifically, this is an excellent vision module for real-time object detection from multiple cameras if setup properly, while maintaining access to the prolific raspberry pi hardware ecosystem which is typically cheaper than the jetson ecosystem.

I'm working on a project that detects climbing holds and lets you set routes by selecting them. (The usual method is putting a bit of colored tape on each hold, where the color corresponds to a route. This works great but becomes difficult to read once more than four or five routes share a hold.) YOLO made the computer vision part of this pretty smooth sailing.

As you guessed, high-speed machine vision stuff is frequently used in manufacturing settings for sorting or various quality control tasks. Imagine a picking out bad potatoes on a conveyor belt moving 10s of potatoes per second, or identifying particle counts and size distributions in a stream of water to gauge water quality.

I've been very persistent over the past few months in developing a system for agriculture as a primary use case. I want to deploy features to classify crop type, height, vegetation stage, and other important metrics to achieve real-time or near real-time analytics.

Do you have any suggestions on how to proceed further? So far, I've procured a Jetson, five cameras, a stand to fix and calibrate the modules, and a cam array hat to equip four cameras and the jetson. I was checking out VPU and NPUS and other hardware as well but struggling to identify compatible hardware. How can I get ahead and build such model to test and validate in 3 Months of time ?

I've used YOLOv5 models in robotics for object detection. While VLMs are great at describing images more generally, having a bounding box of a detection with a confidence score is very useful when paired with depth cameras for locating objects in an open environment. Especially when it can be run on-board and at framerate.

Field mice that I thought were moles have destroyed my yard. There’s so many tunnels that I can’t tell which are most active. A camera AI that can show which parts of the ground changed significantly would be nice.

At a hotel, we had a problem of luggage carts going missing. There’s a few ways to deal with that. A generic one that would support other, use cases would be to let the camera tell you the last room it went in. Likewise, outdoor cameras might tell you which vehicles had a customer walk in the hotel and which might be non-guests.

> I've also been a part of the Kinect craze and made 3 demos (games mostly) using their SDK

I ought a Kinect but never got it working correctly with a Mac. What is the state of the art here ? Is there active development anywhere ?

To surveil people in the streets.

Breaking: people in public don’t want to be seen by the public.

It’s a detailed breakdown of a technically impressive project and your main takeaway is the 5 seconds in the demo where a guy covers his face?

Kudos to the author for making something neat and sharing it.

A good point that shouldn't be dismissed so quickly. This tracking also has little use beyond surveillance, so one has to wonder what is it that the author thinks they are doing, or why they think it's interesting or useful. That they then go ahead and film a crowd without their consent is more telling of their position on this moral question than it is any direct harm to the people in the video.

Oh, the horror. Holy shit was that Jason walking by? He told me he was in Alaska...I'm going to need to have a talk with him. Thank goodness I ran across this video.

The ultrascale is definitely pricy for non-industry applications. The FPGA design is probably larger because of the 4x camera pipeline.

Perhaps with a single camera you could port this to fit on a Zynq 7000 footprint with something like a Pynq Z1 or Numato Styx, which are in the $250 hobbies price point for example.

Depending on what you're trying to do you might not need it. A popular option for object detection people use with their home hosted video surveillance (using Frigate and Home Assistant) is the Coral TPU and Raspberry Pi or some decommissioned thin client.

~400kLUT is very obtainable these days - if you don't need the ARM cores:

https://www.lcsc.com/product-detail/Programmable-Logic-Devic...

(420T/480T are priced similarly new.)

At the moment I am wondering if I could build an accelerator for george hotz' tinygrad[1] with cheap FPGAs (i do have an arty a7 35K, this might be too small, i guess?). According to the readme it shall be "easy" to add new accelerator hardware. Sadly my knowledge is still a bit limited around all the python-machinelearning-ecosystem, but if I understand it correctly you "just" need an openCL kernel and need to be able to shove the data back and forth somehow.

Didn't have enough time to dive into it yet and still working on some other project, but this still tickles the back of my head and would be cool even if I could only run mnist on it.

- [1] https://github.com/tinygrad/tinygrad/

It's line speed processing of multiple cameras in HW - it should be less power consumption than equivalent GPU or Jetson.

That already exists. Lilin is one of several CCTV companies implementing on-camera YOLO. Axis, Hanwha, and i-Pro all have options for you to run your own software/models on camera as well.

According to [1] the manufacturers claim 2% of the TOPS of a RTX 4090, and only 0.8% the power consumption.

$200 in prototype quantities [2] which is 12% the price of a 4090 - but perhaps the price drops when you order them in bulk?

They claim it compares favourably to an 'Nvidia Xavier NX' for image classification tasks, providing somewhat more FPS at significantly lower power consumption. 218 fps running YOLOv5m on 640×640 inputs.

They're completely silent about the amount of memory it has, but you can fit int8 YOLOv5m into about 20 MB so it'll certainly be an amount measured in megabytes rather than gigabytes.

Their target market is "CCTV camera that tracks cars and people" rather than "Run an LLM" or "train a network from scratch"

[1] https://hailo.ai/products/ai-accelerators/hailo-8-ai-acceler...

[2] https://up-shop.org/hailo-m2-key.html

Hailo is one of the newer GPU startups, so not surprising that many people have not heard of them yet. So far price/performance/power consumption of the Hailo products seems to be filling a rater large gap between the Amba stuff that is very well suited for 1-4 camera streams in a typical SoC-based device, and the Jetson, which is really kind of overpriced and power hungry for a lot of video applications (at least IMO).

Very cheap and power efficient if you're willing to run in int8/int4 and have unlimited time and patience for development

It’s a competitor to Google Coral (seems abandoned) and NVIDIA Jetson. I’ve been using it for more than a year and the hardware seems to be one of the best on the market. The software (how to actually do inference on the chip) is subpar though.