Frigate: Open-source network video recorder with real-time AI object detection
frigate.video
frigate.video
Frigate standalone works super smooth with no hiccups at all. I am using object detection for people and have not yet had a false positive or false negative. Additionally, I record not only the events, but but also a 24/7 video. Frigate takes care of garbage collecting old assets.
I have it hooked up to my Home Assistant running on the same raspberry pi. From there, I get notifications to my phone which include a live video, snapshot and video recording. The UX and configuration options are way better than any commercial end user product I have found.
It’s been a literal lifesaver, also fun and easy to use. Would recommend 10 of 10. I have no affiliation with the maintainers.
It's nuts how cheaply you can make such a good system with AI-detection features, its more than paid for itself vs commercial options with monthly fees. High quality weatherproof PoE cameras are crazy affordable now too, and you can VLAN them off your home network with no connection to the internet to further harden the system.
Curious how you claim zero false negatives though (as in missing a person it should have detected), unless you’re reviewing all the data or have another system hooked up to it verifying? Or perhaps you simply mean to imply nothing bad has happened due to a missed detection?
In curious hope it did during Halloween with all the costumes? Are you able to have it pre alert you that kids are coming to the door?
The “yard” part, I review every week. Here, of course, I have no data to compare it to, so I cannot say if it’s missing anything.
So yes, I’m certain it misses nothing that’s important to me - is there someone at my door and who is it.
It works well with other costumes like Trail Running and Hiking gear. It also works in the dark (IR camera) and in crazy hard tests like just peeking around the corner with only part of the face visible.
If you’re interested, here’s an intro to the relevant legislation: https://www.edoeb.admin.ch/edoeb/de/home/datenschutz/ueberwa...
The whole thing is running in docker-compose on Raspberry Pi and Coral TPU.
How would you know if you have zero false negatives unless you watch the whole video stream everyday?
Can't say it saved my life, but I programmed my smart bulbs to go red if a bear was spotted in any frame in the past 30 min though.
Frigate is great and worth of praise.
I did hear about the farting causing VOCs to rise and that kind of happens with us too.
Whose lives have been saved?
> also fun
Do people actually work in security companies for the LOLz? Nobody told me.
Here are the observations:
* You don’t actually need hardware decoding or a Coral, but they do help. You will of course need to provision more CPU horse-power for NVR. * Motion detection uses the usual implementation from OpenCV. Unfortunately this algorithm is not very good in my experience. Many things I would consider as motion are missed (false negative), many things I would not consider motion are being detected (false positive). These factors mean that one is tempted to go ham on masking to filter out false positives, which then leads to further false negatives. I’m genuinely surprised the motion algorithm that’s implemented in OpenCV is still the state of art of what’s available openly. * Object detection is somewhat knee-capped by the models available publicly. They are not very good either. Frigate has built its behaviour around these models with an assumption that these models are largely pretty accurate, which in my experience has ended up with quite a few missed recordings for important events, which led me to switch to create recordings based on motion (I’m not in a very densely populated area and reviewing the recordings isn’t too onerous.) * Support for coral is… shaky at best. There are some indications that the production of these devices has largely stopped (and finding them to purchase is hard and expensive,) and maintenance of the drivers and libraries to interface with coral seems to be minimal or non-existent to the point where some Linux distributions have started dropping the relevant packages from their repositories. On the upside, running these models on the CPU isn’t that expensive, especially considering that the models are invoked very sparingly.
I’m currently thinking of moving over to continuous recording, perhaps trying out moonfire-nvr or mayhaps handwriting a gstreamer pipeline. Simple software -> fewer failure modes.
(NB: I worked at a computer vision startup in the past, my views are naturally influenced by that experience.)
I've recently gone through the process of trying to install pycoral on Rocky Linux 9. I had to build from source, and there was some challenge because documentation for the build process was sparse. There was some conflicting information about files I had to edit, values I had to set, what was supported and what wasn't.
Moonfire's author here. Please do give it a try! Right now it's a little too simple even for me, lacking any real support for motion or events. [1] But I'd like to keep that simple server core that just handles the recording and database functionality, while allowing separate processes to handle Frigate-like computer vision stuff or even just on-camera motion detection, and enhance the UI and add stuff like MQTT/HA integration to support that well. I'd definitely welcome help with those areas. (And UI is really not an area of expertise of mine, as you can see from e.g. this bug: <https://github.com/scottlamb/moonfire-nvr/issues/286>.)
For now I actually run Moonfire and Frigate side-by-side. They're almost complete opposites in terms of what they support, but I find both are useful.
[1] The database schema has the concept of "signals" (timeseriesed enums like motion/still/unknown or door open/door closed/unknown), but my code to populate that based on camera or alarm system events is in my separate "playground" of half-finished stuff, and the crappy UI for it is rotting in one of my working copies. I'd like Moonfire's database/API layer to also have a more Frigate-like concept of "events" and one of "object tracks".
So more realistically, I see Frigate as something I use in the short- to medium-term and take inspiration from in the long term. I'm not above essentially copying their motion and object tracking algorithms into a plugin, with proper attribution of course.
Frigate is neat because it challenged my idea of what the "minimum viable product" for an NVR could be. When I first looked at it, IIRC it didn't really have any UI at all and didn't support continuous recording. Instead, it just saved events as .mp4 files and published metadata over MQTT for a Home Assistant-based UI. It hadn't occurred to me that would be a useful system, but of course it was.
In frigate 0.13 (currently in beta) the motion detection has been fully rewritten, which has been a large improvement in my and other's experience. We also have docs now that walk users through tuning the motion detectoin.
This is along with many other changes along what you are describing like object tracking and improvements to initial object detection when motion is first detected.
Handwriting a GST pipeline is pretty much what I did. I start with frame differences(I only decode the keyframes that happen every few seconds, so motion detection has to work in a single frame to have good response time).
Then I do a greyscale erosion to suppress small bits of noise and prioritize connected regions.
After that I take the average value of all pixels, and I subtract it, to suppress the noise floor, and also possibly some global uniform illumination changes.
Then I square every pixel, to further suppress large low intensity background noise stuff, and take the average of those squares.
I mostly only run object detection after motion is detected, and I have a RAM buffer to capture a few seconds before an event occurs.
NVR device code(In theory this can be imported and run from a few like python script), but it needs some cleanup and I've never tried it outside the web server.
https://github.com/EternityForest/iot_devices.nvr/blob/main/...
GST wrapper utilities it uses, motion detection algorithms at top:
https://github.com/EternityForest/scullery/blob/Master/scull...
My CPU object detection is OK, but the public, fast, easy to run models and my limited understanding of them is the weak point. I wound up doing a bunch of sanity check post filters and I'm sure it could be done much better with better models and better pre/post filtering.
My bigger priority has been moving from Mako to Jinja2, especially for some particularly horrid templates that could not be highlighted or formatted because there's not many good Mako tools, JSON schema validation, but I definitely agree type annotations are critical.
VS Code is smart enough to catch a lot of stuff sans annotations though, so you can get by with a lot of nonsense, especially when half your time is just fighting GStreamer and you're not paying as much attention to the python side.
There's nothing better than GST that I've ever seen for dealing with media without actually having to touch the performance critical stuff in your own code, but it is not easy to debug stuff buried in autogenerated python bindings to C code, especially with an extra RPC layer to use a background process and defend against segfaults.
There's also lots of other weird stuff, like imports not at the top of the file, meant to support systems where some module wasn't available, and generally all kinds of cleanup that's slowly happening.
Yep, the current models are based on ImageNet which is of course wildly different content to the typical security camera. It's no surprise that its recognition is often pretty poor, especially from the typical ceiling angles that cameras are mounted at
I highly recommend - best model we tried out of dozens of public models.
Space management for rolling retention is also a new feature in Frigate and very basic, I don't think it has a way to do different retention policies by camera group and alarm.
There's only been one incident where I would have liked continuous, I've tweaked events to be more than enough.
Frigate already supports customizing recording retention per camera for 24/7 and event based recordings.
You can also set different retention periods based on the type of objects that were detected.
Digital zoom also can be more useful.
What I’ve learned from clients is to get the best resolution cameras you can and PoE cameras only if the use case remotely is safety or security.
go2rtc also works nicely for on demand transcoding of my H265-only cams to H264 to view the live stream in Firefox.
for security based purposes, why would you want to save all of that data that is not changing? you'll just end up fast-forwarding to the interesting bits anyways if you have to go to the footage.
Unfortunately consumer devices are almost always cloud-based, where storage but especially upstream bandwidth are much more costly considerations, so recording only on detection has become the norm in the consumer world.
External triggers are also an important feature in commercial systems that a lot of open source projects miss---but Frigate isn't guilty of this one, it can receive triggers by MQTT, which is the same thing I do right now with Blue Iris. That's the big thing that has me optimistic about Frigate going forward. Because motion and object detection are so inconsistent, triggering VMS events based on access control systems and intrusion sensors is often a much more reliable (and even easier to maintain) approach.
If someone was walking across the yard it would save every frame. The movement of the sun would move shadows enough to trigger a new image every few minutes. A bug flying past was small enough that it wouldn't trigger anything. The result was you could get a short video of everything interesting that happened through the day: shadows of trees sliding over the ground, every frame of the car pulling out of the driveway, shadows sliding over the ground some more, cat walks across the yard then lays down, shadows pan around more while the cat sits still, cat gets up and walks away, shadows pan around until the delivery guy comes...
It was an incredibly low-CPU way to see everything that happened without missing anything, and without having to fine-tune the motion detection very much. You just mask out any areas with constant motion, then adjust the slider for how much change triggered the next frame, which would let you adjust how fast the timelapse would go during the boring parts.
I've always wondered why the technique never became widespread.
Essentially, the same concept, just need enough of a buffer to allow for the pre-roll which wouldn't be a lot at the lower bitrate IP data coming from the cameras
- Redundant disks/mirroring on the NVR
- Replication of Frigate's Event Database and Recordings to remote network storage
I primarily use Frigate as a general event index, with 'active-objects' as its recording criteria, and look at the NVR when there may be gaps in Frigate's coverage.
I've also been writing my own software to integrate with Frigate to help make better sense of activity and events at a macro level, compared to its current user interface.
Can I somehow use these with Frigate? Is there a way to root these Ring cameras and use them?
I never liked the idea of paying a service fee, nor having Amazon pull the videos into their free neighborhood watch program.
Any suggestions on using them now?
Also most of those ring cameras are 2.4ghz only so you need a dedicated 2.4ghz wifi network. It won’t connect at all if you have an SSID that is broadcasting both 5ghz and 2.4ghz on the same name.
Paired with running Scrypted for HomeKit Secure Video (have also found using the RTSP streams rebroadcast from it to be more stable than having multiple sinks connected straight to the camera), and this makes a really good persistent NVR solution that I can also use to monitor remotely without necessarily VPN’ing back into my home network or exposing Frigate thru a separate reverse proxy.
Frigate also has pretty solid OpenVINO support now which means accelerated inference on modern-ish intel cpu/gpus, which is a game changer when you have several cameras.
Great docs, too.
- an intel-based PC (can be a minipc, doesn't need a powerful CPU)
- a USB Coral TPU ($60)
- some wired PoE cameras (from $60 each)
My question: what do people typically use to power the cameras? A single PoE switch, or multiple PoE injectors?
My Arlo Pro 2 cameras are apparently EOL and might stop receiving free cloud services in a couple of months. So this seems like a good time to upgrade to higher resolution cameras.
(The Frigate docs advise against using Wi-Fi cameras, which would otherwise be my preference.)
Chances are, a single switch is more cost effective than multiple injectors. But you also need Ethernet routed throughout your house. One alternative is to have the G.hn (powerline) adapter with PoE. This way, you can be both network and power with one plug without wiring your house.
Edit: not sure why you’re being downvoted
Yes, that's the thing I like about the Arlo system I have now: it has its own wifi network so, even if it's using spectrum, it's probably not affecting my LAN throughput.
> It's actually easier to run ethernet than power, especially outside.
This is true, but the house where I live already has power available everywhere I might need a camera. The thing I don't like about running new cables is the need to drill holes through exterior walls.
If the current setup can't, plugging the cameras and the NVR into a separate switch and none of that traffic will go near the rest of your LAN.
Wifi on the other hand, there's really no (practical) segregation to speak of - the spectrum has limited bandwidth, it doesn't matter if it's a different SSID / wifi network, it'll affect your Wifi!
For me: Wifi is great! But, whenever it's practical, I avoid it... Everything with it is a tradeoff!
A GbE switch with wire-rate transfer guarantee can handle 1Gbps traffic between arbitrary combination of ports. All the camera traffic coming from port 9 to 16 going to NVR on port 7 have no impact to traffic between upstream router on port 26 and your PCs on port 3 and 5. That cannot happen with Wi-Fi because everything is inherently on the same shared port 1(sometimes literally); each 4Mbps incoming is 4Mbps of download speed taken from your laptop. Double if destination is also on Wi-Fi.
This might be fine if there's just few cameras, but it's something to be aware of.
Wifi6 is changing a lot of this, but generally speaking Wifi performance is not optimized for media style traffic. Media traffic does best with low jitter (variance of latency) as this tends to keep buffer sizes low and avoids dropping frames. Wifi is not very good at low jitter, and though Wifi6 is a lot better than previous Wifi standards, it's still much harder to keep jitter low on Wifi than it is on a LAN. On top of that, as the sibling commenter says, even if you have a separate Wifi network, spectrum doesn't segment that neatly. Wireless traffic uses multiplexing methods (there are several and if you're interested, the methods are fascinating [1]) to roughly use the same spectrum. These multiplexing methods obviously need to do more work the more traffic there is on the spectrum.
If you can route your media traffic through LAN do it. Obviously as you say, running new cable is a lot of work so it's understandable why you use Wifi. But LAN is just so much better that if you have the time/money (doing it yourself/hiring someone) to do it, I highly recommend you do.
Unless you're using dedicated APs it is absolutely affecting your other wifi users.
Frigate links some Dahua camera recommendations in their documentation: https://docs.frigate.video/frigate/hardware/
I installed them and they've been rock solid. Low light performance is excellent. The turret form factor is nice and unobtrusive.
I found that one key type is for wifi daughter cards so I was able to save a pcie slot or nvme slot using that in the small form factor host.
I would not use WiFi cameras. Standard RTSP PoE h264 is the way to go.
For PoE, I'd just do whatever is convenient. I've done setups with 2 PoE switches before so I could just run one cable between the front/back and then branch out from there.
Wifi cameras are more for convenience than reliability or dependancy.
It basically doesn't matter at all - I have a mixture of both in my home, multiple PoE switches and multiple PoE injectors for things like cameras, wireless APs etc. Use whatever fits needs/budget/location, you don't have to go nuts buying a single high end PoE switch. There's often good deals to be had on used PoE switches on ebay etc too if really budget conscious.
The only real advantage of going with a single or fewer PoE switches is you have less things to put on a UPS, if you require the system to still work when power goes down. A UPS that can run say 4 cameras, the PoE switch and a system running Frigate for more than a few hours can get pretty expensive too, in my experience - most cheap UPSes are designed to get you enough power to save some files and shutdown a PC in a matter of minutes, not hours.
Cheap intel box with a Coral runs Frigate fantastically, and if a tower build plenty of room for internal storage drives.
I've gone the other route, with a simple power supply that charges an ever-evolving fleet of whatever cheap 12-volt batteries aren't doing anything else, which then feeds DC-DC converters for the various loads. For stuff that's natively 12-volt like my wifi router and cable modem, I just run those directly off the battery rail.
This setup is quiet, efficient, and presently runs the modem, router, service pi, and my RIPE Atlas probe, for somewhere upwards of 20 hours, for something like $150. If I added a 12v-to-48v converter and a small PoE switch feeding a few cameras, it would probably cut the runtime in half, but I could just throw more battery at it for pennies on the watt-hour.
Also learned that go2rtc allows me to have one connection to a camera but then restream it to homebridge, frigate, etc. I use to fumble with ffmpeg to do it but go2rtc is easier.
I have a small form factor Dell desktop refurb with an m.2 coral tpu in the wifi slot. I use the coral for object detection and the Intel GPU for decode acceleration.
I’m curious as to why I haven’t seen NVR software that uses cameras built in object and motion detection and listens for those alarms or events?
Question: how tough is it to integrate a roboflow dataset in the custom model section?
My wife can use the BI app (even though it uses an outdated UI design). I haven't found any Frigate-based UX that would meet that bar (and that bar is not even particularly high with the BI app).
Would love to know if I'm missing a Frigate UX option that is "family friendly".
For anyone that uses it, can you prove since details on value?
[1]You could always mount a cloud drive within Frigate's Docker config to have frigate upload camera footage to a cloud server.
- https://github.com/blakeblackshear/frigate/discussions/7932#... - https://github.com/blakeblackshear/frigate/discussions/7932#... - https://github.com/blakeblackshear/frigate/discussions/7932#... - https://github.com/blakeblackshear/frigate/discussions/7932#...
I have found my frigate+ model to be much more accurate and crazy good even at night. Will be curious how things change when it snows here more often, since I've not submitted any examples of winter at this house yet.
Also an interview with the Frigate author Blake covering the custom models and the project overall: https://youtu.be/04GZBbn_nRE
I always wanted to set off an alarm if my dog tries to pee inside.
If your dog has a consistent space it pees, maybe. You would setup a camera in the room and define an area (basically draw a box around the rug) and further refine with parameters like “Between 9.00 to 17.00 on Monday to Friday, Alert”
Insofar as I’m aware there isn’t any specific dog detection models shipped with the tool. You might be able to find image detection models to add-on easily, but YMMV.