Turning petabytes of raw video data into a high-quality ML dataset
medium.com
medium.com
> You have 10 cameras recording footage 24/7 at 30 FPS, 1920x1080 resolution. They’re all configured to send video clips back to the cloud in 10-minute intervals. A standard compression setup results in ~1 TB of video stored per day or 27 million individual frames .
Incorrect, I generate about 130MB files for 10 minutes segments on a camera of exactly those specs. Given that there are 1440 minutes in a day, that results in about 19 GB of data per day. The author assumes that there is always a unique frame but that's not true.
It depends on how much movement there is in the frame and the GOP interval.
H264 compression will not generate new frames if nothing has changed. There is an interval called the GOP interval which enforces that say every 2 seconds a keyframe is sent, meaning a complete raw frame is sent. This is to ensure that new viewers can catch on waiting max the GOP interval.
I'm actually the author of the post as well. Two things: 1. I stated 10 cameras, not 1 so according to your calculations that would mean 190 GB per day instead of 19 GB. 2. I didn't make the assumption that there is always a unique frame but rather got this estimate by trying to save videos using OpenCV across various types of cameras. The variable I didn't change was the environment I took these videos in, which tended to be high activity ones where many frames did have some motion in them. 1 TB might be an overshoot sometimes, but the point of adding it into the article was to highlight the order of magnitude for a setup, that all things considered is pretty small. Another thing is that in the context of computer vision training data, the number of frames is something we care about more than the amount of space they take up which is why I also state 27 million frames.
Thanks for pointing it out though. Might be better not to even mention video file size given the high variability.