mvoodarla

Original dev here. Unable to replicate this on my end, try refreshing?

Original dev here. Unable to replicate this on my end, try refreshing?

Original dev here. I tend to agree for this particular demo video as I'm reading a book and I don't blink in the original.

The model tries to copy the blinks of the original video so it's possible that in other conditions, you'd notice less of this.

Fun to see this feedback though, definitely something worth improving :)

Original dev here. That's right, NVIDIA has a version available which we reference in our blog.

https://www.sievedata.com/blog/eye-contact-correction-gaze-c...

Newer models have come out that allow the same thing to be done and control even more than the eyes.

See here: https://github.com/KwaiVGI/LivePortrait/blob/main/assets/doc...

For web-conferencing, local use is great so NVIDIA's tools are what we recommend in that case.

This is an interesting idea. We are a little farther off from being able to do this but agree it would look really cool.

Original dev here. Agree this video looks like uncanny valley but it's likely because the lighting of the original video is off + I baggy eyes (I was sleep deprived).

Would recommend trying it on other videos, it is surprisingly good. Although there definitely are areas to improve.

Congrats on the launch! I work at Sieve (https://www.sievedata.com/). We do a bunch of stuff with AI and video. Excited to check this out :)

Hey folks! We just built a cost-effective, lightweight way to generate audiovisual summaries for videos.

* Process videos up to 12x faster than realtime * Costs <$0.01 / min of video * Combines visual and audial components

The goal here is not to build a single E2E model but something that could actually be used in production while preserving relatively high quality.

You can try it out yourself here: https://www.sievedata.com/functions/sieve/describe How we built it: https://www.sievedata.com/blog/describe-video-summary-beta-l... The code: https://github.com/sieve-community/describe

What was the orchestration system used for? Was it in the case where there were many models that needed to be run one after another. I know it's a huge problem in video processing to be able to increase speed a ton. My company Sieve (see profile) is building infrastructure specifically for running ML models on video which is why I'm curious.

Yeah sorry about that! We'll add that soon.

This looks really cool. Are there ways to orchestrate jobs? Like having one notebooks output trigger another based on some logic? I'm imagining running a bunch of different deep learning models on separate notebooks, or running the same model on different chunks of a piece of data in parallel.

It's a good point. While Microsoft and Google offer video processing / analysis, none of them are API-first in the same way we are.

1. We process videos, and can store their metadata / embeddings in perpetuity. This means a simple query API, don't worry about storing a DB of metadata yourself.

2. "Reverse Image Search" - Say a user cares about something specific, like a blue box that's toppled over. Using high-dimensional representations of certain frames in video, we can surface every time this scenario shows up in the future, based on a user having marked it as "interesting" in the past.

3. Our "killer" app is the video processing infra that's super modular. We give this flexibility to the full-stack dev to plug in their own models, or use ours.

A final thing to note is that we're going after a completely different set of companies, specifically companies focused on building software for a specific industry (i.e. https://www.equipmentshare.com/) that eventually want to add video analytics as an offering. It's also unclear who Azure or GCP are targeting though it seems like it's for general media content.

Thanks!

Most use-cases have limitations in bandwidth, so not all data is streamed up to the cloud. Typically, people will run simple motion detectors on the physical camera device itself, and save content in the cloud that they think might be valuable. Our service can then be used to make that content searchable, past simple motion detection.

There are other use-cases where people store tons of data in the cloud already. Media content is an example of this. In this case, companies might want to analyze all the raw data for which we then charge only for intervals of video where there's real action. If the video is static, we won't charge the full per minute pricing.

Thanks for the thoughtful questions!

1. People have definitely tried this in the past in terms of having similar tech, but I don't think anyone has sanely approached it at the same angle. We're seeing so many more companies like EquipmentShare (https://www.equipmentshare.com/), UpKeep (https://www.upkeep.com/), Spot AI (https://www.spot.ai/) and others which are building core software for industries without necessarily having tons of video expertise in-house. Also other video-first companies like Gong (https://www.gong.io/), Mux (https://mux.com/), and Loom (https://www.loom.com/) are building platforms that might benefit from visual analysis at scale. Things like serverless functions were also less cool before.

2. There's a bunch of ways to search using Sieve. Some are fixed variables, others are visual similarity based. See here: https://sievedata.com/faq

3. Our target is the full stack developer within companies building cloud-first software for specific industries. Easy-to-use APIs are key because of this. They are the ones with the expertise to package it into a product that really works for the end user, and we're the ones with computer-vision and video expertise.

Thanks for the comment! Audio is definitely an interesting angle. We haven't spoken to a lot of folks yet that want that feature, but I'd guess some productivity tools that let you rewatch Zoom calls, baby monitoring, media playback, or a few other things like that might mind audio useful. We'll explore it for sure.

We currently process on the per-frame basis but have the optionality to increase the window size of 1, to any arbitrary size like 300 frames. This is what allows us to currently detect "actions" for some of our customers. Of course, it's not the same as a sliding window but it's something we're considering.

Thanks! The API hub seems interesting. Want to send me an email (see bio)?

> With your massive parallel infra you're still processing 27 millions frames, right?

No, it's not. We first run a cheap filter like a motion detector on all of the video, which is inexpensive. We then stack other, more expensive filters on top of this depending on the use-case and eventually run the most expensive metadata-generating models at the end. We also don't do this on every single frame and can interpolate information using surrounding frames. Our parallel infra speeds this up further.

> That's exactly transfer learning is for. I am saying to do it with because you automatically get a pretrained model which can do much more than that. Imagine doing a search "a person wearing red cap and yellow hand bag walking toward the exit" or "A person wearing a shirt with mark written on it". Can your system do it right now?

The issue is that there are very few text-image pair datasets out there, and building a good one is difficult. We constantly use transfer learning in-house when working with different customer data and typical classifier / detector models but haven't yet had success doing so with CLIP. Our system can't semantically search through video just yet but we're exploring the most feasible ways for doing this still. There's some interesting work on this which we've been reading recently:

https://ddkang.github.io/papers/2022/tasti-paper.pdf

https://vcg.ece.ucr.edu/sites/g/files/rcwecm2661/files/2021-...

This is totally understandable. We're thinking of the best way to satisfy personal users given we've heard this security use case many times.

Sounds like another use-case that fits! We're currently only offering this via hosted API so it won't work offline. I'd be interested in understanding the specific use-case further.

If you have any connects there, feel free to reach out to the email in my bio :)

Most of the time, we work on processing video data after it's been hosted somewhere for us to seek by frame or download. We don't yet support video streams but have heard multiple people bring it up.

Our infrastructure is really good at filtering for parts of video where things are happening ("interesting") and being able to parallelize video processing by cutting it up into multiple pieces, running more granular models on those parts, and smoothly interpolating metadata using surrounding frames. You should check out the processing section of our FAQ page (linked in original comment) for a more detailed explanation of how our processing works.

If you're worried about just streaming seamlessly, Mux has a great solution: https://mux.com/live.

Also, for detecting things in e-sports feeds, feel free to reach out to the email in my bio!

You're right. thanks for the feedback, we'll add this soon!

Will definitely try this out on personal time. Sounds like a fun weekend project :)

1) Wow I've never heard that one before but that's so interesting. I'm just thinking of my parents who love gardening and the insight they might get from that. If you want to leverage our platform for a personal project like that, let us know! Might need to add a few new features to make that seamless.

2) This one we currently support + work with. Thanks for bringing it up again :)

Well, this depends.

If you're looking at some generic things that are similar to what CLIP was trained on, this would work. Say you're interested in specific physical security metrics, or monitoring defective parts, or specific things about traffic, etc. CLIP might just say "people walking" or "car in intersection" or "part on conveyer belt" which isn't meaningful enough if all your images are exactly that, but with other small differences.

Another important aspect of this is the amount of frames you need to process. Running CLIP even on 27 million frames (1 day of footage) is super expensive. We've built some infra that makes processing video efficient (forms of parallelization + filtering), without you having to think about it.

Interesting application! We're working with a few people that monitor public streets in different ways so I can see this being useful to you as well! Feel free to reach out to me at mokshith@sievedata.com and we can chat more :)

Nest already provides some smart people / motion detection. I could definitely see personal home security use-cases if you're building something DIY. Curious as to why you might prefer a system like this over what Nest provides. Is it the more granular metadata?

Thanks! We're don't currently have plans to make it a real-time but if there are enough use-cases here, we'll definitely think more about it. Other than security, where do you see this being useful?

I'm thinking possibly a live trend analysis / alert sort of thing might be useful if say we detect too much motion, too many people, or some other variable that we determine to be "out of wack".

Thanks for sharing your use-case Tom!

Love that you bring up delivery bots. They're the moving example, but you can also think of a parallel application where you have stationary cameras monitoring something of interest (worker safety, defective parts, security, etc).

Interested in hearing where else people are struggling with videos.

Hey HN! I’m one of the creators of Sieve (https://sievedata.com/), and we’re happy to be sharing this with all of you!

Sieve is an API that helps you store, process, and automatically search your video data–instantly and efficiently. Just think 10 cameras recording footage at 30 FPS, 24/7. That would be 27 million frames generated in a single day. The videos might be searchable by timestamp, but finding moments of interest is like searching for a needle in a haystack.

We built this visual demo (https://sievedata.com/app/query?api_key=AIzaSyAfKwf0tuuNOHbY...) a little while back which we’d love to get feedback on. It’s ~24 hours of security footage that our API processed in <10 mins and has simple querying and export functionality enabled. The demo is a security use-case but our platform supports a wider variety of use-cases and metadata. We’re working with a few early customers to explore which ones to dive deeper into (happy to hear ideas on this front from the HN community as well!).

To try it on your videos: https://github.com/Sieve-Data/automatic-video-processing

Visual dashboard walkthrough: https://youtu.be/_uyjp_HGZl4

General FAQ: https://sievedata.com/faq

Thanks for the comment!

I'm actually the author of the post as well. Two things: 1. I stated 10 cameras, not 1 so according to your calculations that would mean 190 GB per day instead of 19 GB. 2. I didn't make the assumption that there is always a unique frame but rather got this estimate by trying to save videos using OpenCV across various types of cameras. The variable I didn't change was the environment I took these videos in, which tended to be high activity ones where many frames did have some motion in them. 1 TB might be an overshoot sometimes, but the point of adding it into the article was to highlight the order of magnitude for a setup, that all things considered is pretty small. Another thing is that in the context of computer vision training data, the number of frames is something we care about more than the amount of space they take up which is why I also state 27 million frames.

Thanks for pointing it out though. Might be better not to even mention video file size given the high variability.