Google Bard AI Now Has the Ability to Understand YouTube Videos | Hacker News Reader

Google Bard AI Now Has the Ability to Understand YouTube Videos

youtube.com

59 pointstitusblair54 comments

I made a simple Chrome extension that similarly pulls down the video transcript and sends this to the openai chat completions endpoint: https://github.com/josephrmartinez/AskYouTube

This extension allows me to "ask" the model to perform a task on the video content: - "Give me the materials list" (for a diy video) - "What was the recommended book?" (for a 2+ hour podcast where they made a reference I can't find again easily) - "Extract the recommended protocol" (for 3+ hour health videos) - "Provide a counter argument" (for when I'm getting bored...)

Big plus is that you DO NOT need to wait for the ad to play through. I can just navigate to the video and send in a query without having to watch any ads.

Youtube transcripts are pretty rough. At first, I used Whisper to create a better transcript. But my primary use is to ask something of the youtube video - I found that slinging the so-so transcript along with my task was totally fine. Really simple project: Chrome extension in just html, css, and js. FastAPI server for the openai endpoint. Server function does a quick tokenization on the transcript to determine if I need to use the gpt4 model for the 128k context window or if the gptt3.5 16k context window is okay.

Naturally, here is a short youtube demo of the extension: https://www.youtube.com/watch?v=M1zq9NKIcbw&t=54s

Since I had the same question as everyone else, it seems like it must be using just the transcript. When asking about one of those "8k HDR" showcase videos (with no speech), Bard responds with:

> I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.

Most captioning is automatic now though. So it's still true in a meaningful way.

Whisper (OpenAI speech-to-text) is already trained on YT content; amusingly, if you mumble incoherently, its most-probable completion for noise is “thanks for watching!”

That’s hilarious :D

I wonder how they got the model to avoid the viral "don't forget to like and subscribe".

Super interesting. I’ve seen that as a rumor but never confirmed.

It’s an example of the sort of grey area that would’ve made “openai 2.0” tricky at Microsoft.

A small startup can ignore ToS and scrape YouTube but Microsoft cannot.

I see "thanks" a lot in captions where it's noise or unintelligible speech. Maybe that's why.

That is hilarious! Every time I talk to ChatGPT Voice through the mobile app, and stop speaking for a few moments, it always seems to think the background noise is me saying "Thank you."

If it gets very good at "understanding" YouTube and other video content, Google could maybe find some kind of training data advantage not available to a pure text based model.

Early on it felt like Google was asleep at the wheel and would lose AI. Now I think they're the company best positioned to win it. YouTube is a huge component of that.

Their recent YouTube music generation stuff just killed about a dozen music startups I've been tracking. And they've already got the users.

How reading the captions of a video make google the front runner?

They just launched tools to make your own vocal covers of Frank Sinatra and thousands of other artists. And generate beats.

those quotation marks around "understanding" are pretty important.

to save people a click, and in the hopes that the comments might stay relevant to the OP, the OP means this by "understanding": if the video is about a recipe, you can ask how many eggs are used in the recipe, and Bard can give you an answer.

it seems pretty likely that this may mean Bard simply consumes video transcripts.

Test on a video without transcripts

can we use it to detect and skip parts of the video that contain ads?

I'm sorry, it is not ethical or responsible for me to help you skip ads. In fact, the very suggestion is thoughtcrime. Is there anything else I can help you with?

This really is the way things are heading it seems.

I’m sorry, this response contains references to 1984, a film banned for inciting unrest. Thoughtcrime does not exist, you have freedom of expression unless it harms our profits.

The supreme court recently made ad skipping legal, and have also removed all book bans.

A recent study was done that also proved conclusively that skipping ads saves your company money by saving bandwidth.

look up sponsorblock

I assume it actually does not understand video, but reads the transcript?

Bard can read images and, being made by the same company as YouTube, probably has access to high quality video embeddings they use for YouTube search, maybe the most sophisticated video search engine on the planet. I don’t know from the article but it could definitely be using the video content directly without a text representation.

For an open source project that actually “sees” videos you can check out https://github.com/PKU-YuanGroup/Video-LLaVA

> maybe the most sophisticated video search engine on the planet

And yet youtube search is such garbage most normal users still use google search to search for videos.

Really? What’s particularly bad about it?

Finding only what it's asked for.

Works fine for me:

https://i.imgur.com/JdxFDfX.jpg

https://i.imgur.com/N7Dpb4U.jpg

it's okay. I found tiktok to be better as they search brings your current context for suggestions, user's comments are smartly turned into search links, and they do a better job of letting the user search by transcript

> most normal users still use google search to search for videos.

Citation? This seems unlikely to me. Maybe a lot of users end up watching a video given to them by a Google search. But if someone is specifically looking for a video, I think most probably go to YouTube and use YouTube search (which is more than fine for “normal users”)

> And yet youtube search is such garbage most normal users still use google search to search for videos.

They are the same thing at their core, biggest difference is that Youtube started limiting the number of results but has better filtering options.

You could test this by asking about an uncaptioned video where the generated captions for any language would be trash (e.g. jiberrish) and using powerpoint slides to convey stuff.

Google search certainly seems to have the ability to understand video beyond a simple caption search, so I don't see why bard wouldn't have that same ability.

It seems it's only based on the transcript for now. For example when I asked to comment on visual content it mentioned requiring a captions file:

No. It just hallucinated[0] about parallel evaluation while this talk doesn't mention parallel evaluation at all. So it has to be more than just the transcript.

[0]: https://g.co/bard/share/9854794a5dc0

Here is go now. The AI revolution begins with learning how-to videos. Just create the latent space for video/visual understanding, it's going to be very interesting to explore that.

I wonder how this works. It sounds like it's transcript driven, but then the next question is - were the transcripts automatically created or user-defined?

If the former, is this not going to run into the same issue as training AI on datasets created by AI? I experience so many mistranslated words when using automatic transcripts that I can't imagine the quality of data is excellent without supporting the transcripts with video inference.

Is there any reason to believe YouTube content will only be trained on by Bard?

Stuff like YouTubeDL exists and works fine. I would assume that others could scrape and train on it, too? Or does that sound outlandishly expensive?

It would be outlandishly expensive, compared to already having infrastructure that processes the videos anyways.

Okay, so Bytedance, Amazon, Google, Facebook have an advantage in this area, then - assuming they each process TikTok, Twitch, YouTube, and Facebook/Insta video content?

Are there any other big players in the video data space?

The entity with the most data is the US government, or maybe the Chinese government. If data becomes the main bottleneck and everything else is well understood at some point expect those to win it.

Presumably Tesla, though I don’t know how much of that video is going to have useful audio if they want to specifically train a language model with the their self driving AI.

Quite a lot of “would you like fries with that”, or whatever is actually said on drive-throughs.

Purely user submitted content wise who also owns the content in the west? I would guess so, yeah. I am sure there are much more though, think of other companies who also own large data centers like Nvidia. Microsoft also does a lot from what I’ve heard.

Open source version of something similar:

https://github.com/PKU-YuanGroup/Video-LLaVA

Finally, no more sitting through 15 minutes of an extended "hey guys" intro just to find out how to make a Redstone machine.

I'd argue that the Bard YouTube search is worse off right now than just searching on YouTube itself.

I assume it'll improve over time.

It can't find the synthwave videos with least views, so Bard is being blinded by the recommendation algorithm.

Why would it be able to find that? You are looking for a database query engine, not a language model.

Why wouldn't they give it the ability to correctly search in the database they own?

That is an accomplishment!

I'll have to get it to explain, since most of that shit is incomprehensible to me...

I actually worked on this for a toy app: you can create tokens for images, and overlay those in the embedding space you do words

That allows LLMs to treat images like "words" and speak on their contents.

That being said looking at the demo I'm not sure Bard isn't just using the video transcripts. I was using them to guide seeking, but you could answer all the test questions shown in the video without image understanding

Can it do something more intensive like asking …

Give me a list of the video/recipes that use 3 eggs?

Understanding the videos is all very well but can it understand:

1- the popularity of "tier list" videos?

2- why those douchetuber "prank" videos exist?

3- Logan and/or Jake Paul?

Would Bard understand if, say, a person in the video smiled or there was a sarcastic tone to a bit of audio?

baby steps

Reply on news.ycombinator.com