So a cool demo, but sadly useless for anything more.
Nearly 90 percent of comments on posts about LLMs are people talking about how the near future is about to boggle our minds and that general intelligence is near, but all my experiences with these LLMs show they’re capable of making the most basic of mistakes and doing so confidently and that’s just the tip of the iceberg in terms of their problems.
I’m having a hard time buying into the hype that these will be able to competently replace nearly any job anytime soon. They’re useful tools but they all come with a big asterisk of human hand holding.
The big difference here is that these models can scale the work beyond human capability.
Why pay 10000 mechanical turks to extract information from vids, if you can deploy N of these models, and get the work done at a fraction of the time?
Instead you can keep x% of the MTurks to check the vids where the model yields some high uncertainty score, and randomly audit other vids for quality assurance.
There's crazy amounts of potential in these things. Hell, the place I work at has already replaced certain human tasks with LLM-integrated solutions, with extremely good results.
I didn't bother fact-checking every other book because I thought highlighting one mistake would illustrate that the results weren't accurate - which is pretty much expected for anything related to LLMs at this point.
Curious to see how I fared at the task (first vid), I used just over 4 minutes writing down the books with readable titles - and got 36 of them. Seems like there are 56-57 or something like that. So I roughly got two thirds of the books in the video. But that's still 4 minute of pausing and sliding the video for the book titles alone.
The first huge wave of ML/AI automation will involve all the things you don't notice straight away.