VideoPrism: A foundational visual encoder for video understanding
research.google
research.google
present tense :)
Be glad about where you are knowing that you won't have to do as much as others to pay down your sins in this industry.
This is only a problem in a dystopian shit future were we are too stupid to get around this.
Work doesn't make people free.
and yes i do absolutly believe that the pressure will be good for us as a society as we will have to rethink.
Plenty of people spend their time different.
Plenty of people will learn how to do something else if they actually don't have to work.
"We hope VideoPrism paves the way for future breakthroughs at the intersection of AI and video analysis, helping to realize the potential of ViFMs across domains such as scientific discovery, education, and healthcare."
... for those of us internal to Google Research, and possibly large corporate clients when we choose.
Custom internal dataset, no model weights, no API access, just "FYI, this works pretty well when you have our data."
I do appreciate publishing of successful architectures, genuinely, and VP looks like it has some nice ideas, and it is very useful to know that training such a thing is probably not a dead end, so for all that, thanks.
At the end of the day, with large competitors committed to open models/weights, I think momentum is on the side of needing to push out open weights at least, but my guess is GOOG will be the last to transition to this of the big tech cos. I could understand it as a business decision if they were quickly opening this stuff up on their cloud, and competing that way, but they still seem to be in this world where the DeepMind crew continues to push out novel and interesting research projects, and the cloud crew struggles to deliver a competitive LLM at the same time.
I wonder where they'll be in a year. I'd like to hope up to competitive standards so that we've got an additional player, but progress seems slow to me right now.
So yeah, they learned their lesson. You want to complain, complain to those who did not play the game fair and square.
[0] https://arxiv.org/abs/2402.13217 this, not the blog
But consider that others really expect data and full transparency to call research "reproducible", so is not as simple as you make it sound. Plus, grandparent is not asking about the dataset perse, but rather the dataset summary statistics or at least metadata/information about the dataset, a bare minimum in my view.
Being this opaque only messes up research efforts from other institutions.
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3383002/?ref=da...
[2] https://researchguides.uic.edu/reproducibility
[3] https://open-science-training-handbook.github.io/Open-Scienc...
I truly don't understand why that is. Creating the dataset is part of the experiment. It is reproducible because if you follow all the steps in the experiment you will get the same outcome (or, pedantically, a similar outcome, which is true of any experiment unless literally every aspect of the experiment is the same). If you create a dataset of 36M video clips with high-quality manually labelled captions, 170M (video, ASR transcript) pairs from 44.6M YouTube videos and 71.5M (clip, machine-generated caption) pairs from 36.7M YouTube videos, you will get a similar outcome. If you don't, the experiment is not reproducible.
In fact, fully replicating the dataset is more conducive to proving the effectiveness of a given method to produce a specific outcome, as noted in your first link:
>The standard of reproducibility calls for the data and the computer code used to analyze the data be made available to others. This standard falls short of full replication because the same data are analyzed again, rather than analyzing independently collected data.
Ultimately, the question trying to be answered by the authors is not "does x code, when applied to y dataset, create z outcome?" but rather "how is z outcome created?" To prove that following the sequence of steps outlined in the article is sufficient to produce the claimed outcome it is required to recreate the dataset in any event. Releasing the dataset would help in that regard only inasmuch as it would help other people create their own datasets by way of example. If you think about non-coded experiments, it is clear to see that this is the case. Nobody would ask for someone else's reactants and products that they got at the end of an experiment. They would just ask for the procedure to generate the product. And in fact you will find that often times these procedures are not as exact as you might expect. For example it may say something like "this reactant was heated to x degrees" without mention of how fast to heat it up or how exactly that was done. And it isn't necessary because the goal is not "reproducible" as in a reproducible build artifact where every bit is the same but "reproducible" as in "the experiment once replicated will have the same outcome."
I will try to come back to this paper in a year and see how many citations it has and to what extent it was useful to others. Ie. will it be written something like "we follow the architecture outlined in [1]" or will it only be mentioned as previous work? If it is the former it is hard to say that it isn't considered publishing research if others are using it for their own research.
I don't really care what the definition of "reproducible" is. I have always understood it to mean in the context of experiments as “the extent to which consistent results are obtained when produced repeatedly.” When people talk about an experiment not being reproducible or of the "reproducibility crisis" they are using my definition.
Maybe, but the question asked by the people in the field, and even general public, is different: we're asking, "is this even real, or are they exaggerating or plain bullshitting us?". It's a reasonable question to ask when the results seem groundbreaking, and there's lots of money or reputation at stake.
Consider if they came out and said, we've invented machine converting electricity to mechanical motion with near-zero energy loss at room temperature. The design is complex, and uses exotic materials no one outside few megacorps could reasonably afford to get, and we just plan to use it to sell mechanical work as a service, so you're never gonna see how the thing is built (unless you're really important and sign an NDA) - but trust us, it works. Wouldn't some questions about reproducibility be warranted in such a case?
1. Download the datasets and train a small version to get a feel for it.
2. Download the models and deploy them to use it and get a feel for it.
3. Talk to it via a paid API
Why do I want to do that? I'd like to know how capable this architecture is, and get a feel for how capable it could be with different data / scale up / fine tuning, etc.
We know a better way to do this ("this" being "the development of foundational technology for the next century of human civilization"), so of course it's frustrating to see the way it's actually being done.