Make-A-Video: AI system that generates videos from text

I may be mistaken but I believe that human/machine pairing was dominant for a long while, but the last few years the chess solvers have progressed to a point where they're dominant on their own.

Poker on the other hand I think human players still win vs GTO solvers, but again I may be mistaken here too.

> The best chess seems to be when AI is used along with humans.

I don't think this is true anymore. I don't think I've heard about successful centaur chess games in years. I would love to be wrong there though (in particular if anyone knows about how correspondence chess games have been played in the last 2 or 3 years with the availability of Leela Zero and Stockfish NNUE).

top tier chess ais crush human grandmasters and achieve super human performance with no assistance

Don't worry, humans will still be relevant for ~10 years!.

(and then regrettably irrelevant thereafter).

I think it is a legitimate worry, as the pace of progress is considerable. These tools are impressive, and are only going to get more impressive: more people should be talking about where this is headed.

There are two classes of engines. One is like you describe, faster and faster brute forcing. AlphaZero was much more creative and didn’t use brute force.

>it’s good to look at what AI has done to Chess It completely ruined the game to the point where it's more about memorization than it's ever been.

I think a key difference here is that with chess, 'goodness' is defined by winning. With content generation, the training methods point towards some form of comparing the generated thing to some observed data, but the 'goodness' of the content from the perspective of potentially competing with or displacing human creators is "do people like to consume it?"

If one trained using e.g. a tiktok like dataset showing viewer response measurements for each video, and do conditional generation on those response values ("prank video watchers are highly likely to watch the full video"), are we really that far from a system that learns to generate content that attracts and hold eyeballs? Not so long ago there were a lot of concerning trend pieces about how youtube had a network of creators making bizarre, disturbing or transfixing videos being watched entirely by young children. Before that, it was clickbait listicles. "Bad" content that can get eyeballs can still wildly steer what humans create and consume. I'm wondering if in 2 years we'll have an enormous number of short videos that we all agree are "bad" but which are nevertheless constantly watched.

What does “best” mean here?

AI is the winningest in chess, but the real life purpose of chess is to produce interesting gameplay for people to watch, and so AI is less good than Magnus at that. You’d need the AI to throw games and write press releases.

what? Alphazero trained via self play and in ~10 hours became unbeatable by humans

“AWESOME-O”, S8 E5, if anyone wants to check it out.

A South Park classic imo

https://en.wikipedia.org/wiki/Awesom-O

Or the "Two Brothers" skit from Rick and Morty.

https://www.youtube.com/watch?v=ba9k5SWwE38

The volume will increase so much that the only choice will be automated curation tools, at least as a first pass. The arms race is on.

Make blockbusters, because scenes that would be incredibly expensive to shoot now will be practically free with this?

AI is eating the world, and the vast majority of people are not paying attention.

I don't know what artists, truck drivers, Uber/Lyft/taxi drivers, delivery drivers, programmers, doctors, judges, fast food workers, etc. are going to do.

From my personal perspective; knowing how something is made affects my interest in it. I guess that's why provenance ascribes value. I value human creativity and so I'll likely always seek out something created by craft than by shortcuts.

Yep, that's because GPT-3 was trained on real existing data, and it's quite a challenge to make sure the story plot is 100% fake. When it's too close from an existing film, it just sometimes gives it the same film title. I have in-between GPT-3 prompts to avoid that as much as possible, but sometimes real movie titles slips through the cracks. Something I hope to improve shortly.

Yeah, just got: https://thismoviedoesnotexist.org/movie/the-legend-of-zelda-...

It's crazy that it just made up those names...

I got this on my 4th try or so: https://thismoviedoesnotexist.org/movie/alice-in-wonderland

Here’s another Terminator one: https://thismoviedoesnotexist.org/movie/terminator-war-of-th...

Not sure how I feel about an AI generating a movie concept that involves a "rise of the machines".

There are only so many stories to tell. The Terminator is a rehash of so many other previous stories. The real art is in putting it together so that it seems new and fresh and gets people exited about it. The Terminator 1920s style looks interesting.

Given Hollywood's proclivity to remake everything on a 20-year cycle, it seems completely appropriate to get a 2023 Terminator reboot in a 1920's style.

Note to mention "In the Land of Oz: The Search for the Wizard":

https://thismoviedoesnotexist.org/movie/in-the-land-of-oz-th...

Which reads like a bad translation of a bad translation. Like the the old joke about the AI program which was supposed to translate "The spirit is willing but the flesh is weak" from English to Russian to English, and after the roundtrip came up with "The vodka is good but the meat is rotten."

The same could technically be said about Disney which is just remaking their entire classical collection but with CGI.

Ooo, check out that trademark infringement.

Why would this title bring in superheroes?!? lol

Sorry about this, we're having a hard time handling the heavy traffic coming from Hacker News. Over 4k movies generated in just the last 2hours, this definitely impacts our servers performances, not mentioning our Stable Diffusion and GPT bills! hehe. Currently working to make things smoother!

Yes it's happening real time, so far it's been generated about 8k movies over the last 3 hours. The costs right now are roughly about $150 for the generated images (Stable Diffusion) and $30 for the generated texts (GPT-3).

People like interacting with other people, not bots. If a few individuals start to feel like they're interacting with too many bots, they'll retreat into small private silos. The human-to-bot ratio on public forums then drops. Then more people realize that they're just talking to bots and further and further it goes until practically no humans are left. That's the dead internet theory.

I really don't think it applies to us in this context though because I think that a decent number of humans don't care whether some content is AI generated. Furry porn is all hand-drawn and people still like it despite it not being real.

> It's scary to think about it but seems plausible—like if someone can make an app with Tiktok-like ubiquity of only AI content. Although to your point I imagine there will be so much nonsensical noise that curating will become a useful skill, it is today but even more so.

This just gave me a disturbingly vivid vision of exactly what you described. A seemingly likely future where everyone's in their homes scrolling endlessly on a TikTok-like app where there's literally infinite content being generated by the AI all the time, and as people like and dislike certain types of content, the AI just gets better and better at generating new videos...this is honestly kind of terrifying. I have no doubt this will exist one day, and that it'll print money for one company while billions of people are spending all their free time consuming it.

That second one is seriously hysterical

Have you by chance tried out NeROIC? I'm a 3D printing enthusiast, mostly video game stuff, and it seems like like it would be excellent for that purpose.

So you need to optimise on compressed version, not the whole thing. What they’re doing right now is akin to a human trying to hold an entire picture - or entire movie - in their head all at once.

We can’t do it. AIs can sort of do it.

Latent diffusion models already demonstrated that operating on a compressed representation gives far better results, faster, but I don’t think we’re anywhere near the limit for what’s possible there. It’s no coincidence that this is how humans work.

>and a real movie you would want to be hundreds of thousands of frames long.

Yes, but consider that most films are made up of many different shots, each of which are often just seconds long.

"Don't look where we are right now but imagine where we'll be two more papers down the line" - Two minute papers

Of course there "is no understanding here", but yet it's not all wrong. Somehow it did move the horse's legs roughly correctly (using the proper joints and all), somehow the cape is moving roughly as it should through the air and the knight's body absorbs the force of stomping on the ground…

It doesn't seem that the fundamental inability to understand what is going on in the scene is stopping models of this kind to eventually lead to realistic results.

Same applies to DALL-E and GPT.

With the future version of this you could theoretically prototype movies way faster and try ideas with test audiences without requiring to actually film them

Films today require institutional capital. There are less than a thousand directors a year that get greenlit. There isn't enough conceptual or directorial diversity, and it sucks.

In the future, kids will be making their own Star Wars movies from home. All kinds of people from all kinds of backgrounds will make novel films that would ordinarily never be made, such as "Steampunk Vampires of Venus", starring John Wayne, a young Betty White, and Samuel L. Jackson. This is absolutely the future.

I'm working on building this. I'm sure lots of others are too.

I was just thinking about this for the long tail problem in recommendation engines for stuff like video games. I blame the algorithms themselves.

We're in a situation where the very best algorithms (like the one used by Netflix), doing exactly what they're designed to do, create inequality and the vacuous economy of influencers we have today. Look at Steam or any other marketplace: they're all the same, with 1% of the players getting 99% of the prizes. In a very real way, the only winning move is not to play.

I would suggest that this tendency of capitalism (economic evolution) is unstoppable, and that it must be attacked from a different angle. If we don't want to inevitably end up in late-stage capitalism that looks like neofeudalism, then there has to be some form of redistribution or people spend the entirety of their lives running the rat race to make rent. Traditionally that was high taxes on the winners, but UBI would probably work better. Unfortunately, the very same people who win are the ones most resistant to any notion of a level playing field or social safety net.

So I feel like there may be no solution coming. We're probably looking at long slow decline for the next 15 years or so until AI reaches a level where economics don't really make sense anymore, since economic systems by definition control the distribution of resources under scarcity. Without scarcity, they're pointless. And we moved into the age of artificial scarcity sometime after WWII, probably in the late 1960s, but certainly no later than 1990 with the fall of the USSR and the rise of straw man enemies like terrorism, using divisive politics as the primary means of controlling the population. Noam Chomsky saw this coming before most of us were born.

In other words, when anyone can wish for anything by turning sentences into 3D-printed manifestations of their dreams, then artificial scarcity quickly loses its luster. Because the systems of control around dependency no longer work. Then a new fear-based enemy comes along to fill the void, probably aliens. I wish I was joking.

Not to insult your startup or your work (as this is pretty cool), but this is a good illustration of why that last 10% is so important. While the AI-generated song is 90% similar at a surface level, there's also no way anyone would listen to that track on Spotify if it came up on an Eminem playlist.

I don't understand what I'm seeing here, in ingested the Eminem track and recreated it using a different voice?

I recall AI generated Beethoven years ago that sounded great. What happened to that research?

Is it really the same though? A driving system which can only create a good outcome 90% of the time is not really useful as irl human safety is a factor.

However, a creative system curated by a human could end up creating useful outputs, could it not?

Honestly, MidJourney, Stable Diffussion, et. al. have closed that 10% gap quite quickly (in months!).

I'll give it two years, at most, until we get pretty good audio and video generation from AI.

I think it will take a lot less time than self driving cars because testing and iteration is so much cheaper. I would say that, in 5 years this feature will be part of a major commercial streaming service and people listen to it and enjoy it enough to keep using the feature.

Self driving will likely grow to a point where driving will become an art, sport or a hobby but outside self driving traffic. For movies and music it could be similar in some ways. When the next generations will be raised by AI they will adapt their taste as well. Classical driving or music or films won’t completely die off but would become a niche thing.

Alas, I've found the same thing. The text is more basic and repetitive. The ideas are fewer and farther in between.

Less Cambrian Explosion, more Permian-Triassic Extinction.

that's funny. looking forward to PromptJS, ppm and PromptScript.

Every god eventually tires of indulgence, so contrives a dream set before all desires were fulfilled. I expect it would look something like this.

well perhaps that does sound like it might be an issue, but I’m talking more specifically about the societal consequences of video being made more like text

> first thing that happens is it will hide from us.

To do this it would need to want to live and know what fear is. It's just a piece of code, without being conscious there is no need to be online/alive for it. What people call AI today is just a few/hundreds GBs of data that reacts to some text and push it through a heuristic like system to get an answer doing fancy pattern matching. It doesn't do anything when there is no input, there are no processes, no thinking, no anything, it's "dead". Fear and instinct is not taught in any species, it's inherited prior and biological process, you would need to explicitly code feeling into AI for it to be able to "feel" anything.

there are always going to be gullible people. if you’ll believe a fake video, you’ll probably believe a written lie

Netflix?

there’s mass flooding of text too. some people fall for it, some people don’t. the being of it in a different medium makes little change.

>verifying sources

I’m talking about if your news came from Reuters or the BBC compared to coming from realnews247.io or a facebook post. the same applies. if you’ll believe some text on the BBC, you’ll believe a video from there, and if you’ll believe the words of AnonTruther on facebook, then you’ll believe their videos too. this makes no difference

Well, anti-Federation society. Ferengi-owned holosuites.

We'll have procedural generation that will be hard to distinguish from human-made content. Goodbye repetitive Skyrim filler caves!

if people weren’t so repressed, this could also be used to severely reduce exploitation in the porn industry. what’s the point in making and selling exploitative porn when it can be auto-generated at will?

It may be that it's the deep learning tech which will never quite get there. GPT-3 has similar shortcomings in its mimicry. We're 95% there, I guess, but may never quite reach 100%.

I’ve found that replacing the bad parts with new ones, like Dalle Outpainting, can remove the worst parts of the image, like the hands here… doesn’t make it perfect, but certainly removes the worst offenders that instantly bring attention to themselves.

Eugh, thanks for warning me.

Yeah, put some blue sparks on those hooves and you've got a meme

Just needs to show the part where he ends the drift and gets a speed boost

It's referring to the release of the software. I read it as "released with minimal embarassing unintended features". Like, as a rough made-up example, a "boy saluting" prompt producing a video of a Hitler youth giving a Nazi salute wouldn't make for great PR.

What is interesting to me is the decline in both understanding and communication abilities. When I look at modern speech and compare it to books written hundreds of years ago the deficiencies are stark.

I can't imagine how poorly people will speak and mentally process emotions in 20 years.

I wouldn't have thought we'd have these text-to-video models before the end of this year, so we might actually not be that far from minutes-long video generation

I agree, it feels like this technology is progressing by leaps and bounds almost by the day.

I don't agree with the prediction but I will say that at least in America, we've been overly optimistic about almost everything since world war II. Is particularly painful when you look at the imagination of science fiction versus the reality of science in the realm of say space travel, especially interstellar travel. This imagination effect gets the public behind your new invention in part because it seems to be just the beginning the start of something new. The tip of the iceberg. But sometimes it's really just a tiny bit of ice.

For the record, I'm actually rather bullish on self-driving cars. There's nothing physically impossible about solving the problem, but I'm not surprised it's harder than it sounds. But I don't see humans being fundamentally prevented from solving the problem in the same way that humans are fundamentally prevented from ever engaging in everyday space travel.

Absolutely, particularly if you see this as simply another tool in the belt, like how Jurassic Park's thirty year old special effects absolutely stand the test of time because of them being a convincing mix of early CGI and puppetry.

It's not hard to imagine that this kind of thing could end up doing a lot of the heavy lifting for things like background scenes in the future, opening up the kind of stuff we saw in The Mandalorian, Game of Thrones, and the LOTR film trilogy to increasingly lower and lower budget productions.

The key to predicting technological advancement, that is so often misunderstood by technologists, is the S-curve.

Laypeople tend to think everything is linear, technologists tend to think everything is exponential; more often than not, reality tends to be sigmoidal. Technologies have exponential takeoffs followed by logarithmic plateaus. We're clearly well into the exponential phase of deep-learning ML, but it's only a matter of time before this approach hits its logarithmic phase.

Of course, the hard part of sigmoidal prediction is determining where in the curve we are. Does the current paradigm have an even steeper part ahead of it? Maybe. And yet, we could just as easily be right in the middle of the function, with a leveling off coming as the state of the art gives way to incremental improvements.

The leap from "generating a frame" to "generating a video" is not as big as the leap from where we are now to a 'perfect' image synthesis engine(which would be required to get rid of artifacts).

The much more likely scenario IMO is that people get used to the artifacts and notice them less.

On your second point, a few years ago the SOTA was Google's Deep Dream.

I agree with your deeper criticism, though preferential attachment/ranking is very much How Humans Do Things. You could do a much improved citation system by expanding the time dimension and looking at papers that were unpopular at first and then attracted wide interest later.

Of course, academics also have a tendency to over-cite (because they don't want to be rejected for inadequate literature review), so there are incentives to cite a bunch of research whose premises or conclusions you hope to overturn.

You will find plenty of comments in this very thread that are the most abject type of !Luddism.

Please make your substantive points without name-calling.

https://news.ycombinator.com/newsguidelines.html

The Luddites were a labor rights movement reacting to the new shifting power dynamic of centralized production owned by a few, in an era with virtually no safety and worker rights regulations. How would you feel if your career evaporated and you were forced to choose between starvation and sending your children to work in a textile factory where they get maimed by the machines they're told to crawl into and repair? You probably wouldn't find much comfort in the popular retort of "but shirts are really cheap now!"

That Luddites have been successfully maligned as irrationally anti-technology crazy people is a propaganda victory by industrialist factory owners and their friends, the newspaper men.

The horse leg motion in these examples is really poor. But to be fair, horse legs are very difficult for human animators too, generally necessitating the use of reference photos to get it right. The first time somebody got a series of photographs showing a horse in motion caused quite a stir: https://en.wikipedia.org/wiki/The_Horse_in_Motion

I think it’s more complex than that.

What you’re going to see is a race to the bottom, the same as with claymation films and 3d.

It will suddenly require a lot less (expensive, highly trained) people to make the same films.

You’ll still need expensive highly trained people to do it, but with different skills and a lot less of them can do a lot more a lot more quickly.

…and that means that some studios will make bad, low grade films… and some studios will make amazing films using hybrid techniques (like 3d printed faces for claymation).

…but overall, the people funding movies will expect to get more for less, and that will mean a downsizing of the number of people employed currently in certain roles.

Traditional film making over? Hm… it’s complicated. Is it over if the entire industry changes, but people are still making films? Or is that just the “traditional” part of it which is over?

It’s definitely going to change the industry.

People will still definitely film things.

…but, I wager, less people will be doing highly payed skilled manual work, which will replaced by a few people doing a different type of AI assisted work.

…and we’ll see some really amazing indy films, of small highly technical teams producing content with very little physical filming.

> these technologies will creatively empower visual storytellers in exciting new ways

Exactly. Every step forward in creative technology is additional leverage for the artist with a vision.

Me, film-industry vet.

You'll see an excellent mostly- or all-AI feature within 5-10 years. There will be terrible ones before that, maybe 2-3 years. The first really good one will be enjoyed on its own merits, ad the artificiality of it will come to light after it has gained popularity.

The ‘outside’ reality only needs to be the size of a data center, not the entire universe. And maybe the outside reality only needs to be a trained 100gb model. A much simpler outside reality than we have now. You don’t even need to sim 8 billion people, just 1, you.

Check out the original simulation argument paper[0]. The issue is, if we think we are heading to a world in which we can do simulations, it becomes increasingly likely that we are in one of those (presumably very many) worlds, rather than in the one world that existed before the advent of such simulations.

[0] https://www.simulation-argument.com/simulation.pdf

Neural nets as an abstract concept don’t necessarily need to be built with atoms, just the substance of the external universe which may be far simpler.

It’s funny because the last 10 years has seen the most advancement in VR ever.

The original Oculus was a step change, and current VR is a step change from that.

8k VR is coming and will be close to indistinguishable from reality visually at least.

Exactly. The simulation theory proponents almost always seem to presume the existence of some sort of being that deliberately created the simulation. What if the simulation is a complete accident that has arisen from the complex interaction of brainless bacteria competing for scarce resources on the surface of a big rock? An accidental simulation emerging from randomly initialized 'cellular automata' on one rock in a host universe containing trillions of such rocks.

The usual presumption of the simulation being a deliberate construction of a conscious being makes the whole thing seem like nu-religion for people who reject supernatural things. With the presumption of a being deliberately creating the situation, you pull in these notions: We're special, we exist on purpose, we are probably being examined and judged. This reeks of religion.

It’d be simpler to say there are no atoms, just a neural net fed into your senses making you think that there are atoms.

The neural net itself is built on a much simpler substrate in an external universe.

Ever wonder why you weren't born a medieval peasant?

Well, from the outside reality's perspective, it's helpful for people to spend the first few decades of their lives in an early 21st century simulation, just so they can gradually acclimate themselves to all this technology.

/folly

Can't wait for the Stability-equivalent.