Sakuga-42M Dataset: Scaling Up Cartoon Research
arxiv.org
arxiv.org
A large chunk of an anime's budget goes to in-betweening. Essentially human interpolation of graphics between two key frames (Its usually like 6-12 inbetweens per key frame). People hate this job, and it is highly unproductive, so generally outsourced by Japan to other countries.
Western animation decided to abandon it altogether, and move first to flash, then to 3d animation. But in retrospect that was a mistake, as it lost so much of the creative flexibility of 2d animation. Anime today is substantially bigger than western animation as a result. Crunchyroll has 13 million subscribers.
AI will solve the problems with 2d animation. Something like SORA fine-tuned on anime-keyframe data like this. Can probably easily solve in-betweening. Then the 2d animation workflow will dominate 3d. Its so much easier to just draw a beginning and end key-frame, then have the AI fill it in. Than to model and rig and render a entire scene.
That being said, I'm sure this is going to take a while to fully replace the manual efforts. There will probably be awkward phase in between the outset and the perfect modelling efforts, and I'm sure lower budget shows (or ones looking to cut corners) will be the early adopters.
The more rigid the object, the better this works. 3d cars look better than 2d cars, even in an otherwise 2d show. Mechs look a bit worse. And human characters look horrible.
Yet anime studios still do it. Including for critical highlight scenes like dance scenes (Check out Love live dance scens), because it is so, so hard to draw humans dancing.
So if anime studios are willing to do something, that looks obviously bad, as a widespread practice. There'll be 0 barriers to AI inbetweening adaptation, which would likely look BETTER than human inbetweening within a year of release.
AI anime art has already wiped out the lower-end of patreon artists, and is heavily impacting the mid-tier. Because AI has gotten more technically proficient than the average mid-tier artist. Pretty much only the higher-end can hold their heads above water. Or they have to transition to drawing comics with storylines, instead of just simple images.
This is quite debatable, if you notice that the car is a 3D object, then something is already wrong.
But for "boring" rigid objects, there's less of this advantage; hence, the consistency benefits often are more important.
There's a lot of impressive work in 3D animation that looks quite good. Outside of Bandai Namco's work on idol anime, Studio Orange has made some of the best looking 3D modeled anime lately and a few other studios have been getting into it. I'm more familiar with video game animation, where Arcsys Works has made great strides too, by using animation on threes, manual tweening, stretch and squish bones, and carefully UV mapped textures for crisp color boundaries.
I'm not sure if I'll be able to find this, but there's an episode of Steven Universe with an extended reference to a scene from Kill la Kill of a transformation sequence.
It looks maybe 1% as good; not only that, but the character turns into a pure white silhouette for the entire transformation because it doesn't have a character design suitable for being transformed. (Instead it has one designed to make the animators' lives easier.)
When it's bad, you recognize that it's janky crap tier 3d animation from a company that either didn't care or was put under such a tight timeline that they simply couldn't care.
An employee of an animation company describes in a comic book his experience on working with people drawing the in-between frames. They were paid literally with rice bags [1].
[1]: https://en.wikipedia.org/wiki/Pyongyang:_A_Journey_in_North_...
(Rough generalization, but anime is more likely to do this with foreground vs background elements, while Western animation would do this with different characters. In anime the keyframe artists tend to draw every character in the frame, and put more of a personal touch on them, so it's easier to see their individual styles.)
Since this is a "sakuga" dataset, Mitsuo Iso is one of the most famous examples of really natural looking movement here.
https://www.youtube.com/watch?v=NTMJ8dGFUkc
When they're bad at it you get this effect where characters constantly seem to "settle" into a keyframe pose that looks realistic but too static, and then immediately go back into inbetweening that moves a lot but isn't physically possible. I feel like B-grade Disney stuff is the worst here but don't have an example on hand.
Of course, for a user to use the dataset, they'd have to download it. Whether or not that's a copyright violation depends on their local laws.
That's probably still illegal. The implied purpose is copyright violation and piracy. Judges aren't computers - they're capable of discerning when someone is trying to skirt the law by saying "here's something you could use to do something illegal...wink wink nudge nudge".
Those other sites that got in trouble are links to all pirated content, for the purpose of pirating content.
And we still haven't decided this isn't fair use (it's non-commercial and used only for research, so it can't really be said it harms the copyright holders' interests), and fair use is by definition not a violation of copyright.
> [May 16, 2024 Update] Due to the recently added anti-bot measures by the data holders, our downloading pipeline is no longer working. The video links are still accessible through a browser but not via our python crawler. We are working on a workaround and will make an update once we find one. At this time, we are still providing the parquet files to researchers, but researchers will need to find a way obtain the video data. Thank you for your understanding.
Ah, so on top of making a “dataset” of a category of specific works, they are also making people hammer the servers of other parties who never agreed to pay the bandwidth for a bunch of “researchers” wanting to download all of these files. Classy.