Colorized and upscaled footage of a snowball fight filmed in 1896
petapixel.com
petapixel.com
The funny thing is he has been kind of attacked on Twitter recently for bizarre reasons ... people are pointing out how colorization detracts from the source material, or ruins history, etc.
What this really is, is a hacker making the world's best neural net in the world at a certain task. (It appears to have also become his day job now.)
Let's try to appreciate what an achievement it really is. Pretty much all of the automatic colorization you may have seen online in the last year is some version of DeOldify.
Though I understand the historian side of things... every time I've seen major colorizing or cleanup efforts, I've seen far more people looking at them and talking about the history around that time than I'd normally see in, like, a year or more. Feels like a massive net-win to me.
In that regard, the only downside for historians is that it might become more difficult to find the source material if upgraded versions of it proliferate, but this is not like rewriting or erasing history like some have claimed.
You mean the contrarian side of things? There isn't a big movement of historians against this.
> There has been renewed interest in this 2018 post after it was quoted in a piece by Thomas Nicholson, ‘YouTubers are upscaling the past to 4K. Historians want them to stop‘, for Wired.co.uk. The piece is chiefly concerned with the recent ‘upscaling’ of early films done by Denis Shiryaev on the Neural Love site. I rather admire this work, which is reasonably honest about how it has treated the originals, and is fun. Contrary to what the article’s title states, this historian does not want them to stop. I just want there to be better understanding, not least of the consequences.
In the original, I see a bunch of Vaudeville "clowns" doing their antics, and a "straight-man" appears on stage, and tries desperately to escape the situation. The result is a feeling of the film attempting to induce mild amusement. Somewhat like watching a film where someone slips on a banana peel, or sits on a breakaway chair (slapstick comedy).
I believe that this is due to all the exposure I've had of the Vaudeville material has been based on grainy jittery monochrome film, so I interpret the material based on the typical Vaudeville setup.
I still remember an example from a series of photography books that my parents purchased for me when I was teenager that went into this.
Two scenes from an inner-city neighborhood of children playing and adults standing around. The black and white version emphasized the general grittiness. (Which by that point in the 1970s we were proabably at least somewhat conditioned to often associate B&W with.)
By contrast, the color version had lots of bright colorxs and really emphasized the children happily playing.
[0]https://en.wikipedia.org/wiki/Bataille_de_boules_de_neige
Then you're left 20 years later with a horribly jpeg'd mess in 800x600 px.
I don't have a source on this, but it's my impression a lot of libraries did this in the 90's or 2000's (if someone knows specific cases i'd love to hear about them, I hope I'm wrong)
https://www.reddit.com/r/OutOfTheLoop/comments/j523qi/whats_...
I found this point interesting:
"The thing about old photos is that everything about the image is part of the information we can learn from it—not just the picture itself and what it's a picture of, but how it was composed, what its medium was, the balance of lights and darks... It's all part of the document. And when you start to alter it, that's creating a new document reflecting a conjunction between the past and the present. "
Few historians will rail against an inaccurate portrayal of historic events in fiction. For one, many enjoyable books or movies would be rather dull if authors didn't have the liberty to beef up the story.
However, historians will pipe up when fiction is taken for a historical fact. That's what historical research is all about: uncovering the facts and testing how those facts support different theses, narratives and perspectives.
This debate though, isn't about misrepresenting history through fictionalized stories. This is about enhancing the quality of the footage itself and how that impacts our response and understanding of the past. Which is something else.
The issue here is the intent with which such enhanced footage gets purposed and repurposed later on. Today, it's sights and sounds of cities, or a snowball fight. Tommorow, someone might apply those same techniques to reels of film which are culturally, socially and politically sensitive in order to elicit a specific, strong emotional response with their target audience.
If done subtly, an unsuspecting audience might never even realize that they are watching enhanced or re-touched footage, instead assuming or accepting that the footage was made during that era.
That's what is pointed out here: by enhancing the footage - removing the artefacts that show it's true age - you also remove barriers that create a gap between you and the past. For better or worst, that is.
I was surprised enough there was footage at all, anything to colourise. (Sure enough TFA says the creators were pioneers.)
You'd be hard pressed to find people on HN to believe that this enhanced footage is original. Why? Because this audience understands that these techniques exist and is able to apply their knowledge to what's presented to them.
However, the world is big and not all humans are in a position to spot the difference. Either because they never attained that level of education, don't have access to knowledge, lack the incentives to get educated, aren't aware of their own biases, and so on. This isn't a small group of people, and going out you're going to encounter people who would totally end up believing this is original footage if you primed them before showing them.
Now, you might think "Well, luckily I can spot that difference." but you're not entirely out of the woods either.
See, this is a clear example of altered footage. The contents and the time period are clear from what you see. But things aren't always that clear.
For instance, someone decides to make a documentary on a contemporary hot button topic. The premise of the documentary already aligns with your point of view on that topic. Now, the makers of that documentary could decide to spruce things by interspersing their film with carefully chosen, spliced footage which is subtly enhanced. That footage is then passed off as "fact" in order to make things even more poignant.
If you watch an hour long documentary containing five 5 second long shots with such enhanced footage, would you be able to pick them out as you are watching the documentary? Without anyone telling that those shots were there in the first place?
Now, to be clear, I'm not against enhancing footage. I think that's inevitable. And it's not a new thing either. Altered copies and mucking with documents has existed since the dawn of recorded history.
What matters, though, is understanding how these techniques could be used for better and for worse; but - above else - also daring to question your own biases and perspective on the world coloured by your personal circumstances.
I'll give two reasons why assumptions about general knowledge levels are biased:
Geography:
> The Council on Foreign Relations (CFR) and National Geographic commissioned a survey to gauge what young people educated in American colleges and universities know about geography, the environment, demographics, U.S. foreign policy, recent international events, and economics. The survey, conducted in May 2016 among 1,203 respondents aged eighteen to twenty-six, revealed significant gaps between what young people understand about today’s world and what they need to know to successfully navigate and compete in it. The average score on the survey’s knowledge questions was only 55 percent correct, and just 29 percent of respondents earned a minimal pass—66 percent correct or better. [1]
[1] https://www.cfr.org/global-literacy-survey
And, far more important, literacy:
> According to data from the 2014 U.S. Census Bureau, 21 percent — or nearly 60,000 — of working age adults in the city lack a high school diploma. At the same time, 19 percent of adults cannot read a newspaper, much less complete a job application, according to the National Center for Education Statistics. The situation is just as worrisome at a national level. Approximately 32 million adults in the United States can’t read, according to the U.S. Department of Education and the National Institute of Literacy. The Organization for Economic Cooperation and Development found that 50 percent of U.S. adults can’t read a book written at an eighth-grade level. [2]
[2] https://www.washingtonpost.com/news/answer-sheet/wp/2016/11/...
The real trouble with a digital world, in which the wide proliferation of these technologies is driven by their low cost and ease of use, is that entire strata of the human population lack the critical skills, time and energy to even remotely come to develop a critical view of anything which is presented to them.
Sure, and I agree that's an issue, but here we're talking about historical footage for the purposes of understanding history.. frankly anyone that 'matters' and especially actual historians aren't going to think it's original, contemporaneous footage, whether they understand how it could have been produced/altered or not.
If it 'goes viral' and the uneducated assume it's an original C19 recording... It doesn't really matter? It's sad for misinformation to be 'viral' instead of stellar reporting, sure, but it doesn't alter learned knowledge of history.
The thing with "modern" quality is that it distracts so much less that it becomes far easier to consummate. And that does not just apply to young kids but to me as well.
So I don’t mind the old stuff, it does not distract me at all.
But I can understand that if you only experience 1080P and above your whole life, 483 lines in b&w is distracting.
Which show did your children gravitate to that’s been colorized/upscale? I want to see if it has the same effect.
I agree that the originals shouldn't be lost but there's no reason not to do restoration if it is done carefully or tastefully.
The material is already hugely compromised as a historical document of the reality on the day the events were captured. Let's not put it on a pedestal. For example, if we remove the effect of hand-cranking on film to make it more watchable, are we destroying or enhancing? I would argue the latter. The hand-cranking adds little of value.
I view this as one half of the equation, if people can relate to the past more (like WW2 in HD, They Shall Not Grow Old) etc, then suddenly they can connect and understand that human problems have always been around and we aren't so different.
As long as we can balance that effectively with the historical preservation part I'm all for using things such as ML upscaling/recoloring etc in order to make the past much more accessible.
I had the opportunity to see the Apollo 11 documentary on the IMAX last year when it came out, it was one of the most incredible movie experiences of my life and helped me to "get it" in regards to how the people acted, the public's reaction, enthusiasm etc.
Here, though, is a similar technique applied to the Wuppertal monorail.[1] The colors are all wrong. The supports was really lime green, the overhead track was brown, and the carriages were a glossy red. Since the training set lacks data for 19th century supported monorails, it's just guessing.
But isn't that exactly the point?
Instead of making the past "exotic" and different and remote... it makes it seem just like us now.
Because let's face it: people were the same back then. But because of how serious and black-and-white photos usually are, we're usually led to believe the past was far more alien than it was.
I do wonder why they didn't bother to fix that a little?
So they started from a low-res scan, because film is not SD, you can do a 4K scan from film too. Of course you wouldn’t necessarily have 4K of useful information, it might just be grain. But it’s interesting to think that many of these videos probably start with subpar digitizations of the source material, and wonder what it would look like when based on a proper high resolution archival scan
So if you want to start from a 4k source, you have to train your network on 4k data, and that becomes a ludicrously expensive exercise quite quickly.
Like, they're just doing this in the middle of the road! Very few automobiles to spoil the fun.
Personally, from my completely-uneducated viewpoint, I wonder if this shows the current boundary of what's possible, with the AI not having the knowledge of what it sees. Presumably you need to first teach the AI about all of the things in the old photos/videos, and then somehow show what color each of them should be. Particularly, I'm rather skeptical about the latter part because e.g. clothes have plenty of variability in color, and properly you'd need to guess the kind of clothes, the material and the period to take a stab at the color. Like, no blue in the middle ages, except for rich people.
Conversely, I'd imagine that if the AI is trained on ten different examples of pants of various colors but about the same darkness, it can't pick any particularly saturated color because the choice would be mostly random. But ‘drab’ is always safe.