Hacker News Reader

Comment by ComputerGuru | Hacker News Reader

The "cleanliness" of the resulting images is undeniable, but once you get past the sheer awe at how crisp and clear the upscaled image is, you'll immediately notice the loss of detail. It completely does away with any and all texturing, which is especially noticeable in the last image ([1] vs [2]) - look at the scales and patterned lines on the snake (?) around his neck and the white strands in his hair, and of course, the letters have been turned into (unrecognizable?) squiggles.

Still, in terms of pure shock and awe - they're jaw-droppingly nice for upscaled versions, to the point where if you didn't have the original, it wouldn't occur to you that this wasn't it.

1: http://ukiyo-e.org/image/mfa/sc165440

2: http://i.imgur.com/541uG5t.png

View on HN

I find this an unfair criticism.

This is trained to upscale anime images and not woodblock prints - and anime images are typically flat, uniform colors. This may have issues scaling up a still of a background scene from 5cm/second but would fair much better with a character still from .hack//sign. You have to keep in mind what it is trying to scale up.

>(Naturally I could train a new CNN to do this, but it may not even be necessary!)

Training to upscale woodblock prints and retaining the texture might make sense if you care to retain texture. It only works as well as it does because the style is very similar.

One thing I should note is that when looking at prints (at least for when it comes to technical analysis) being able to see accurate representations of the lines is far more important than the uniformity of colored regions. Color is almost always at the whim of the printer on any given day, whereas the black lines (from the keyblock) should always remain the same. Granted you're going to have issues either way (using this tool or doing normal scaling) as the source material is inherently compromised.

Although it's not clear what scenes exactly the upscaler was trained up, I suspect that it's currently best suited towards scenes that have lots of large bold lines and not lots of tiny details.

I imagine it was trained on 'typical' anime-style art of black/bold character outlines and mostly flat colors.

A common solution to resizing anime characters is to create a colored vector of an image. The differences between these vectors and the original stills are minimal and usually 'satisfactory'. There is an entire scene of people who create these vectors and another scene of people who use the vectors to create wallpapers and other graphics. [0] Waifu2x can help replace the need to vector these images by increasing the quality of upscaling them.

This is the prevalent 'style' for anime - at least from the past 8-10 years or so. There are a few outliers and I imagine Waifu2x would work poorly on them. For example, I do not see it working well on a still from "The Garden of Words". [1]

[0] http://img04.deviantart.net/96b1/i/2015/105/8/d/oumae_kumiko...

[1] https://24framesps.files.wordpress.com/2014/11/the-garden-of...

cheers for everything, and this comment

One potential point of improvement is training the neural network in prints with Japanese text, this seems to be the weakest point in using the Anime trained one.

To be fair, the demo site provides a configurable level of artifact reduction. This article uses the highest level. Here it is with none and some:

http://imgur.com/a/cVVnC

Damn, but the "some" looks great.

Yeah, on second thought, after seeing the low noise reduction result again, I suspect that may be an even better result for what I'm looking to achieve. Many of the details in his rope are preserved and the calligraphy appears to be in better shape.

I've updated the post to include additional low-noise-reduction filter examples. Thank you for reminding me to look at them!

Maybe? I'm definitely biased in that I have substantial computer vision & image processing experience, but the output looks riddled with obvious filter and vectorization artifacts to me.

Re: Image 2, http://i.imgur.com/541uG5t.png

Yeah - the background looking like crumpled-up wrapping paper is definitely not ideal. I suspect that it's having trouble with mostly-uniform areas of color that have slight variations. It appears to be extrapolating and creating these larger effects.

Have you tried running the original through a high pass filter (to get the textures) and applying it over the vectorized version? It might work for the background texture, though it would probably suck for the text.

It would preserve all the JPEG artifacts, which are the main reason the naively upscaled version looks so crappy in the first place.

There are well-known techniques to clean JPEG artifacts, but they need the tables from inside the JPEG to work best.

Since JPEG uses an 8x8 block transform, you can find artifacts by shifting the image a few pixels over and looking for how the transformed block changes, basically.

https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_...

Also, using a better chroma upscale can help for small images. libjpeg just uses nearest-neighbor (no "real" resizing) and hardly anyone notices, but it helps with lines and edges.

I have not - although that's an interesting idea, thank you! Relative to my other projects this is a very low-priority exploration. I was very interested to see if there could be a "cheap win" for this particular sub-problem that I will be dealing with, should I get around to digitizing these books.

mostly-uniform areas of color that have slight variations

That's precisely what it's attempting to smooth over - and it works well for anime images because in them, those variations would be considered noise.

Wondering if it would anime-ise it, I fed Waifu2x the standard Lenna image - twice - ended up with this:

http://i.imgur.com/iNWRjIS.png

That's fantastic!

How did you achieve that? I've run it through several times and whatever settings I use, can't get anything close to yours?

I took the original, scaled it down to 1/4, and then back up to original with Waifu2x at 2x scale, maximum noise reduction.

The article is unclear, but I think he was upscaling from the small "source images" in the article, not the full images linked through them. Compare that (http://i0.wp.com/data.ukiyo-e.org/mfa/scaled/sc165440.jpg) to Waifu2x low noise reduction (http://i.imgur.com/pDmgNZS.png), and sharpness and detail definitely improve. And then they get worse again with high noise reduction (http://i.imgur.com/541uG5t.png), so that says something about the best parameters to use.

(Edit: It looks like the low-noise-reduction version was added later, and you were talking about the high-noise-reduction version, in which case, fair enough.)

The version he gives is smaller than the source image, which makes me think the image is upscaled from the grainy preview.

I tried it on the actual source myself (using http://waifu2x.udp.jp/), and there was very little actual loss of this kind.

I feel like you hit a borderline-pathological case for the noise reducer with that image. It hasn't just blurred the cross-hatching (? is that what those patterned lines are called?), its completely removed it.

I tried it with just upscaling and no noise reduction, and the result is about what you'd about: a really nice upscale, perfectly preserving all those patterns (as well as the noise, unfortunately). Doing that and filtering in another program might work better.

Reply on news.ycombinator.com