citnaj

Hello- Jason here (one of the authors). I'll answer questions! I might not get to them right away as I'm about to go to bed.

OMG dude that's such an accurate description of my life at that time.

Very cool project. And 11th grade!?! Wow....I definitely wasn't doing anything like this at that age.

Something that came to mind as a potential way to get clearer and more realistic results is to add GAN loss. You can reference this as an example (shows pretraining on L2/mse and after with GAN):

https://github.com/fastai/course-v3/blob/master/nbs/dl1/less...

I'm not 100% sure if it'd work but it seems like a potentially cool followup project.

That is exciting! I'll be watching on Twitter :)

>"More recently, there is a shift towards using a Transformer architecture, and right now I’m experimenting with that as well."

I'm really curious- any early results to share on that? Attention really does make a big difference on a lot of things (including work I've done so I know first hand). It should improve the coherence of the entire music piece in theory at least, right?

Update: I got this working, and dude- it's so awesome in every way. This is the most substantial improvement I've seen yet. Most importantly- it massively reduces memory requirements. Thank you so much. I'll commit within a day or so and make sure to mention you, on Twitter.

Yeah the more I churn over this idea in my head the more excited I get about it. This really sounds like a big win.

I'm not sure what I want to do about the Kanban board versus issues tracker yet... I'm used to JIRA mostly. I'll figure it out but do know your contribution is very very much appreciated. I don't think I would have come up with that.

Oh shit yeah that really does look good! Amazing really. Ok...I'm going to put these notes on the project board.

Yes..I definitely glossed over the proposed workaround and I apologize. Thanks for this.

That's very interesting to point out Thank you!

I was actually thinking along the same lines because yeah...if you could break this problem down into smaller pieces, it would probably be the most effective way to reduce memory requirements. But I do think that's easier said than done. This is where I think I'll have to rely on Ian Goodfellow and others to come up with another something brilliant for me to stick in the code lol

I should add too, just to be clear- I'm not at all involved in academia.

I understand the frustration and in fact share it to a certain extent with science in general. Keep in mind that this wasn't intended to be published as a paper or anything like that. I'm just a software engineer who picked a problem and found a pretty cool solution.

Primarily I thought it was cool because it should be useful in many other image modification domains. And then it blew up in popularity today (didn't expect that). But yeah in the notes in the readme at github I do say this:

>To expand on the above- Getting the best images really boils down to the art of selection.

I added that after getting some feedback similar to yours, because before that, this disclaimer wasn't quite cutting it apparently:

>You'll have to play around with the size of the image a bit to get the best result output.

So yeah I'm trying to stay honest here. I'm not going as far as picking completely random samples, admittedly, but really what I'm trying to drive at here is you can produce cool results with this tool. It's not perfect, but it's a tool. And even if you pick at random, they still look pretty damn good. Just sometimes it renders the tv as color and sometimes it doesn't, and i picked the cool option.

Author here. That's actually what I find quite fascinating myself about the results- that they look almost perfect at first glance, yet you drill down a bit closer and you see another "zombie hand". The resolution issue you mention is definitely something I'm painfully aware of- it just comes down to lack of memory resources to support bigger renderings. That's going to be something I'm going to try to attack next.

LOL I think that'll be my go to excuse for lack of documentation for now on....

Added this to the Known Issues section.

Yeah I tried pointing that out in the Known Issues section by alluding to adjusting the size parameter as a means to get the best images. But I think I'll just go ahead and be crystal clear on the "art of selection" part so that this doesn't come across as snake oil.

Author here. Easy to answer that one- altering the training photos with random lighting/contrast changes (yet keeping the color targets the same) really helped to deal with varying qualities of photos. But also neural networks are just particular good at picking up on context, so that has a lot to do with why the results are so robust.

Author here: It's early, and currently resolution is limited primarily by model size. Which drives me nuts. It's one of my top priorities to address because that would be a great improvement. Adding super-resolution to the pipeline should also be pretty easy but I want to at least output a reasonable base resolution on the photos first before I go that route.

Oh yeah to answer your question- super resolution does indeed make up details as you describe there and arguably does blur the line with restoration/story telling. But so does colorization- not all the colors added by the model are going to be what was actually going on there, of course.

Author here. Yeah you're basically right. GANs vastly improve the situation though because being safe with "green for grass, blue for sky, brown as default" doesn't work in the Generative-Adversarial setting. The critic will assign lower scores if the generator keeps outputting brown. Now I'd think the generator would get more creative than going for blue constantly but that might just be a matter or more/better training (...?)

Author here. Yes, it drives me nuts :)

Author here- So I'll just be brutally honest on that one- not all renders are doing that. I cherry picked the one that did that because yeah, it's amazing. There's a simple explanation for why it sometimes doesn't pick up on the guy on tv to color it- The source material is fuzzy and small.

I wish I could claim it was something more awesome than that but that's the truth! I'm treating these outputs as an art of selection to a certain extent because it's simply not 100% consistent yet. That's one of the things I'm going to continue to try to improve upon.