Supercharging Style Transfer

Surely those aren't good ones to compare against due to bias. A more fair way to judge would be to have people rate unknown artists' work as well as the machine output. Then see how that pans out over time.

Judging from some of the stuff on display in galleries right now, I'd be shocked if machines can't outperform or match. And they'll only get better, right?

> Until there is an element of randomness fueled by context, they aren't creating anything new.

That argument could be said about any innovation.

The Google paper is an iteration on neural style transfer; just because it doesn't mimic the "soul" of art doesn't mean the field is not worth pursuing.

> Notice how the transferred images have none of the power of the originals.

Partly because they are like 200x200 pixel images. There is no power in the original either that way. And I've been seen a ton of these posts like "look at these beautiful images", and the images are absolutely tiny. It's really ridiculous and annoying.

I agree with the first half of this comment, but I don't follow you here: "Until there is an element of randomness fueled by context, they aren't creating anything new. That's where genius lies."

I suspect reframing of problems, seeing things in new light, paradigm shifts, new ontologies, whatever you want to call it are not quite so simple as context-dependent randomness! I don't think we understand this process very well right now.

Finally; I think there is an artfulness in copying existing patterns, because the way in which you "abduce"[1] an observation-explaining theory out of the infinite space of possibilities is a creative and aesthetic process.

[1] https://en.wikipedia.org/wiki/Abductive_reasoning

I totally agree that the transferred images don't have the power of the originals. I know nothing about art (well, painted art), but its striking how the transfered images seem like cheap imitations in the style of the originals. And yet, im still impressed by the technology.

There is an element of randomness fueled by context, at least in the technical sense of there being random variables with distributions conditioned on the inputs.

As a recent devotee of Prisma, I would agree that the highest-level discretionary function is (as yet) totally missing from the computational side...

...BUT I would also say that this technology is yet-another-one which can act as an effective 'force multiplier' in the sense that if you already _have_ a reasonable 'eye' you can use this as a very very powerful tool.

One of the benefits of that is that if like me you have a 'decent' eye only, but no e.g. visual genius, an app like Prisma plus a modest amount of straightforward judicious additional editing / cropping, can lead unnervingly quickly to results which I find _very_ satisfying.

To put that another way, I find that I can now in-phone produce images which for my own interests and tastes are quite arresting.

They are not fine art, or innovative... but they represent a rendering into pixels of notions I am not otherwise able to execute, and frankly am not able to even foresee as effective.

I find then that I am often in a very specific aesthetic mode using them: discrimination and rejection of all but the strongest candidates.

Feels like I am executing a GA selection function, essentially, and then just doing a little post production...

> Notice how the transferred images have none of the power of the originals.

They're not meant to. This is the development of base, mechanical technique, not of composition and theory. Lots of artists paint fairly mediocre pieces when developing technique and very often copy other works of art in the process.

> I don't believe that neural networks will ever be artists because they work on copying existing patterns. Until there is an element of randomness fueled by context, they aren't creating anything new. That's where genius lies.

What you ask for is actually a fairly simple architecture, it's just presently computationally intractable. So we're seeing if we can optimize the approaches to run on modern(ish) hardware.

Magenta's goal is to make tools for artists. Sure, regular people can use them, but only a good artist will be creating incredible results.

> deeper meaning eludes machines.

We don't have to have "deeper meaning". We just have to pass enough double-blind trials of whether an expert can distinguish "deep meaning" from "AI-generated meaning". I'm not a big fan of the "humans are special and machines can't capture magical human nuances" hypothesis.

In the context you're talking about, I agree, but non-contextual/pure consumption art is a different story. Think about how game art/textures, Instagram/imgur/reddit images that people look at for one second, like/upvote, then move on, YouTube background music, etc.

We might not get world-changing, paradigm-changing, transcendental, strong social commentary-inflected art, but we are getting some cool stuff people like looking at. Not to mention meme material ("Dog or Bagel" and friends) that uses stuff like this as part of its input (which, I would argue, is its own kind of art.)

Why should we be disappointed if it's not actually artistic-genius-in-a-box? Here is an interesting new technique that's just been invented. Its artistic possibilities have barely been touched.

You're correct. It's imitating style, not creating substance.

This is effectively the 2d image version of those "random text generators" that ape a given style. It's really interesting and cool but it's not "Hollywood AI" in any sense.

> Until there is an element of randomness fueled by context, they aren't creating anything new.

A neural net has plenty of randomness (which shouldn't be confused with nondeterminism) inside it, right from the start with its randomly initialized weights & biases; no two trained style networks will be exactly alike and produce exactly the same output unless one takes major trouble to do so.

Any artist (or writer) has had the experience of people finding meaning and depth in their creations which they had no idea about, and may have thought one of their worser works. This can reflect unconscious work, of course, but it also reflects that the artistic process is as much about throwing random things at the wall to see what sticks and triggers an intuitive reaction in other people that 'hey that's good!'. One may have fail hundreds of times before getting a hit. Rarely does one sit down and think 'I am going to write an intricate story about dying in Venice with such and such symbolism, a buildup to the ending, and referencing these specific past authors', write it out in full in 1 pass without revision, decide it's good, and have everyone else agree. Art is as much about about editing, curation, and recognition as it is about planning. A neural net could create great works of art without any planning or intention of doing so, if the right person sees it and selects it out of the vast stream of images the net will generate.

After being infatuated by a lot of art and medium, I recently found myself perplexed by painting (and some forms of illustrations). There's some magic beauty there, that I often feel is lacking in, say, movies, that can't be transfered. Something about re expressing our biological perception and the "soul" level ones, intertwined.

I think it's a lot more reasonable to think of algorithms like this as new media for artists rather than a replacement for them. Obviously the choice of subject matter is important for art, but the subject matter in these examples was chosen by humans was it not? Humans took the original pictures that the algorithms were applied to. Their choice of subject wasn't made for artistic impact though, it was made to give them a base to demonstrate the techniques with. Artists do this will all types of media while they're learning and trying things out. There are already some artists out their doing some really cool things by incorporating neural nets into their work and I expect their will be many more as the tools become more mainstream and available. Why do you feel the need to be so dismissive of new ideas in art?

Something that could help is feeding the NN more paintings in the same style as a style base. Otherwise it seems the NN gets "too fixated" in trying to make a painting that's similar to the original (see suns appearing when transferring from Starry Night, buildings disappearing, etc)

While I agree with everything you're essentially saying...

> Notice how the transferred images have none of the power of the originals.

I disagree. Maybe that's because I've been described at emotionless but the original art doesn't have any "power" to me at all. I honestly see all of them as basically the same as far as impact goes.

Now I'm likely in the minority here but it shows that art is going to be very personal and there isn't going to be very objective ways to measuring "power" or "emotional impact" except through an aggregate polling.

> In the case of Munch's The Scream, and Starry Night, and every Monet, the subject matter is as important as the image.

Not to mention the transformations the artists made that the software cannot: turning pinpoint stars in the sky into huge pinwheels and fireballs; turning a human face and hands into an inhuman creature.

This incarnation of neural networks will never be artists true, but some future iteration of combinations of algorithms (some of which will be informed by current neural networks) likely will produce many great artists per turn.

That said, when people express skepticism about current AI produced art, what they are really trying to express is that current AI lacks a theory of mind in its productions. Artists are not simply sampling from a generative model, they're generating a construction informed by complex inner states while also predicting how arrangements of states will affect a perceiver. Great artists, especially writers and poets, are able to operate at above average levels of intentionality.

Until we have AI with basic commonsense, the good news is—for an undefined period of time—AI will foreseeably be tools which augment artists (humans) and raise the floor for what counts as creative work. Even within a couple years, things by talented teams that would impress today will be run of the mill thanks to intelligent tools but one can hardly imagine what talent of the future enabled by semi-autonomous tools will create. This help could hardly be more timely thanks to rising production costs, falling profits and new content intensive modalities as VR.

If you're interested in theory of mind, this is a good recent read: [The Complexity of Jokes Is Limited by Cognitive Constraints on Mentalizing](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4864470/)

This has bugged me for some time now, but how does an artist like Beyonce know about the "black struggle" in America? She's indeed black, she is an American and she lives there, but judging from half a world away it seems that she lives in a parallel universe compared to what black people who are poor experience day-to-day by living in the States.

Years ago a man made a computer program that could compose music. He kept getting told things like what you are saying. That it's soulless, that computers can never make real art, that it has no emotion, that it's just regurgitating patterns, etc.

Eventually an orchestra agreed to play one of the works. People liked it. Until they found it was composed by a computer, and then they hated it. People are insanely biased against computers.

That said, the algorithm was not a neural network. NNs do have problems producing music, they forget long term patterns quickly and so the works are easily distinguishable from human music. But it's possible this could be solved in the future with some new techniques at training NNs with longer term memory. I don't think it's long before the world is full of AI produced music. And you won't be able to tell.

This instagram account - https://www.instagram.com/prisma/ - was posted below and contains selected images produced by the Prisma app. They might not compare with the masters, but they are definitely art in my opinion. Examples:

https://www.instagram.com/p/BL-ngYTjXZH/

https://www.instagram.com/p/BLfxvgyD-mu/

https://www.instagram.com/p/BHnZyddjumg/

This is a very new field, I find it very exciting and can't wait to see where we are in 5-10 years, personally.

Deeper meaning is inferred by the viewer/listener every bit as much as it is implied by the creator. I think the future is in empowering artists with better computational tools to iterate, create, and explore.

That said, I think saying never is a bit strong. People like an amazing variety of things, and given enough design, statistical modeling, luck, and randomness something interesting enough for a listener to infer meaning can pop out. You should check out this article about David Cope's work [0] and thoughts on creativity.

[0] https://www.theguardian.com/technology/2010/jul/11/david-cop...

Also check out Prisma app for a slick mobile-friendly UI & subset of pre-selected art styles. (layman's application of this tech)

Thanks for releasing your code for this! Also really enjoy your cs231n lectures on youtube.

There is some grain/small repeating pattern in many of the style transfer pictures I've seen - could it be an artifact of this? http://distill.pub/2016/deconv-checkerboard/

Hey jcjohns, fan of your work.

I've noticed that your project for fast-neural-style does instance normalization over batch normalization.

Batch normalization has the benefit that you can merge the gamma & beta into a convolutional layer on the forward pass, which makes it a lot faster by allowing you to skip a step when building the styled images using a trained model.

Can the same be done with instance normalization? I didn't see a formula in the paper but I would think so, since they are fairly closely related.

Did you try putting more structure (concentric outlines or a gradient or something) in the input image? Seems like style transfer would fall flat on a binary image.

Good to see more people with neural images on their wall, here are some hand painted: https://i.imgur.com/6nOllzf.jpg

Protip: Often when it's overcast and unpleasant in the Marina it's beautiful at Ocean Beach (and vice versa).

"I'll take a Wes Anderson movie about volcanoes, with Bill Murray starring in it, computer."

Musicians have always been about the performance more than the creation of the music. A ton of musicians do not write their own music, for instance, but are still really popular.

Interesting. Your comment gets at the question of what constitutes art.

I think there are many forms and purposes of art, but the kind we most often consider authentic usually seems in many ways crude compared to what preceded it:

- Jazz began as something that sounded cacophonous compared to the more obvious order of hymns and classical music. Jazz transformed what had been brief moments of dissonance and arrhythmia into elaborate intereleavings of tension and resolution, structure and chaos.

- Rap began by taking the repetition and rhyme of poetry and the rhythms of african music and created something many would not classify as music when they first heard it.

- In order to be deemed fresh and worthy of recognition, fiction must not feel like a rehash of the great classics. In most cases the best new fiction smashes conventions in at least one narrow way. I'd argue that an unattributed work from a great author would not likely be publishable today because of that. Quality is not just subjective, it is relative.

- In fashion, the moment a style is obvious it becomes outdated and means something quite different. In some cases the latter can be marketed successfully, but it is hard to think of it as art by that point. Think of the way iconic things felt when they were new (and ugly to most people at that moment) and someone was making them iconic.

So while these algorithms create believable surface features mimicing established instances of "art" (they might persuade me that a short story was written by Fitzgerald when in fact it had been written by a grad student and run through the algorithm) they won't necessarily make it a good short story.

But in another year researchers will likely understand what makes humans consider things stylistically novel and then all the above points will be irrelevant.

> I wonder what is the point to have artists in the future and how the artists can survive?

I feel like that's a bit like someone in the 1800's asking if artists will still exist once photography technology improves.

If art creation is completely monopolized by AI then I think it's likely most of the rest of our jobs will be too.

> (...) how the artists can survive?

Let's take "survive" to mean survive financially based on one's ability to produce art with commercial value during their lifetimes.

As others do, I think art is much more than than the piece itself, through whatever senses we use to perceive it. It is also the story of its creator and the wider context of its creation, as well as history of its movement through the world including those of its various owners.

I think this leaves tremendous room for human artists today and far into the future. Further, I believe some artists, and there certainly exists those who have tried already, will direct machine learning based art, adapting it to be more likely to succeed based on those factors I indicated.

The moderately impressive part of these algorithms is the fact that they can "learn" a style from just one image.

However, in reality they are not nearly as universal as these papers make them seem. Pretty much all my attempts to use these algorithms resulted in garbage with ugly artefacts. Which, actually, was pretty informative, since I saw exactly what kinds of features they pick up. I am disappointed that these blog posts don't include any "failed" images of that sort.

More than likely you're just biased because you know ahead of time it was done by a computer. We could probably take one of the images, present it in the right context, like in an art gallery where we pretend that it was created by some fictional up-and-coming artist, and people would love it instead.

Love that idea!

Prisma never had "ownership" of this effect; the papers of Neural Style transfer have been out for months, with this Google method being another iteration.

Prisma was just the first polished app released using style transfer. (However, since the technique is public, it will make an acquisition of Prisma less worthwhile)

Prisma was a land grab because the original authors didn't make an app. Hell I started working on my own version at the time since it was such an obvious app.

The original authors finally got around to making their DeepArt app, so we have at least two players in this space atm.

I think either of these two players will incorporate this research, but there's no clear incentive for anyone else to try and get in on this market now. It's not clear that there's a whole lot of money to be made here, but who knows maybe usage will explode. I mostly expect an acquisition.

I agree. While this is super impressive stuff it seems to match palette far more than style. Van gogh has a distinct style even though not all of his paintings have the same color palette.

There has been other recent work (Preserving Color in Neural Artistic Style Transfer) that allowed control, separate from the "style", over how much of the original painting's color palette to use:

https://arxiv.org/pdf/1606.05897v1.pdf

It isn't too hard to extrapolate from this to using separate images as sources for the style and palette, and from there to transforming the "content" image's color palette to resemble the palette of the "style" image in terms of saturation, value, contrast, etc. (but not hue) by manipulating histograms, and then using that result instead of either the "content" image's or the "style" image's palettes.

Because if you have a photo of a bright red cardinal and a desaturated pastel drawing of a pale blue boat, the result you are likely to be most pleased with would be a pastel cardinal that is, well, pink, and not a pale blue cardinal.

Of course, the harder version of this would need to recognize the semantics of both the "content" and the "style" image in order to construct a more meaningful palette to be applied (which is what you're getting at), and that same semantic information will also help with transferring the style more meaningfully (eg. crude palette-knife houses in the background but small brushed strokes depicting the people in the foreground).

That's a pretty cool use of a twitter bot.

There is also DeepForger (been borked for a while though):

https://deepforger.com

https://twitter.com/DeepForger

Great, what I've always wanted, more Monet. Now only if I could have less problems...

There was a twitter bot that would do this for you. Unfortunately it got to many copyright complaints and was shut down. Here's one it did for me: https://twitter.com/sep332/status/705882039720013824 (scroll down for the result)