Implementation of Imagen, Google's text-to-image neural network, in PyTorch

Second this, one of the biggest contributors to open source AI and someone everyone should sponsor.

Absolute power, corrupts absolutely. Perhaps it’s a game theoretic approach to leveling the playing field.

Publishing high impact research gives credibility to the ML teams, which helps recruiting and prestige.

It's less cynical, more incentive alignment.

They did it out of goodness of their hearts. :)

I think there are three main reasons why:

- being open is kind of just how things in ML generally work right now, it's in stark contrast to things like chemistry or physics where paywalls are pretty common

- it's a matter of clout, ML is moving ridiculously quickly, with work from just 5 years ago being considered outdated in terms of capability, if you don't publish, someone else will and they'll get the credit. This likely also matters for the researchers since they get credit too. In a sense this is just publish or perish culture from academia.

- it's also somewhat about hiring, which is related to the clout. By putting out this kind of research, they're attracting talented engineers to consider working for them. This of course is pretty relevant to the rest of their business, especially given how heavily Google leans on AI to handle moderation.

What would the product be? My experience is that most ML papers are very brittle, for every cool result/example you see there's a plethora of nonsense spit by the models.

1. This finding is hard to monetize, especially with ROI that Google typically does (for example a an app that makes $500 a month isn't worth it)

2. Deploying models in a cost effective way is hard

3. Lessons learned from building this model can indeed be monetized and many of them may be kept secret.

Because the competitive advantage is also in the training datasets and ML infrastructure.

> Public companies can't do stuff just for the fun of it, right? So there must be some commercial reasoning behind it?

Yes they can actually.

If you are a shareholder you can either sue (unlikely to succeed) or vote against the board. That's pretty much the only recourse.

Two reasons: 1) Even though it's all technically very impressive, so far there's not a huge amount of commercialization potential here. OpenAI is charging for its GPT-3 model but its revenue is probably negligible next to the hardware costs (sunk + ongoing) to train it in the first place, let alone the researcher salaries they're paying 2) Most of the stunning examples are cherry-picked. These things fail much more often than they are willing to admit, and they're probably assuming (correctly) that not enough people are willing to pay for something that only-sorta-kinda-works 1/3 of the time, when you're holding it the right way.

It could be classed as a way to attract talent. It is basically then a very expensive fusball table.

It’s simply flag planting. If they don’t do it, some other player will.

I think it is because the proprietary part for them is the data, not the particular algorithm. They benefit more from other people making advances on their technology because they have the data to get more benefit than anyone else. If they kept it to themselves, they would get no "free" advancement. So they trade-off the secret of the technique in the hope that others will advance the technique, making their data more valuable.

You realize this doesn't include the trained data, right?

Researchers like to talk about and show off their work outside the company. If you don't let them, they get unhappy and leave.

Google started publishing for several reasons but the primary one was recruitment (showing off was a secondary goal). The mapreduce, GFS, and bigtable papers played an important role in attracting an early generation of distributed computing/high performance computing people from around the valley and CMU/MIT, who helped build the second really successful versions of the web search engine (retrieval and ranking), ads serving (the auction, the logs joining pipeline), etc.

The other reason is that the leaders at Google at the time believed that we would achieve the singularity faster if Jeff Dean periodically sent ideas back 10 years in time to Doug Cutting.

This is an important question.

You're saying it's Google who have done this research. In a way that's true. But really it is Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet and Mohammad Norouzi who did it, with material support from Google.

It's likely that some or all of these people would have refused to do the work they do if Google kept it all as their secret sauce.

And moreover, there are excellent reasons why they wouldn't want to. It's not just the obvious that if it all were secret, they wouldn't be able to use it in their non-Google career advancement. It's also that research without the freedom to talk is far more difficult and frustrating.

On paper, scientific papers are supposed to document the whole of the discovery/innovation. So you might think that an insider, who got to read all the secret Google research papers AND all the public ones would have an advantage. But problem is, even the best written papers with full code and comments inevitably leave out things, especially of the "why this and not that" type.

If you're a researcher in the free world, you can just ask. Especially if you have a public track record of great papers yourself, they will WANT to talk to you. You can learn so much more from the interactive process of back and forth questions than you can from a static piece of information like a scientific paper.

If you work for a secretive and command-driven organization, you need to be careful about what you reveal of your own research when you ask. You can't talk freely. The thought of having to justify your communication to some old-school IBM lawyer type, is going to chill even the most enthusiastic reseacher. It's easier to just stay in your own corporate bubble, and focus on the things your corporation does well since at least you can talk freely to your colleagues (although in really paranoid organizations like the NSA or old IBM, even that may not be true). But then at best you specialize, at worst you fall behind.

There is a dataset of 5 billion image-text pairs (laion-5b) scraped by various parties. This can then be filtered and used to train these models. Cost is a bit of an issue but there are orgs that have provided compute for open model training. And Imagen is nice because the text encoder part is already available and doesn't need more training, so it would just be the diffusion model components being trained. I'd guess we'll see a biggish training run starting in a few weeks.

I feel like the fact big porn hasn't poached talent and jumped all over this suggest at least 10s of millions. That said some for profit no-rules deepfake service for disinformation and illegal content has to be in the works.

In fact, this kind of reverses things, doesn't it?

Open source is built on the assumption that you can do more with source code than with binaries. In the case of AI models, the computed weights of models are what's valuable, and the source code used to achieve them is less useful.

Images actually used our open LAION 400m dataset plus 400m of their own.

On compute there is more than enough compute available to open source now via LAION and Eleuther AI to train these models, will just a bit of time.

How big is getty? I think this is a billion dollars business.

r/wtfstockphotos will be full of this

And medium posts about React hooks and Go generics are gonna be full of “Animated Drake meme but with Rick Astley” kind of fun.

They can make more profit if they don't need to pay artists

Definitely. With the right training set, video game assets too. Imagine being able to generate N variations of any asset in any video game style...

You'd have to train it first.

I have to clarify I am visioning DALL-E-2-PORN.

Couldn't you just scrape porn to get copious amount of dataset? Scope would be narrower and thus require less classification and in general it has common themes.

I'm more concerned how expensive it will be to train it on GPU instances. We are looking at A6000s right? That's like $15/hr.

And what is going to take to generate videos, not pictures?

They specifically mention in the Images documentation that one reason they haven't publicized this yet is due to porn or deep fakes risk. Assuming this works very well already.

This tech is actually pretty bad at human faces - it all appears unnatural and distorted. You get things like the occasional extra earlobe...

Lol. Not the same thing obviously but your question just reminded me of @robotpornaddict, a neural net that watches porn and tries to describe what it sees.

https://twitter.com/robotpornaddict

How can this be the most upvoted comment?

What you describe is already illegal on many jurisdictions.