lucidrains

It's sobering if your culture had over-romanticized the ability to do so for centuries.

Another work in the opposite direction, introducing gating in Transformer-xl https://arxiv.org/abs/1910.06764

On the other hand, it must be sobering to realize that even with an entire lifetime of practice, you will never be better than a simple residual network trained for less than a week.

Yes. The last one was 8.3B https://arxiv.org/pdf/1909.08053.pdf

They are going commercial soon, so we shall see how well they stand up to different open world environments.

Oh! You are right! How did I miss that..

Hmm, no mention of Megatron in their timeline? https://nv-adlr.github.io/MegatronLM

Love it when the code is presented up and center as in Figure 3!

Awesome, thank you! I was waiting for someone to do this! :)

Great job! Could you share how many parameters and the amount of training data per language you used?

Afaik, it is parsed, but limited to the current camera view

This has already happened to some degree. The technique is so easy, kids out there are using it to immediately renovate old games overnight https://www.theverge.com/2019/4/18/18311287/ai-upscaling-alg...

Language, music, and now amino acid sequences. Attention is all you need.

Yeah, I know what you are saying... But let's just let somebody try this experiment (and somebody eventually will), and we can judge what can or cannot be learned by the results.

We will definitely get a great code autocompleter at the very least..

I've tried it as well and got good syntactic results. For more sensical programs, I think we will need more layers & attn heads. Perhaps someone will fork gpt-2 and add the sparse transformer to it.

Given how easy it is to train a Transformer on any sequence data, and given how plentiful open source code is, I'd say "CodeNet" is probably less than a year away. OpenAI will probably do it first given they already have the setup.

Yea, somewhere in between the released model and the model they presented, the deep net has learned narrative structure.

Started a Google colab with the interactive text generation script. https://colab.research.google.com/drive/1da54684tFMjPbR5idbv...

Thanks! Initially, I thought about just generating a big batch and cycling through it. Then I thought it would be more dramatic if the machine was "dreaming" up a face every 2 seconds in real time. I went for the dramatic approach just so I could phrase it that way to my non-tech friends!

Looks like someone else has already used another neural net to find our president in the latent space. https://github.com/Puzer/stylegan

and it seems like someone already did it as of 9 hours ago https://www.reddit.com/r/MachineLearning/comments/aq6jxf/p_s...

Haha, having to work around the computation limits are welcoming! It feels like building web apps back in the late 90's again. These days we have so much memory and disk space at hand it doesn't even feel like a challenge anymore.

That is, until Graphcore delivers their IPU.

you'll have to build an encoder to encode someone's face into the latent space. then you'll have to dive into the latent space and find the dimension(s) that controls for hair style (just fork the colab and start experimenting with interpolations)

then yes, it should be possible

you can download the training set here https://github.com/NVlabs/stylegan#resources it's based on 70k high resolution flickr images. try interpolation on the colab link above so you can be convinced its capturing the features rather than just memorizing

Great job on the Danbooru training! I've been following you on twitter and machinelearning for the longest time haha

no not naive at all. this method isn't specific for just extracting features from faces. it can disentangles features from any kind of images. in fact, the next dataset i might train on is on flowers (or birds)

https://twitter.com/hardmaru/status/1095639937842638849

if we train a reverse encoder of the output of the GAN back to its own latent space, it's probably possible to find your face and mess with it.