nschucher

Maybe we can look for historical examples of how legislation had a positive impact on a widespread harmful habit.

  Early on, the U.S. Congress adopted the Federal Cigarette Labeling and Advertising Act of 1965 and the Public Health Cigarette Smoking Act of 1969. These laws—

  * Required a health warning on cigarette packages
  * Banned cigarette advertising in the broadcasting media
  * Called for an annual report on the health consequences of smoking

https://www.cdc.gov/tobacco/data_statistics/sgr/history/inde...

They trained a >600B parameter translation model (GPT-3 is 175B parameters).

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (https://arxiv.org/abs/2006.16668)

I'm as impressed as anyone with GPT-3 samples, but you're sort of ignoring the symbol grounding elephant in the room regarding language models (https://openreview.net/pdf?id=GKTvAcb12b).

Language models are not grounded learners. The language produced does not really correspond meaningfully to our world except in superficial (albeit complex) ways.

Do you have thoughts on how to move forward on this problem? Maybe ask GPT-3 and see what it thinks :P

They have published multiple major TTS papers since the original WaveNet paper in 2016. Including the recent Tacotron results with impressive style control.

Tacotron (Mar. 2017): https://arxiv.org/abs/1703.10135

Parallel WaveNet (Nov. 2017): https://arxiv.org/abs/1711.10433

Tacotron again (Mar 2018): https://arxiv.org/abs/1803.09017

Tacotron 2 audio samples: https://google.github.io/tacotron/publications/global_style_...

This is called federated learning[0] at least by Google. I don't know whether they've added this to more products or whether it works well. It would be interesting to see this done in open source.

[0] https://ai.googleblog.com/2017/04/federated-learning-collabo...

They're most likely referring to adversarial attacks where degenerate inputs are constructed that could cause AlphaGo Zero to perform sub-optimally or catastrophically fail (see OpenAI Research [0]). This is distinct from generative adversarial networks (GANs) or adversarial self-play (which I guess AlphaGo Zero is an example of).

[0] https://blog.openai.com/adversarial-example-research/

It was trained with self-play and evaluated on AlphaGo Lee (the version that beat Lee Sedol) and AlphaGo Master (the version that beat Ke Jie and I believe a number of online matches against top-ranked professional players).

>No one is ever going to use word vectors to figure out what genders are capable of what jobs.

How can you possibly make this claim?

Biased word embeddings have the potential to bias inference in downstream systems (whether it's another layer in a deep neural network or some other ML model).

It is not clear how to disentangle (probably) undesirable biases like these from distributed representations like word vectors.

Have you seen the results from Dynamic Memory Networks? [0]

The relevant example from the paper:

  I: Jane went to the hallway.
  I: Mary walked to the bathroom.
  I: Sandra went to the garden.
  I: Daniel went back to the garden.
  I: Sandra took the milk there.
  Q: Where is the milk?
  A: garden

Obviously just a toy task, but as you said, progress is rapid!

[0]: http://arxiv.org/abs/1506.07285

DeepMind is the name of an AI company that was acquired by Google.

https://deepmind.com/

AlphaGo is the program that beat Go. These game playing algorithms are variations of Deep Q Learning Reinforcement Learning algos.