Read it. Worth a read, especially for those not closely following the machine learning world.

The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most.

I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve. If someone like Google, with far more different problems to solve, can get this type of model to work and demonstrate its effectiveness, I think it would be a big step towards solving artificial general intelligence.

Jeff Dean has a lot of respect and influence inside Google, and his ideas tend to get implemented. I'm looking forward to it!