I did address those earlier in terms of association, I'm not sure if I understand the aggressive tone of this response however, or of a few of the assumptions in it. A few other people have made similar points about this culture as well.

The fluidity of language is one of the most fascinating aspects of it! English isn't English. It's much more narrowly defined with a geographic and temporal context. Meanings of words and phrases shift over time. Idioms come and origins are forgotten, but it's all folded into cultural dialects. Each generation is able to have their own mark on language, and there's probably a story told in the evolution of the English language across time that maps out the cultural and geographic changes as well. Now I'm curious what sort of research has been done into this. Regardless, the point is language changes. No one can stop it. Raging against it just reveals that you're old and out of touch. After all, I doubt those people raged against the slang of their youth. It's always the Next Generation that is The Problem.

I'd like to create it in the future, but I can share a rough version of what I have. Basically, as long as you gain more information than noise by scaling your experiments down (i.e., the smaller models create similar directions in the needle), then it's easier to run lots of experiments, and much more cheaply oftentimes too. Remember that performance oftentimes runs on a log(1+p) curve in terms of cost/time/complexity vs reward.

If you write your codebases to minimize the expected time to make a particular change (fewer files, simpler proxy problem [as long as it transfers to your larger-scale problem!], more 'flat' structure, simple dataloaders, etc), then that is extremely valuable as well. What you are optimizing is the average time from idea-to-answer.

What is nice is that experiments go from strongly-planned, biased (from human beliefs, etc) things to a more high-variance process where there isn't as much requirement to double-and-triple check everything before running it. There is often an _extremely_ poor mismatch between the very first impression of what many of us think will work and will not and what actually works when the dust settles. Sorta like how SGD bounces around the loss landscape before settling in.

One thing that also is easier with fast experiments is to make the 'opposite' change if a planned change doesn't work out well, just to see how it goes. A tiny thing, but if you're going sequentially instead of batched (in terms of batched experiments) it can be useful.

I use as few libraries as possible. A lot of the bulky monitoring solutions can be good for some enterprise things, I don't find they're good for innovating, but instead, integrating. You innovate on small test problems that are appropriately representative. When done, you integrate and see what difference it makes.

Doing dumb things is one of the best ways to learn. Having access to everything to be able to print it out/log it/look at it in a chart is great too.

But everything is bottlenecked by speed. Increase your experiment speed, where (critically!) your speed includes the time from idea to first answer coming back (and not just plain ol' runtime), and you'll find oftentimes it's easy to get 5-10x research speed improvements. We just don't do research all that efficiently these days, I don't think. But we can. And I'm sure we will, eventually. :3

There are a few things like scaling that can be difficult, but math is still math, and if you pick your proxies right, it should be an explored-translation-step after the proxy problem behaves as well as it is reasonably able. And if you have a secondary, slightly larger proxy problem, that's an even better thing in my experience, since it prevents the sticker shock of broken graphs and things failing for an unknown reason. All about how well you can pass the peach from one person to another. :')))) <3 <3 :')))) :'D :'D

That's a great point, I hadn't really thought about that a lot, and definitely had not made the hard connection.

I do love how he continues to spin out new potential modalities for Deep Learning. While underrecognized due to the hype today, I think we darn well need far more of it these days! :D :))))