1,357 karma · joined March 28, 2007
When I mean work well, I mean a family of models which are generally easy to describe as well as be generally easy to train. Nearly all neural network models fit this bill as a they can be described with a loss function. A loss function being something which scores how accurately the current NN performs with a particular set of parameters. This loss function is then usually differentiable, and thanks to autodifferentiation tools, you can obtain the derivative of this function with respect to some parameters in this function. With this derivative, you can go ahead and run SGD to find the optimal parameters. We have now both empirical and theoretical evidence that SGD works fairly well for optimizing functions, particularly the high-dimensional ones neural networks describe.
So, the insight is we need a name to describe all these models which are differentiable and nice to train because of that. So now we can say what really ties CNNs, MLPs, RNNs, etc is not some biological metaphor, but that these are all expressible as some loss function we can find the minimum of using SGD.
I don't want to trivialize compliance. Even ostensibly simple requirements are never quite that, and every second spent on them is time not spent on your product.
Tech has already and will continue to interact with laws/lawyers. At some point open source libraries will appear to streamline compliance. For now it sucks but ya gotta muddle through or call it a day.
Now if you want theory, I continue to recommend Algorithm Design by Kleinberg and Tardos.
Another lesson that it uniquely provides is that in day to day life you don't use these algorithms. You create new algorithms using the ideas behind the famous ones as a guide. An algorithms book or class that doesn't teach you that problem-solving mindset is basically useless.
Additionally, most online communication is intrinsically asynchronous. It's hard to have a chat online that's anywhere as fluid as what happens in person. Even the flurry of twitter replies doesn't quite compare.
Many ideas that seem easy to invent only seem so in retrospect. The article repeatedly points out that inventing rope isn't obvious.
https://www.numbeo.com/cost-of-living/compare_cities.jsp?cou...