DeepMind AI reading list [pdf]
storage.googleapis.com
storage.googleapis.com
The list also includes very beginner things. Maybe this could be ordered as in a way to progress through subjects or at least something better than alphabetical (and have a section for "entertainment" which would include things like 3B1B, Lex, Robert Miles, etc, which are useful but not hard literature).
Additionally: Title should be "DeepMind AI __Resource__ List," many items here are not ones in which you can read.
Lastly, I agree that the title should be resource list.
On the other hand if you are trained in ML, most of these are not detailed or extended enough to give you anything useful. Doing “a hacker’s intro to X” over and over really, really doesn’t give you any skills. This is particularly true for deep domains like reinforcement learning, computer vision / image processing, and natural language processing.
Meanwhile, basic design of experiments for A/B testing, explanatory modeling and simple regressions is a fraught area. Not understanding extremely rigorous details about hypothesis testing, model checking, limitations of statistical significance, etc., can lead to wildly incorrect inferences from poor models that non-experts will completely fail to detect.
At best this list seems like “special topics to seem trendy in AI without getting deep / practical insight into any particular area.” There are one or two minor exceptions in the list.
It's just what happens to a field as it becomes more universally accessible.
It’s precisely the same with machine learning and statistical inference.
The lower barrier to entry just means the danger of releasing extremely unsafe projects is much higher, and the careful validation from experts is that much more critical.
(If you say something like, “just look at the business outcome of the model” you are proving my point about the danger, because that would be a catastrophically bad way to judge the performance of a model. What accuracy metric did you use? Why? Did you understand training / serving skew? Did you look at a confusion matrix or study class imbalance? What about missing data? What statistical test did you use? Did you adjust for multiple-testing? Was there peek-ahead bias? Did you test for discontinuities and non-linearities that can render p-values inapplicable for tests of linear models? What was your training convergence like? What simple baselines did you compare to?)
For example, if I provide some automatic assignment of let‘s say a related object, reducing the work required to find the correct one and the team using it is much faster, then I can measure this and that‘s enough. Likewise for, let‘s say a recommender system. The BI department will look at conversions and that‘s in fact the only thing that matters. Nobody cares how good the model is. What we care about is whether the recommendations have the desired effect in that case.
And besides, what‘s hard about measuring how accurate a model is for predictions? Compared to most things in a typical CS curriculum that‘s rather easy indeed.
Finally, what‘s „dangerous“? ML is usually applied in a business context. Better recommendations; automatic assignment of objects. Automatic classification of pictures. Things like that.
Sure, in a medical context perhaps or financial. Then it may be dangerous. For everything else, maybe you get a worse result. That‘s not unlike everything else in software engineering where many people with different skill levels get very different results (some better some much worse).
I have this anecdote of a industrial sorting/production machine that worked for literally a year until someone opened the door on a windy day and everything went flying thanks to the AI.
Please can you give a study list that would be useful for one trying to ML?
In practice, comments like this encourage self-censorship, even if that is not their intention.
Understanding and reducing negative impacts of bias is important, but the current research field of AI fairness does not do anything like that and has not yet reached any state of maturity where it can be considered a serious subset of research at all.
I would really say even one mention of it on a list like this is purely to do virtue signalling.
It’s the same for “explainability” of models too, another totally bogus field that gets treated as being worthy of attention or societal prioritization purely due to politics.
of course there are a lot of papers of questionable value, but that's a problem with machine learning more broadly -- probably all academic science, really.
lots of institutions (banks, medicine, etc.) really do need explainable models. in practice this often means linear models with simple coefficients or shallow decision trees. there is plenty of useful work on learning these while maintaining performance, or "distilling" them from more complicated models. i know for a fact some explainable models learned with these techniques do actually get used in real life.
likewise with fairness -- end-users actually do care about fairer models in all kinds of areas, especially lending and insurance. there's a ton of frustrating debate about how to operationalize fairness in different settings but it seems like there is actually progress on this front.
what do you find to be bogus about these research areas?
Meaning, you would believe the coefficients reflect a real relationship between the covariate and the target, and even could claim it’s statistically significant, and yet the actual relationship to the target is of the opposite sign! Any further feature importance scoring based off the coefficient estimates would then become catastrophically misleading.
Meanwhile, a model like support vector regression on the same data is capable of automatically handling the non-linearity, at the expense that there’s no more such thing as a coefficient breakdown in the linear space of input features.
Does this make it less “explainable”? That would make zero sense. How can it be worse at explaining a data generating mechanism when it is better at predicting that same generating mechanism. What could it possibly mean to explain something you can’t predict or replicate?
The field of “explainable” models doesn’t even make the slightest attempt to address this stuff - it just beats up on models that are arbitrarily labeled as “black boxes” (what does that mean?)
If a given model X predicts or replicates a data generating process better than Y, then X explains the process better than Y, period.
An analogy: Newtonian physics is not “more explainable than” quantum mechanics. Newtonian physics is just more wrong about how the world works than quantum mechanics.
You may have to explain your decision to a judge or customer after the fact, or you might even be asking a layperson to actually compute predictions manually (as is sometimes done in psychology and medicine for things like triage).
the question of whether or not the model provides a good explanation for the data-generating process is a distinct one. I think the Achen paper makes a good point that people cannot safely turn linear regression coefficients into stories about the world, although it doesn't seem like they've stopped trying.
but assuming you have a way to validate that it makes good predictions (not an unreasonable assumption), an explainable model can be a useful thing to have.
“I can explain how this model works” is not something you can claim about a model. You can only claim it about a tuple of (model, assumptions, data, context).
In context A, some simple linear regression might be very “explainable.” In context B, that same linear model might be totally not explainable (because the mechanism of coefficients based on the regression’s fitting procedure might be totally incompatible with other details of the situation.)
The analogy between Newtonian and quantum mechanics still holds.
The fact that you can more easily map Newtonian mechanics onto English words or pictures absolutely doesn’t make it more “explainable.”
By that logic, saying “a magic wizard did it” would be the most “explainable” model of all.
This is exactly my point. What is “complex” or “simple”? Arbitrary standards of natural language words? The field of explainable models has done no work on this. It starts from some totally arbitrary and confused idea about something both being an accurate model and an explainable model as if they are separate.
The closest academic topic to making “explainability” a serious subject would be the philosophy of language and connection to computability theory, like Kolmogorov complexity, PAC learning, VC dimension, Occam’s razor.
But these are algorithmic aspects of model complexity in the face of a specific data set, it’s absolutely not some hand wavy “oh but a person could ‘easily’ understand certain verbal acoustic vibrations about this” based on nothing.
I'm thinking about this pragmatically: a model is explainable if you can explain the model to some people you need to explain it to. I mean this in a fuzzy, "arbitrary", natural language sense -- the test of it is to try to give the explanation. I don't think metaphysical questions about the nature of explanation itself are really relevant.
Whether this is true definitely does depend on what data, which people, what context, but it's possible to try to make techniques that will be generally useful in many contexts.
Getting into the broader issues, I think Newtonian physics is absolutely more explainable than quantum physics. For example, I've seen court cases around car accidents where consultants had to come in and explain basic mechanics (F=ma, friction, etc.) to the jury without reference to any math; and they did an okay job of conveying the essential ideas. By contrast I've never seen a math-free explanation of quantum mechanics that wasn't a complete mess.
Of course, "a wizard did it" is even easier to explain, and may be appealing for that reason, but ultimately that model will turn out to be totally useless.
For example, let’s say someone starts asking “why?” every time you make a statement about the car accident in the court case. If it’s Newtonian physics, eventually we bottom out at questions that require statistical mechanics to answer and it falls apart. The “explainable” answer is unmasked as a fiction that is incompatible with reality.
So then you might say from a pragmatic point of view all that matters is that the jurors were happy with the explanation. But then who decides what that metric is? What if Feynman is on the jury? What if a medieval religious leader is on the jury?
The standard of “explaining it to people” is not a thing.
This just circles back to exactly what I said before, which is that explainability becomes politics.
You’re trying to say it’s pragmatic, but who gets to control that standard?
Why do banks need “explainable” models? Because of some political fight about who can regulate them. Now you’re trying to convert that arbitrary political goal into some type of formalized standard of machine learning models, which is totally intellectually dishonest and rigged.
If a surgeon uses a computer-vision-assisted robot to save lives, does the robot need to be “explainable?” Who decides what that even means? Is it a political standard because of insurance liability? What if a “smart” person can understand the model but a “dumb” person can’t? What if the model deemed more explainable by a political oversight committee also saves fewer lives, thus condemning people to death for the sake of explainability.
I just don’t see any way your appeal to just some “reasonable” or “pragmatic” idea of “explaining it to people” can be carried to a logical conclusion that makes any sense.
The central dangers, apart from misuse, seem to be a coordination problem and a control problem.
The coordination problem is that AGI will likely lead to a winner-takes-all scenario, implying a per-emptive strike becomes rational once one player seems too far ahead of the game.
The control problem is that the value function of an AGI may diverge from our own value function.