Some thoughts on machine learning with small data
niklasriewald.com
niklasriewald.com
The advantage of physical systems is that it's straightforward to collect data from natural systems.
For a lot of physical systems it might be "straightforward" but expensive. Testing new building designs for energy efficiency is just a matter of building a new test building, why not just build 1000? Testing new car designs to see if they reach the safety qualifications is just a matter of crashing a few of your prototypes, why not crash 1000? It's just a matter of getting more telescope time. It's just a matter of getting more robots to try more materials or proteins. Yes you can do theoretically do it, it's just cost prohibitive to get Google-level datasets for many scientific and engineering problems. Of course that's not true for all domains (high-throughput sequencing for bioinformatics is the prime example of something that became cheaply data-rich for standard machine learning), but there are many domains where getting another data point is always possible but just would cost another million.
High throughput sequeuncing didn't solve any problems for bioinformatics- in fact, the problem is that we have so much data we don't know how to process it to extract the actual value from the data.
Bioinformaticists, with sufficiently accurate and detailed models of how genes translate to $BEHAVIOR_OR_OUTCOME, would need far fewer data to extract meaningful insights. So of course throughput hasn't solved the problem. That wasn't the point of the comment above.
Really, I was making a comment about https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... which in my mind is still unresolved (that is: why do the math models we use for physical systems generalize so well, if we wouldn't expect them to do so?)
The disadvantage of physical systems is the data is usually expensive. In my old field I know someone who had to make a decision on a data set of 5 samples and each sample costed more than a million USD.
I've made models of physical systems that involve kind of life or death stuff with less than 30 samples.
It's not for the faint of heart and if someone doesn't know what they are doing, the system they are modelling, or how to share their results correctly... Yikes!
Are you sure you can conclude your system is safe?
Bias in a lot of contexts is Bad, and large data techniques like DNN are basically about how you can have biases that are so vague the learner has plenty of room to surprise you. But when you add bias that truly aligns with the structure your learner is going to be exposed to, you enable it to reach confidence sooner. Many important techniques for small data are about allowing you to express specific biases. For example:
- PGMs and Probabilistic Programming are about giving you a specific, interpretable structure for how your data are related to each other.
- Picking informed bayesian priors
- Data augmentation lets you add bias by expressing what kinds of variation don't matter (I.e. permitting and fuzzing your dataset)
- feature engineering is about selectively adding and removing biases in terms of what kinds of data transformations are informative
With all that said, the large data ML community has produced an incredible tool for practical work with small data: transfer learning. If your dataset is related to a larger one (e.g. by including natural language), you can borrow informed biases from models that were trained on a much larger corpus.
What does ”permitting your dataset“ mean? Is it a typo? Did you mean ”permuting“?
How much data is considered small? What is the complexity of the dataset itself?
Even in Machine Learning it is possible to learn from small datasets without transfer learning. See meta-learning for instance.
It's very important for anyone studying ML to understand how bias-variance relates to sample size, so I'd encourage finding more resources if this note didn't help clarify! Here's another shot at a summary: for a fixed size of training sample, you must trade off between sensitivity to randomness in the sample (variance) and assumptions that bias the model you train.
It's true that quantifying variance and bias can be hard, and you need systems like PAC learning to go further and actually estimate sufficient sample sizes for a task. But you can still reason usefully about any system that involves using data to select (train) among a class of potential output models!
For example, the statement about meta-learning is incorrect, at least as far as i've seen the term used. Meta learning involves learning hyperparameters (including functions) that are then used to train a model. The extra stage makes these models less biased, but actually require more data. (Of course, in some meta learning systems, hyper parameters are learned with the help of external data - a form of transfer learning.)
Transfer learning can be immensely useful when applicable, but many times it's not (e.g. imagine a medical domain where you track a patient through some process, recording procedure choices, measurements and outcomes, etc., where it can be difficult to find relevant data elsewhere).
Some approaches I've found useful:
* Get to know the domain really well
* It's not a lot of data - that's a potential for rich interactive visualizations that allow you to get to know the data quite well, and grok how it relates to the domain knowledge
* Following the advice that ML models in production could/should start with simple heuristics, view your model more as an augmented heuristic than a powerful model to solve everything - that means also figuring out how to catch and handle cases where it's wrong (which is something one ought to do anyway)
* Invest in tailoring priors suitable to the problem, based on your domain knowledge and understanding of the data. This can range from writing your own loss function to training an ad-hoc type of model, not based on DL, e.g. using metaheuristics (genetic algorithms, simulated annealing etc.). The advantage of small data is that evaluation on it can be relatively fast and ad-hoc models using nonstandard techniques can be realistically optimized (sometimes, depending on context of course).
I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.
It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.
There's much more money to be had convincing startups to use big data infrastructure and train huge nets on GPU clusters etc. than convincing them to iterate on small ad-hoc models developed in-house. Not saying that big data or models are necessarily wrong for small startups, just that they don't have to be the default.
Isn't this a problem anyway, even with models based on much larger datasets?
What I meant to say was that retraining your model might not fix that if you baked strong assumptions into it. I edited the blog post to make it clearer.
Out of curiosity, did you apply the constraint as a term in the loss or did you use constrained optimization?
I've only seen (and used) this for linear / convex models and constraints, so that's actually linear / convex programming (highly recommend the cvxpy library [0]). I was curious if you've integrated that into DL models.
Here's a fun example I did for a uni project: say you have a small recipe dataset, where for each recipe you have the ingredient list ("3 tbsp sugar", ".5 kg flour" etc.) and macronutrients (carbs, proteins, fat), and you want to learn to predict the nutritional content given a list of ingredients.
With a bit of text manipulation you can split each ingredient text to "amount", "measurement unit" and "ingredient type". Then you can decompose the nutritional values into nutritional density of ingredients and the conversion of measurement unit -> actual mass for each ingredient type. Then you can introduce constraints in the form of known conversions between units, e.g. 1 cup = 16 x tbsp, known nutritional densities of some simple ingredients, and known unit conversions for specific ingredients.
It worked better than simple regression (that didn't take into account unit conversions) and a simple MLP, though not sure how it would compare if you actually tried to finetune a language model to the task. The overall predictions weren't too accurate, but for common ingredients it actually gave very close nutritional density values (without being constrained), which was cool.
Edit: another constraint I initially forgot about is that each gram of an ingredient cannot contain more than a gram of macronutrients (it can contain less because water), and of course nonnegative nutritional density.
The breakdown you give above seems to me to be more akin to something DL types tend to call 'feature engineering'. I also have a fun example, in this case it would be identifying land cover from satellite imagery. You can obviously just feed the raw reflectance values (RGB etc.) into a DL model to create a semantic segmentation of classes. However, it's been well established in literature at this point that that is not the most effective way to create a classification. This is similar to my previous comment, where there are lots of solutions that can be found through SGD based on these raw values.
There's a lot of traditional satellite imagery analysis algorithms that are based on very simple 'band-ratios', i.e. NDVI (normalized difference vegetation index) is calculated by Near Infared - Red / Near Infared + Red. This index will visually highlight areas of vegetation that was extremely useful in human sight-based analysis to identify vegetative areas. Now, you'd expect a deep learning model that takes in all the bands to have this information already, it has the NIR band and the Red band. However, explicitly doing the NDVI calculation and using it as an input feature leads to increased accuracies for classification. The exact reasons for this are unknown, but I think you touched on some of this above. With machine learning, and DL even moreso, sometimes it's necessary to hand-hold the optimization to optimize for exactly what you want. It helps 'explainability' and oftentimes helps accuracy, at the cost of some preprocessing.
I teach Data / AI courses regularly and this topic is a recurring one with students. We the instructors like to make students start without even ML in the first place for them to construct a baseline model, usually using simple heuristics (e.g. conditions, regexes, etc.).
At first, students don't like it and tend to skip it because it is not that impressive, but in the end they learn so much about the data and the problem at hand! Then they are allowed to try simple ML models to see if that's a performance improvement and so on with more complex models. In some cases, improving baseline models with complex ML/DL models is really hard (e.g. time-series). Another benefit of simple models is explainability, for which the industry demand is growing everyday.
Can you elaborate?
A nice keynote by Vincent Warmerdam [1] talks about tips and tricking for advanced feature engineering combined with linear models.
Heck, in a large number of domains, simple ratios -- debt to income ratio in finance for example -- will dominate the feature weight for many models and can be used on their own as a pretty good heuristic.
It's a much harder area to hang out in, and pays way less unless you are a world expert consultant. Stakes are often higher. I used to do this stuff, but there's so much hype around ML it's basically impossible to survive in industry even if you do very good work.
For example if you create a model catered to a system and it takes two weeks but is as correct as it can be given what's known about the system, your manager will turn their chin too it while trying to find startups to buy(yes buy the company) that claim they can solve it with deep learning. The startups run away once they learn each sample costs 10k USD or deliver a boosted decision tree model that under performs, but it got so old I changed careers.
One side note, when I speak to other Humanities researchers about this, I always tell them that I have yet to find a technique that will give them novel insights. These techniques almost always tell the researchers things that they already know. I usually follow this up with a note that even formalizing Humanities knowledge in statistical or other computational terms is highly valuable and worth doing. Maybe someone else can take that formalism and build on top of it something truly new.
In statistical learning/forecasting, the researcher typically specifies the statistical model.
In machine learning, the statistical model is approximated by the algorithm.
Since a ML model needs to learn both the model form and the model parameters, it takes more data and also it does not allow for understanding (since it does not output the form of the model it learned).
Also, because you've assumed a structure to the data, you can more easily test if the data has deviated from that structure. This can be data drift or single outliers-- for example, GARCH models (a type of regression) allow the normal distribution of the error to have a varying variance, so you can detect different variance regimes.
In short, they help a human understand and interpret data.
From what little I know, ML is not so good at that. But it has other advantages, and you don't always need or want the understanding. If your want to i.e. detect ground cover in satellite images, then all you care about is valid outputs, not necessarily the importance of near-infrared vs red band.
And ML (can) beat regression models by providing better interpolation, by better handling regions of the data space which violate the assumptions of the regression model, etc.
So it is a tradeoff. Both approaches are highly performant, just at different tasks.
Some of the regression coefficients were negative, which means that increasing performance in, say, long jump, leads to shorter shot-put (I don't remember which throw the model was for) distances.
The model and approach looked understandable and "weird" at the same time. From a purely statistical perspective it makes sense, since that was the results coming from, I assume, maximum likelihood estimation. From a predictive performance, retrospectively it surely worked because it gave good prediction on past data, assuming there were training and test data (most likely, they were not, but let's assume).
But from a future prediction perspective, i.e. the forecasting and thus the manipulation of training to obtain a certain performance, did it make sense? I am very confident it did not, because, among other things, the performances of auxiliary lifts/feats were not independent (you cannot work on a heavier one rep max in the power clean and hope or work toward a shorter long jump performance).
The model by itself might have accurate, but considering that interpretable and thus guiding changes in the training program would have been a quite naive mistake. This kinda mistake is quite common among many who think too much about the machinery of the model and way too little about the domain.
Those are two different questions. For forecasting without manipulation of training it would still make sense. But it wouldn’t make sense for causal analysis.
I believe that (1) the model made little "mechanistic" sense, (1) the forecasting accuracy of the model would have been low, (3) the model had good hind-casting accuracy through overfitting by modeling the "noise" in the data with too many predictors. (4) the model could have not guided any training.
Normality will give you stronger results, but generally linear regression doesn’t require normality. You only need errors to be uncorrelated for OLS to be the best linear unbiased estimator.
Yes, but sometimes in surprising ways.
I build a simple decision-tree model for a medical study, looking at outcomes for acute pneumonia. Went with a single tree over a forest because the model had to be interpretable. Statistically it was almost as good as the forest; I built it using fields with high feature importance values. Thus there is a chance that any 'improvement' by the forest was overfitting. but I digress.
The tree said that blood CO2 levels were the most important factor. The doctors weren't surprised by this (though they had some internal debate if this was more or less important than some other factors). What did surprise them was the cutoff level.
They said they would be concerned if CO2 was above 7. My model had the cutoff at 9.5. Sorry, I forget the units.
Point is, it confirmed what they knew (CO2 levels matter when assessing lung function), but still surprised them (CO2 levels have to be much higher than normal before this becomes discriminant over other factors, such as age).
2. Augmentation
3. Heavy regularization
4. Cross-validation
5. Transfer learning/fine-tuning from related data
6. Sharpness aware optimization
7. Ensembles
8. Any domain knowledge/feature engineering
10. Single-shot / low-shot learning methods
11. Reducing the scope of the problem so you don't even need that much data
12. Iterative model approaches that use submodels focused on different target problems (instead of trying to boil the ocean with a single model)
A lot of these are not what you do, but how you do it, something popularized by CPMAI methodology.
I would also add that I'm not sure the OP was tongue-in-cheek when saying that AGI is coming, so we don't need to worry about low data requirements. But needless to say, AGI is not coming any time soon, and if OpenAI's strategy for AGI is to be believed, it's anything but a low-data method. Unless you're counting on them to build the super-sized model that meets all needs.
The real problem that I see is that people don't pick a task that is granular enough.
github.com/aiqc/aiqc
Is there a good way for a given problem to estimate how many examples would be required to produce a decent model?
Small data depends on the domain/context. It might mean < 30 observations or it might mean < gigabyte scale.
For example, in a tree based regression with a very small number of training samples, the leaves will almost resemble some of the training samples. A good generalization might not be possible at all. But this is not too bad if you know that your prediction samples have high similarity with at least one of the training samples. In the end, this is an edge case for Machine Learning but goes more into the direction of Expert Systems.