Hacker News Reader

Comment by clircle | Hacker News Reader

100 years ago statisticians starting working on the theories of survey design and experimental design to clarify what types, how much, what mixture, and how to collect data. Machine learning people could learn a lot from these old, uncool theories if they want to approach data with rigor. That one can approach data collection and usage in a systematic way is totally lost now, it's just web scraping.

View on HN

Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows.

100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).

Some of them do, but others depend on labeled and augmented data. For example training facial recognition starts with images labeled by who appears in them, then you run a model to find the face in the image and crop to that, then another image tells you where the eyes are and you use that to rotate the image to make the eyes level, and that's the input to the actual facial recognition neural network. And that's for the "naive" approaches that don't have special handling for faces don't look at the camera. And the field still routinely has problems because it turns out that somebody's data set only contained certain lighting conditions, or severely underrepresented certain races, etc.

Not sure about image recognition but for the language models and audio space, semi/self-supervised methods are currently doing quite well.

That is a fairly brave statement given some of the research looking at how failed attempts at ML modeling fall apart [1]

[1] https://ai.googleblog.com/2021/06/data-cascades-in-machine-l...

This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.

I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.

A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.

you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data.

It is still taught at university if you do a master degree or PhD at a decent place.

But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field

Reply on news.ycombinator.com