A Critical Field Guide for Working with Machine Learning Datasets

This hasn’t been my experience. I don’t think this knowledge has been lost and it’s often still applied when incentivized, such as when quality output of machine learning models serves an immediate business need, or in scientific applications of machine learning. By your reference to web scraping I assume your referring to datasets for large language models, but I don’t think the kitchen sink approach is applied out of ignorance, but out of a principled decision to forgo data hygiene in order to collect datasets large enough to feed such large models. Also, strictly speaking, one only needs to apply the critical approach when constructing and using validation datasets, and it’s typical for larger noisier training datasets to outperform smaller more rigorous ones when models trained on both are tested on carefully constructed validation datasets.

A system for this would allow individual collections of documents with discussions attached. Without people, the machines won’t know where to put attention. That may change when they have individual identity, but for now the work probably needs to be done cooperatively.

Machine learning (or more specifically deep learning) is shaking the world precisely because the models at the highest end are able to take in mostly unlabeled garbage and spit out useful results in an unsupervised manner. The world is not full of i.i.d. spherical cows.

100 years of statistical advances have not been able to solve image recognition at scale until convolutional neural networks. Large language models may be somewhat Bayesian but they do not resemble any traditional Bayesian techniques except at the highest level. Clean data and good sampling are incompatible with the real world outside of laboratory conditions (or large budgets).

you are too pessimistic. Senior profiles definitely dived into the statistics aspect of it all. Experiment design is often a big part of modern business cases, because naturally it is part of the cost to obtain data.

It is still taught at university if you do a master degree or PhD at a decent place.

But I agree that many juniors have no clue. Especially those who are self taught and join in from a different field

I definitely agree that taking a systematic approach to data collection and usage is incredibly important. However, I think it is impossible to simply dismiss the ways that modern machine learning has revolutionized the way we understand and interact with data. There is no question that rigorous survey design and experimental design theories still have merit and should be taken into account, but I think machine learning has allowed us to delve deeper into data analysis and uncover patterns and insights that would not be possible without its advancements.

If you want to perform vector search over your data, then Qdrant (https://qdrant.tech) is worth checking out.

We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames.

Would love to get some feedback on it!

https://github.com/Oxen-AI/oxen-release#-oxen

"ML data" could be just about anything: images, sounds, graphs, tabular data, vectors. Depending on what your data is, you might end up with very different storage solutions.

Though I think one common approach is to just dump most data in an S3-compatible datastore of your choice (there's seaweedfs or ceph, or the cloud provider of your choice). Specialized databases make a lot of sense when you need features like vector similarity search though.

Vector databases in general are good for storing large amounts of unstructured data by first converting them into embeddings via ML models. There's also feature stores, which store and organize features for later use in model training or predictive analytics. Feature stores generally come in _before_ models get trained, while vector databases generally come _after_ (i.e. they use trained models).

Milvus (https://milvus.io) and Feast (https://feast.dev/) are two of the most well known vector databases and feature stores, respectively.

+1

I didn’t realize it wasn’t a “real” spreadsheet application until your comment.

The design gave me a bit of anxiety, it was so convincing I was worried about clicking into one of the cells!

Anxiety included - I also love the layout.