Hacker News Reader

Comment by roenxi | Hacker News Reader

Machine learning isn't comparable to software development. It is a statistical modelling exercise. This is like asking why advertising is hard - if a non-expert wades in to a different domain then they may find it has different challenges than what they are used to! This is just a specific case of the normal things that analysts routinely deal with.

The major challenges in this youthful field of machine learning are building appropriate hardware and making it work. That, so far, has kept it the domain of the software engineer. As the situation continues to stabilise this is going to become the playground of statisticians and analysts.

Or to put it another way - if you compare any field to software engineering, the problem is that other disciplines have a much harder time debugging things. Software is almost unique in that debugging is cheaper and quicker than building things right the first time.

View on HN

> Machine learning isn't comparable to software development. It is a statistical modelling exercise.

It's neither of the two. Machine learning isn't comparable to any other human endeavor because in many cases, much more value comes out of the models than (seemingly) goes in.

LLMs for example are punching way above their weight. The ideas underlying their software implementations are extremely simple compared to the incredibly complex behavior they produce. Take some neural networks that can be explained to a bright high schooler, add a few more relatively basic ML concepts, then push an unfiltered dump of half the Internet into it, and suddenly you get a machine that talks like a human.

Obviously I'm simplifying here, but consider that state-of-the-art LLM architectures are still simple enough that they can be completely understood through a 10-hour online course, and can be implemented in a few hundred lines of PyTorch code. That's absolutely bananas considering that the end result is something that can write a poem about airplanes in the style of Beowulf.

I'm not sure it's as simple as you make it sound.

Lots of problems have very simple solutions. And progress often means finding simpler solutions over time.

But coming up with those solutions, and debugging them, is what's hard.

For a comparison, have a look at how pistols got simpler over the last two hundred years. Have a look at the intricate mechanism of the P08 Luger https://www.youtube.com/watch?v=9adOzT_qMq0 and compare it to a modern pistol of your choice. (And the P08 Luger is already pretty late invention.)

Or have a look at modern Ikea furniture, which can be assembled and understood by untrained members of the general public. But designing new Ikea furniture is much harder.

I dont feel like im doing statistical modeling when i do ml. Usually feels more like pipe alignment, followed by tremendous amounts of debugging.

That is also what statistical modelling feels like. EDA and data cleaning.

It depends on what is it you do when you “do ml”.

The heuristic I use for distinguishing between statistical modelling, machine learning and AI is is through feature engineering and model specification:

- Statistical modelling: Manual feature engineering, manual model specification (y = ax + b)

- Machine learning: Manual feature engineering, automated model specification (y = ax + b or y = ax^2 + b, I don't care, the algorithm should figure it out).

- AI: Automated feature engineering (e.g. CNN), automated model specification

IDK about this. The model y = ax + b is not specified, parameters a and b have to be chosen by optimization. Now add regularization that some of those parameters are shrinked to 0 (lasso) and you have "automatated model specification" where only some parameters are left and others discarded.

And furthermore the models are always chosen from a predefined hypothesis set, so there can never be truly automated specification.

Reply on news.ycombinator.com