Machine learning is still too hard for software engineers

Author of the blog post here. It was definitely written from my narrow viewpoint and experience. Our goal is to make more solutions accessible to software developers and instinct was the same as yours - a lot of ML can be within the realm of engineering (even small / one-person teams) and that there are accidental complexities standing in the way of wider use. Our solution (AutoML+SaaS) def doesn't work for every situation. I'm curious to hear more of your thoughts on how ML can be made more accessible to Eng (and vice versa).

I don't think so. Or more precisely, they might look different academically but you need to have both as a skill to build something useful.

ML is kind of like tennis in the sense that you look at Nadal and Federer and all the greats and you can say, "man, I could do that," but you can't, not even close in this life and the next after this one.

In fact, most SWEs who have developed some sort of "intuition" about ML have been quite dangerous, as they tend to be condescending to the real experts, have built things that make no sense at all or fall apart when 1 data point our of 100,000 changes, and when presented with the fact that they have no clue about ML, resort to the "the AI is all hype anyway" comeback. And vice versa (ML practitioners like me who think they can do SWE with one eye closed, "what's the big deal?") is also true.

Making a successful transition to ML in my opinion, depends a lot on the individual. Without a strong background in calculus, linear algebra and statistics, it's going to be difficult. Training a model is what people tend to focus on, but in my opinion, that's the easy part. Evaluating/validating a model, analyzing and preparing your data, anticipating model performance, understanding what to do to improve your fit, model selection or architecture. Developing custom deep learning architectures at times requires a bit of an abstract mathematical intuition that I think will suit many engineers very well. A lot of engineers are well equipped to be successful in making a transition, but on the other hand, at least as many aren't.

In the future, I think the field will have many varying degrees of expertise, with the barriers to entry becoming lower all the time. We're reaching a point where some common use cases can be solved adequately in a nearly automated fashion. Some "autoML" tools don't really require any real understanding of ML, though I think it's not wise to get in the habit of using them without understanding how to evaluate a fit. These tools will be great for people who want to occasionally use ML to solve some smaller problems, but as a part of their larger job function.

In some middle area, ML engineers and practitioners will be training and operationalizing models, and keeping up with major developments in research. But there will be some significant changes in the next decade. I predict the nebulous of data science, data analysis and machine learning will become formalized into 3 major skills - exploratory data analysis, machine learning and advanced computational statistics.

At the lowest level, researchers will continue developing the field, which like you say, is probably not something you transition directly into.

So many organisations can't even get the basics of efficiency and custom service right but they still invest heavily into cutting edge tech. I can imagine plenty of companies joining the ML bandwagon and still not even really knowing what they do as a business.

oh shit, are we all supposed to be now?

Assuming you mean coreutils. binutils is for managing/inspecting binary executables.

But to your point: there were two key innovations and criteria of UNIX pipelines: a common and understandable data format, and writing programs to send and receive anonymous data. Crucially, the input and output formats were the same: plain text, separated by newlines.

In contrast neural networks are applied to a variety of data formats. Images, video, audio, text, social networks etc. each with their own encoding into something an NN can work with, with varying dimensions, features, metadata etc. So it doesn't make sense to bundle them as 'neural utils' but rather utils along whatever pipeline already exists, like GraphicsMagick. Which does leave a huge blind spot for the domain transforms like text recognition.

If you stay within the AI ecosystem, you _can_ set up reusable layers for tensorFlow, but typically you cant swap out something in the middle without retraining all layers below it. Which you might treat as a violation of the anonymity criterion, since the behavior / performance of a layer relies on the specific behavior of those above it.

This bears more resemblance to traditional manufacturing actually. I think there may be some value in borrowing ideas from statistical process control, rather than trying to force predictions into deterministic cases.

Sorry, it had to be done.

I think that is why people find ML so hard. It isn't a single sausage machine to create insight from, it is a set of entire stacks from philosophy all the way down to the electronics.

Also check out Jeremy Howard from fast.ai - also no PhD but amazing teacher and contributes actively to research.

Chris Olah (Google Brain, OpenAI, etc) go to university at all.

PhD definitely not required.

You're absolutely right, sometimes, simple and predictable solutions are much better than AI magic ^^

In general, I feel like ML platforms have the problem outlined by https://xkcd.com/927/ (Standards)

The Nuclear Power industry is starting to think about stopping doing all designs on paper, maybe in a few decades they will have achieved this, sending a message that good data is the thing they should work on first isn't a bad idea.

Who cares if it denies bail to minorities or hits a few pedestrians from time to time?

The problem isn’t that ML is too hard, it’s that it’s too easy. Crazy people keep connecting ML to systems that matter- that have real, irreversible impact to humans- and they don’t understand it.

I wish ML were 1000x harder/more expensive to integrate so the economics would drive away frivolity.

Because of that, doing deep learning consists of a bunch of cobbled together heuristics for getting good results and probing the model to give a human an intuition for whether it's learning correctly. The tricks and tips for steering that black box have mostly been developed in the last decade: it is not a super deep well.

These tricks and heuristics are like the knowledge needed to be a technician in a nuclear facility, not the knowledge needed to build the nuclear facility in the first place. It's not nothing, to be sure, but unless you're a researcher developing new novel architectures, a very shallow understanding of the statistics will go a very long way.

Software developers HAVE to have an understanding of the subject they're developing for. Computers are not brains, and they are not able to understand the objective or context in which they run.

I could spit out their crappy tagline - "the hardest thing about developing __X__ should be __Y__, but it often isn’t." - for almost any topic.

"The hardest thing about developing an inertial navigation system should be getting clean sensor readings, but it often isn't"

"The hardest thing about developing MITM proxies should be getting certs configured, but it often isn't"

"The hardest thing about developing web extensions should be setting up your manifest file, but it often isn't"

"The hardest thing about web development should be handling https requests, but it often isn't"

"The hardest thing about having a baby should be labor, but it often isn't"

"The hardest thing about making a car should be getting high quality steel, but it often isn't"

ON AND ON.

It seems to me the lack of knowledge was not knowing to use a diverse sample set, not some lack of mathematics knowledge.

Which is so much bullshit. The hardest thing is validating your hypotheses, which machine learning turns into a black box. When we have coworkers who insist on operating on wishful thinking we try to maneuver them out of a job. Except every 10-15 years when the built up pressure of fads overwhelms reason and we all get stupid for a generation (which in software is about five years).

The things that started as AI that we don’t call AI anymore, and don’t lump in with AI when discussing successes or failures? It’s because they can be explained in plain English and implemented without much or even any special jargon that marks it as anything more than exceptionally clever Logic.

You don't need to know most of the details how your car works in order to drive. You need much more knowledge to build one, yes, but not to drive.

There are different levels of abstractions and depending on your problem you need to understand them only up to a certain level. And different people have different problems to solve.

In most real-world problems today, the difficult part is indeed the data, not the underlying math of the activation function, loss function, or optimizer. Just Google "data-centric AI Andrew Ng" to read more on the topic from one of the most well-known people in ML.

I work at Nyckel. In fact, I'm the "ml guy" at Nyckel. I have a PhD in ML and did some research at Berkeley, but I mostly consider myself a ML engineer. My most recent job was in the self-driving car industry, leading a ML team there.

Knowing the math/stats is helpful when navigating the vast set of models to choose from when fitting your data. Although I'd argue that some sort of black-magic "intuition" earned by doing this for a long time is more important in practice...

However, when validating a model, there is really only one way: test it on production data. This is what Nyckel does: upload your production data, do some annotations, and see if it works. Nyckel handles model search, cross validation, etc for you which reduces the risk of bugs. In a way we are making the argument that by focusing on your data, you are most likely to do well.

But what about that pesky out-of-domain issue? Like the tank/cats or whatever? Well, our customers are not trying to develop AGI, but solve narrow problems using image and text classification. And they are also doing it for themselves so they have all the incentives to be honest. Consider one example use-case from a health food store we work with: "what type of legume (from the 10 I offer in bulk) is in this picture"? As long as they train and test on production data from the warehouse camera stream, they are in good shape from a statistical perspective. Sure, if they throw in a picture from anywhere else, they are toast, but why would they?