Hacker News Reader

Comment by version_five | Hacker News Reader

I'd argue software engineering is still too hard for ML engineers. Most of ML (not research but commercial applications of what's already been demonstrated) is now well within the realm of engineering, but there are few standard practices, bodies of knowledge, or agreed on processes for doing anything. These are problems that engineering formalism solve, not another auto-ml tool. Maybe I'm saying the same thing as the article, just framed differently?

View on HN

In my experience, both are true. I'm more on the ML side, and I can tell I don't have the kind of routine and habits that good software engineers have, though I'm learning. But on the other hand, and I've seen this from software engineers who've made the transition to ML, and clearly have a good handle on the concepts (in one case even published papers in ML journals), they don't seem to have the intuition that allows them to select the right tool for the job, or understand when a given technique is appropriate; you could call that "modeling maturity" - a combination of mathematical maturity and the skill of relating domain knowledge to modeling choices.

I think the above commenter is more critiquing the notion of relying on intuition all together.

Right, I see. That's not really possible imo. For things like mlops, sure. But model development, selection, evaluation? From what I've seen, it's exactly when engineers reach for standard tools without giving thought to how it applies to the given problem that they do a bad job.

Intuition is always important, but it shouldn't be the last word in an engineering problem. I think there is room for a lot more rigour in how we build, optimize, and validate ML model performance, so less of it gets left to intuition. The discipline is becoming mature enough that this is possible, I think there is a lot of room to build out "standards" and a "body of knowledge" that can be applied to building ML models. We're seeing it in pockets, but in so many cases, it is still a dark art.

And then from an actual software engineering perspective, so much ML code is just run-once jupyter notebook stuff... there is a lot we can do. I need to give this more thought, but I think it's acknowledged there is a big opportunity here

Even in non-ML software engineering you still have architectural tradeoffs, that while you can in part make reasoned arguments about, you still are relying on the intuition of your technical leadership.

Absolutely agree, the major challenge has been to explain to software engineers that the pick up an algo from github, monkey see monkey do a tutorial followed by automl or grid search is pretty irrelevant nowadays.

If you can’t implement any existing algorithm from a scientific paper it is pretty hard to understand the flexibility that is required designing a training pipeline.

In terms of tech stacks, cloud providers’ sample architectures and code samples are a base for cloud components and API understanding and not actual implementations.

I'd argue software engineering is still too hard for software engineers.

I think industry and society just has too low a bar for software development. If the quality of civil engineers matched that of software engineers as a whole, we’d all be dead from the collapsed buildings and bridges. Of course, software isn’t usually such high stakes, but I don’t think it would hurt the industry to have some formal licensing. If you write a terrible API or whatever, then a review board decides if your license to practice should be removed. The field would suddenly get better, as developers would no longer care about the PMs artificial deadline, if they need more time to write quality code, they will take it, and the PM simply can’t find anyone else to cheap out on the work either.

Now, how you enforce this, I don’t know, but this is my pie in the sky dream.

I hear what you're saying.

I would argue that review boards, licensing, etc are only relevant because of the stakes. How to enforce it becomes irrelevant if you can't sell anybody on the idea that it's important in the first place.

Even the strength of your statement "...I don't think it would hurt the industry to have..." doesn't sound like something that any part of government would be willing to get behind.

There's just such a huge leap from "decisions civil engineers make can kill. people" to "decisions software engineers make can annoy people".

Still, you're not wrong that the field might immediately get better in some ways, but I believe we could also go down a lot of paths that would lead to the widespread concept of "stifling innovation so that people are less likely to be annoyed by buggy or hard to use software". I'm sure there are also much stronger arguments in favor of your idea of a better world when you narrow the scope down to specific types of software or use cases.

Your suggestion is extremely laughable and naive and reeks of elitism.

The domains where the quality of code has very high stakes do have very strict processes and have an extremely high bar.

The domains where the stakes are not high, for ex - implementing an API to download user reports, have a very low bar.

And your point about cancelling the license to code if you implement a bad API, that's like cancelling a journalist because they made a typo in their article and it went out.

I'm sure there are plenty of quotes out there that are very similar to this, but kudos in stating what most of us (software engineers) already think, with such clarity. I laughed. I cried. I remembered that this is never going to change. ;)

To add to your point, the Hidden Technical Debt in Machine Learning Systems paper: https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896f...

Most folks don't realize that the vast majority of ML applications can be done with preexisting models, you don't need someone with a PhD to fine tune it unless you need to eek out a tiny bit of improvement. The real value is in the person preparing the data for training.

This is not true at all unless you're only referring to commodity applications. There's no preexisting model for general regression or seq to seq, for instance. You can say, oh, that's a good fit for an LSTM, but you still need to tune the individual layers to suit the application. Although I am slightly conflating ML and DL, since you mention training which heavily implies DL.

Most folks can get away with logistic or linear regression. A tiny portion of those might need to move to random forest. And an even tinier portion may justify deep neural networks. The vast majority of ML is not deep learning, nor is it necessary. We're talking very boring typical business cases.

Yeah, this fixation with deep learning is kinda weird, given that most companies don't have vast amounts of images/text lying around and a pressing business need to understand them.

I'm a big fan of starting simple, which means linear/log regression (normally with lasso, as it does variable selection).

Then, if you can prove business value, it may make sense to start trying to use more unstructured data.

I think it is because linear regression is not science fiction enough even if that is the best that can be done on the data. We didn't hire this AI guy to do simple regressions!

Anyone's who hires an AI person unknowingly, probably doesn't know the difference.

I mean, you can call it optimization with a specially chosen sparse function if you like.

From experience, there are lots of companies looking precisely for an "AI person" who is going to use a bunch of complicated sounding buzzwords and build some overly complex thing without evaluating simpler options. This kind of person gets way more attention than someone who suggests that what the company wants to do can probably be accomplished with a simple shell script, or someone that tells them AI is not magic and in the absence of some data capturing a relationship it can't really do anything. It's a hyped up field and has attracted lots of people who care about hype. (I work in the field, not trying to denigrate it, my point is that many / most business adopters want "AI" for all the wrong reasons

Ran into this very thing writing software for customer support teams. There's a Data/ML team within the company and there was a lot of talk about getting them to help plan the software to add abilities to "learn" what customers wanted with their support requests. The reality of the situation is that you can just take a step back and look at the pain points for the CSRs, the highest cost calls and where the time is spent servicing the customer. In the end, it's pretty clear that we can just do a few queries up front to realize that "oh, this person ordered 2 items of the same size in their last order and haven't returned one yet, it's very likely that they want to return one" or any number of other common scenarios that were found and then prioritized based on a simple cost/benefit analysis.

Could the Data/ML team have done something cool and likely complex to show clearer patterns and maybe get something in place to continually refine that data? Probably. Would it have saved more money for the company eventually? Unknown. Would it have cost substantially more money to design and implement? Pretty sure.

To be fair, any competent data team would have done the same analysis first, before attempting to build complicated models.

As I keep saying to people, sometimes the best model is a well chosen graph.

I don't have a lot of experience working with data/ml teams, so this is good to know. Thanks. :)

Agreed. Lots of companies are more concerned with the marketing optics of using "AI".

Ha, I agree that software engineering is too hard for ML engineers, and even for software engineers like myself who have been doing it for 20 years like zcw100 said :).

Author of the blog post here. It was definitely written from my narrow viewpoint and experience. Our goal is to make more solutions accessible to software developers and instinct was the same as yours - a lot of ML can be within the realm of engineering (even small / one-person teams) and that there are accidental complexities standing in the way of wider use. Our solution (AutoML+SaaS) def doesn't work for every situation. I'm curious to hear more of your thoughts on how ML can be made more accessible to Eng (and vice versa).

Reply on news.ycombinator.com