Machine learning, concluded: Did no-code tools beat manual analysis? | Hacker News Reader

Machine learning, concluded: Did no-code tools beat manual analysis?

arstechnica.com

82 pointsCharlesW43 comments

> Speaking to Ars, Amazon Web Services Senior Product Manager Danny Smith explained a non-obvious conundrum: Waiting for a data scientist to analyze one's data "is like waiting for an IT project," he said, noting that biz analyst customers have told him things like, "'We have to wait 18 months to just get to the front of the queue.' And then the data scientists, who are all scarce resources, they're like, 'Well, OK, what's the project's ROI?'"

> Smith continued: "And the business analyst is like, 'If I knew that, I wouldn't need your help!... I've got this data, I've got a business problem. I've got questions for this data. I don't know what it's worth until we actually do the analysis.' So the business analyst is thinking about just accelerating to the point where they can actually argue for or justify their use of a scarce data scientist."

Well as a data scientist I can tell you the ROI of this project is 0 because its a worthless model. Or really negative because it seems like it'd be pretty dangerous to use in practice.

The critical error here is having the business analyst scope this out at all They don't know what they're doing. They don't know how to evaluate the results or predict the likely outcomes.

This is the major problem with the way data science is managed in most organizations. You typically have business analysts or other management types queuing up work that they _think_ could go somewhere, but they have typically not done the necessary legwork to figure out if a market actually exists for the solution that you're going to build. If a business analyst can't tell you:

1 - What problem you're trying to solve.

2 - What a valuable solution to that problem requires (in gory detail), including model metrics.

3 - What a customer is willing to pay for that solution.

4 - What the timeframe looks like for a valuable solution.

5 - How many resources (people, money) the company is willing to devote to this project.

Then the ROI on that project is zero, because there is a very high likelihood that any solution you build will not satisfy the demands of the market.

Can confirm, from experience in Data Management Consulting in large Banking, Insurance, Facilities Management and Food services companies: they build solutions and hope for ROI.

>They don't know how to evaluate the results or predict the likely outcomes.

Hard disagree. Interpreting baseline models is very simple these days.

Granted our models are based on tabular data so YMMV.

Oh yeah? So what are you going to do with this heart attack model? Because I sure hope the answer isn't "model said heart attack so the answer is heart attack"

We don’t for clinical outcomes.

There are way more interesting things to model in healthcare than clinical outcomes.

Clearly, there's a lot of magic going on behind the curtain with Canvas that is not being exposed. The data handling is all automated, so it's hard to tell what was done to improve its use of the data set with missing data and to produce its output. So as a tool for business analysts, it's great

So we're going to be swapping business decisions being made via excel bugs to business decisions being made via black box ML systems.

Dare I say that might be an improvement? I have observed many businesses being run from broken reporting or from someone’s hunch they invented reporting to support that amounts to random decision making. Even a weak correlation from a black box could be better.

No chance any of these models are sufficiently powered to even test the questions the operators are interested in asking

This is a very dangerous article as it tries to make an argument that a person with minimum to no training can start to run these black box systems.

One of the biggest fallacies that I have run into within this space is the failure to understand the problem statement. For example, this article does not mention or address models performances on very unbalanced datasets. Most classic M/S/D-L models will perform mediocrely if care is not given to understanding the statistical distributions underlying the data (or even understand the dynamics of the system such as seasonality).

In addition, the author does not address how biases are introduced by using AWS' (or insert Google, Azure, etc) algorithms...not all algorithms are codified equally.

Finally, this article demonstrates how ppl are trying to trivialize the complexity of very complex algorithms, statistics, etc with a plug-n-play...guess it doesn't help that many companies (I'm looking at you Meta) treat their DS as high caliber business analysts/intelligence units.

Okay, time to step away from the keyboard...

> This is a very dangerous article as it tries to make an argument that a person with minimum to no training can start to run these black box systems.

it is obvious that a class of executives-in-training want to make Ford Factory Workers out of machine learning operators. Do you want to be a Ford Factory Worker? maybe not but the implication is that there are others who will do it

> Do you want to be a Ford Factory Worker? maybe not but the implication is that there are others who will do it

Unfortunately the vast majority of data scientists I've worked with are already more than half way there.

Very few data scientists know how to actually model a problem. They only know how to take data, turn it into a matrix and run it through a grid search algorithm on a bucket of SKlearn models and parameters. In fact I'm sure there are data scientists reading this saying to themselves "wait, isn't that modeling?".

The frustrating part of this is, in my experience, there are a ton of important modeling problems to be solved at most companies, and yet the huge number of data scientists hired out there don't even see modeling problems that don't fit the form y = f(x).

A good quick sanity check of this is to ask a data scientist how they would model churn. An overwhelming majority will say "I'll take N days of data in the past, label who churned there, build a model and then predict who will churn in the next N days", which is absolutely the incorrect way to model churn. If you ask these people about censored data they'll just shrug their shoulders, completely ignorant of the fact that for decades people have been solving this exact same problem in the medical research community using survival analysis.

In reality there is only a small subset of real business problems that can be quickly modeled in SKlearn, and yet this type of "modeling" is the large majority of what data scientists are doing today.

Your argument reads as, “data scientists only know machine learning! Not me, I use popular models from statistics too!”

Not trying to sound offensive, but it’s the same aggressive stance repeated on twitter by every other stats/econ phd. And it’s hardly ever true at any big tech company, where most data scientists (especially those writing the design docs) have PhDs in fields ripe with popular non-ml methods.

I’ve seen far more e.g. fixed-effect models in production than neural networks. Far more synthetic controls and d-in-ds in research papers than ml models. Far more survey sampling methodologies and power studies than multi-armed bandits. All from the “very few” data scientists who apparently haven’t heard of these methods.

There’s a big difference between data scientists working in industry (at least what I’ve seen) and the “learn ml in 10 minutes!” blog post authors on medium.

A ford factory worker stamps the same part for 60 years. Running ML is not the same as making a cog of a priori known quality over and over again, but lets let the MBAs run their little experiment and watch the whole thing explode so no one seriously wastes more effort on this. You cant no code your way around generating testable hypotheses which is why data scientists get paid the big bucks. Anyone can write up a model in python following some 30 minute tutorial. That was never the hard part.

In the article, the report from Data Wrangler actually tells about class imbalance and report a whole other bunch of ML specific issues like target leakage. There’s a blog post on it (granted there are other pre canned tool you could use to do this) https://aws.amazon.com/blogs/machine-learning/accelerate-dat....

Canvas also seem to have nice histograms in their UI

There is a difference between reporting a stat and understanding the stat. Another thing that many ppl (including ppl who work in this field) fail to understand is the theoretical (or statistical) underpinnings of the algorithms they are trying to deploy.

Many assume IID or NID to make the problem tractable, but that's not how the world typically works even on the very very large data scale. More things on why many data science teams/groups/fail because too many ppl treat it as a BI/BA organization...I really should step away from the keyboard now. LOL.

> class imbalance

Worrying about class imbalance is a classic example of how ignorant many data scientists are about how the statistical properties of the models they're using work.

For example if you are to try to "solve" class imbalance issues in a logistic model by over/under sampling you're actually throwing out important information about the prior probability of an event. Class imbalance in the data, if it reflects the real distribution of observed events, is often valuable information that will give you a better model.

Yet so many data scientists I've known see class imbalance as a major issue and when you ask "why?" it's clear they don't know even the basic principles underlying the models they're using.

There are times we you are forced to deal with a class imbalance issue due to the way this can impact training data size, but these cases are relatively rare in modern data environments.

And this is where DS with a deeper more rigorous understanding begin to differentiate themselves by being able to step back and reformulate/re-model the problem into something more like an anomaly detector (in keeping with this example).

I can see the argument where ML Engineers and DS w/out a more advanced statistics/STEM background would fail since they would continue down their list of libraries w/in their prescribed toolbox. Granted many problems can be approximated to be good enough, and let's face it, the FAANG/MAANG/whatever companies aren't running things so super critical where a user getting one extra email or ad presentation will cause them serious injury or death.

Btw, appreciate your comments.

Could you expand a bit about what kind of information class imbalance can reveal?

It's not about revealing information, it's about how you model the problem. Say you're using logistic regression. If you upsample the minority class (never downsample the majority class because that's statistically inadmissible), then your classifier might get better at modeling the decision boundary. However, that comes at the cost of losing the best part of logistic regression: probability prediction and calibration. Sometimes the trade-off is worth it, sometimes not. Depends on the type of problem.

For more details I suggest reading Frank Harrell: https://www.fharrell.com/post/classification/

The class imbalance quite literally represents the prior probability in the model.

As an example, train a logistic model on imbalanced data with both the real distribution of imbalanced classes and over/down sampled classes and you'll notice that the distribution of the predictions from each model is different.

This is hugely important because if you are using the output of your model as an expectation to feed into another model (for example predict expected value of a new user account given P(purchase) * E[value of purchase]), rebalancing will give you the wrong answer. This can be corrected in a logistic model by tweaking the intercept but doing this requires you to make some assumptions, and it's more artful of a process than one would hope.

You can also see this if you compare the log likelihood of the true data with the given model and the new data. The model trained on the real data will have a notably higher log likelihood.

Then there are separate issues with over/down sampling.

If you are over sampling the standard error on your coefficients will be artificially reduced by the over representation of examples from the one class, leading you to be more sure than you should be about your parameters. The point estimates should be more or less the same, but if you care at all about inference this change in standard error is a big deal. I believe, but would have to double check, that this would likewise impact the effects of regularization on your model.

If you are down sampling you are quite literally throwing out observations, and using less information than you have when modeling. Additionally you'll get the inverse problem which is artificially increased variance in the parameter estimates. Again, even if you don't care about inference you still have the problem that your regularization is very likely not working how you expect.

In my experience nearly all data scientists think they are experts in logistic regression when very few really understand this "simple" model. All of these issues here will manifest themselves in more complex models but understanding how (and how to correct them) is a much trickier question.

This article just reinforces that management thinks that data scientists spend most of their time building models. In reality, data scientists spend the majority of their time munging and wrangling disparate data types (that are typically a total mess), understanding the data well enough to fix problems, translating business requirements to code, and converting model outputs to human understandable presentations. Building models is a trivial amount of the effort for most projects, typically 10% or less of the time. Most organizations will eventually move over to AutoML type solutions, but then will be shocked when they fail to achieve any significant gains in productivity because they're optimizing one of the shortest steps in the whole process.

LOL...typically i would agree but i find DL requires a lot more intense engagement (for now) followed by lots of idle time reading hacker news as the models complete training.

DS/ML/BI/WTF EVER has all become overly incestious filled with buzzwords generated by hiring managers/executives who really have no idea what they're talking about when they write the job descriptions (or they're being seriously cheap by blending job functions together such as Head of Data and Analytics...wtf is all that about. From a business function sure sounds great, technically it's a recipe to set ppl up for failure).

The comments here all sound like "this thing can never replace me". But if it's an inscrutable black box that performs better on the bottom line metrics (which is something AI can be trained for) then how is it different than all the other AIs that play chess better etc. but can't explain how they do it, because it's a strategy encoded in a million-dimensional vector space?

Most real world business problems don't involve board positions and rules and winning moves as simple as chess or Go, and do involve a lot of people that need to be persuaded. So the ability to answer "why's that the winning move?" or "what would change that conclusion?" or "how confident are you are data is reliable enough?" is a lot more useful than a slightly better [over]fit.

Even with this example where it's analysing probabilities of heart attacks which can be [retrospectively] validated with precision, I absolutely do want the doctor who says "you're high risk, these are the main reasons why and how you mitigate them" than just a model with a lower false negative rate that can't tell me why.

In line with the other comments, let me give you a recent example from my work.

We had a dataset with just about 30 records and about twice as many features, about 1:3 class balance between positive and negative. Not an impossible situation at all, but there was a big premium on performance, so we threw the book at it.

Lo and behold, SVM with some particular parameters comes out on top. Ship it, right?

Not so fast, what decision did the model learn? Turns out, the model was learning decision boundaries that did not make sense. It was overfitting because the model did not know about what kinds of decision boundaries were allowable.

In the end, someone looked at a 2D scatter plot, drew a line and called it a day. It took a fraction of the effort and time of running through a box of models. It'll save the company $MM and it's trivially explainable to engineers and execs.

AutoML with blind application would've completely botched this.

Or you have the classic case where the model trains on the wrong things (e.g. the classic husky/wolf and snow case). One example I heard was in heart attack risk prediction from x-rays, the model was focusing on the present of stents in the x-rays. Which is a trivial prediction because the doctor already knows the patient has a stent, so the model is not adding anything of value.

No, I think the arguments you are reading is that this article was poorly written (granted its Ars) and a more appropriate article would have considered broader perspectives.

This feels like something an undergrad would write for their business management class...btw, your chess analogy is flawed in that there are millions of patterns that a computer can simulate for next play. I wouldn't really call it AI per se.

The problem is that the real job that the data scientist does is what the article suggests the business analyst is doing. The business analyst doesn't have a clue what they're doing, and training the ML model is not the difficult part.

"The compute time I spent for the whole project came to about $1,000"

Any comparative costs not using aws? Seems expensive for what he did.

From what I could see, the dataset has 300 data points (and less than 100 attributes/features), so it should take less than a minute to train a state of the art model on a normal computer. Cost: 0$

Right, it's a small dataset no GPU, guess your paying for aws software but I think there are plenty of open source equivalents that could match/beat these results with similar level of effort.

Any affordable(for a small business) no-code/low-code ML for small business?

Training an ML model is very easy. You can learn to pass things through an sklearn decision tree model over a couple hours.

Interpreting it is a more battle hardened task. Don't blindly trust such tools.

KNIME is a free, fairly well used tool in some sectors, for doing low and no-code analysis and machine learning models.

https://en.wikipedia.org/wiki/KNIME

This one perhaps [1]? I only tried it for ten minutes but it seemed nice.

[1] https://github.com/biolab/orange3

Ah Orange, that takes me back to university days. What a pain that thing was to work with.

No joke, it was the clunkiest library ever. I wonder if they ended up improving it.

I think the only reason why he used Sagemaker is that it provided him an easy out of the box set of tools that required minimum-to-no setup.

If the author took the time, he could have stitched together a completely zero-cost alternative that could have run locally on his box, but that would have gone against the argument that this is so easy that "even a monkey can do this". Don't get me wrong...AWS has some very nice libraries, especially their DL and encoding/decoding options.

I just think it was a bit heavy using Sagemaker especially for such a trivial use-case...maybe even the free Kaggle notebooks would have been a better alternative?

I think that this Q explains why Gartner and other garbage analytic companies are so full bs. Hold them accountable to what they said 5 years ago!

Also laughable about the 90% threshold that seems to STILL be the magic number for success within ML. Try applying that in your private space.

How much does this cost on aws?

Recent threads on datarobot make me wonder how viable their product is when stuff like this is available from the major cloud providers.

I know Alteryx is quite expensive, over time probably nobody can beat AWS on pricing

I look forward to seeing what will be able to beat AWS in the long run. I don't know what it could be. I guess, maybe, if they stop innovating in the space others will catch up and then price will become the differentiator?

Idk, right now AWS is a money printing machine.

Reply on news.ycombinator.com