Why is AI so useless for business?
mebassett.info
mebassett.info
Think how much Microsoft Office and competitors have amplified business productivity over the last 20 years (yeah yeah, make your jokes too). Word and Powerpoint and Excel are full of AI whether it's spellcheck or auto-fill, drawing algorithms like "fill with color", etc. So many things that were AI research papers of the 70s, 80s, or 90s. And those innovations continue today.
Logistics companies rely on huge amounts of optimization and problem-solving. Finding routes for drivers and deliveries, planning schedules, optimizing store layouts, etc. -- that's AI.
Employees use AI tools to improve their lives and productivity whether it's a rideshare or maps app to get to work, speech-to-text, asking Siri for answers, translating web pages, etc. All of this comes out of research in AI or related fields.
How many office jobs don't require someone to use a search engine to find and access information related to a query? Information retrieval is one million percent AI.
Robotics and automated manufacturing has been huge for a long time -- robotics is closely connected AI and related problems like control theory.
The best applications of AI have almost always been to support and enhance human decisionmaking, not replace it.
I've never thought of it that way before but alternative medicine and AI have a lot in common.
I mean if I take some herb and (successfully) use it to treat abdominal pain without knowing what that herb is or how it works, would that herb be classified as medicine or not?
I have read description of many medicines and found out that we still don't know how those medicines work. But they are still called medicines.
So what is medicine?
Of course, alternative medicine proponents rarely do randomized controlled trials.
If it can't, then frankly, you don't really know it works.
As for the opposite case - if there was a thing you wanted to test that had a strong effect, then the company that owns the current treatment would happily fund you the tests in exchange for dibs if the tests pans out.
Normally, we would not only want a proof that something works but also verifiable explanation of HOW something works.
Medicine is a problem because we don't understand how many of the chemicals work but the problem is so important (and also it is big business) that we just let it pass and we are happy that we can get some results even if we don't completely understand how we get them.
You mean 9 minute beat poem.
Edit: Since people are bringing up ancient examples and kind of missing the point of the question: I'm not looking for examples from the Roman Empire here. Let's stick with the past < 50 years. Maybe something your parents might actually remember being dismissed as "alternative medicine" (whatever that meant at the time) but which now is clearly just accepted "medicine". Basically, try to find something that's in the spirit of the question. The goal is obviously to find things that modern medicine actually previously dismissed and later accepted, not to find a loophole in the question.
Not all hyperbole are straw men, the original question was about how the current distinction between medicine and non-medicine works. Historical examples are relevant, but not what intended in the question.
Does rely on accepting "cleaning things" as medicine. It's certainly a great way to prevent disease. Does it count as "alternative preventative medicine"?
AI doesn't have such an authority. New AI methods can be deployed without asking for permission from a governing body.
19th century views on medicine is actually a better comparison than something from the last fifty years, because advancements in modern medicine is largely shaped by governing authorities.
I also vaguely recall that the doctrine of signatures wasn't a complete and abject failure, but some cursory Googling indicates that maybe it was after all.
Effective treatments as been subsumed into "mainstream" medicine, perhaps with a bit of purification/standardization along the way, while the ineffective and unproven stuff remains "alternative medicine".
If it somehow turns out that homeopathy is an effective therapy for something (besides dehydration), it'll end up with a procedure code, the hospital pharmacy will stock 10x tincture of whatever, and eventually it will be just another mainstream medical treatment.
https://en.wikipedia.org/wiki/Artemisinin
"It was discovered in 1972 by Tu Youyou, who was co-recipient of the 2015 Nobel Prize in Medicine for her discovery.[2] "
There are "scientific/medicine" results these days what it even works against cancer, for example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6347441/
There's things like "gargle with salt water to cure a sore throat". That's commonly done remedy that was done forever and we now understand the science behind it. This page[0] labels it under an "alternative medicine" tag, as does this page[1]:
[0]: https://www.webmd.com/cold-and-flu/features/does-gargling-wl... [1]: https://www.sharecare.com/health/alternative-medicines/artic...
The conclusion then is that there are things we consider medicine today that used to be alternative medicine.
That would imply examples from 400 BC would have the same relevance as examples from 50 years ago. I assumed it was obvious that that's not the case, but it seems I was wrong.
As long as pain and death exist our medical knowledge is incomplete. That doesn't mean it isn't the best we have or that the process isn't the best we are capable of, it just means the job isn't done (or, in some cases, mistakes or oversights occurred).
The saying is also intentionally provocative, and in my interpretation is meant to promote curiosity and open-mindedness rather than an interest in any specific alternative therapy.
An example of an alternative therapy gone mainstream (among many others in the thread that I think fit) is folic acid supplementation for those who many become pregnant. It was not universally practiced even after some physicians had linked it to spina bifida prevention (this is my layman's understanding of history, not medical advice or judgment): https://pediatrics.aappublications.org/content/106/4/825
I would say many alternative therapies revolve around some supplement or another. Only rarely does the medical community agree that it is of high importance (if they do, again, I'm not a doctor I might be very wrong) and the previously alternative therapy become mainstream.
St. John's Wort started out as alternative medicine but is now officially approved and prescribed for treating depression in many European countries -- I believe this got started in the 1990s.
I think this kind of thing is quite rare in medicine, though. Whereas it's not hard to find stuff that was done by AI researchers that somehow no longer counts as AI.
> Complementary medicine refers to therapies that complement traditional western (or allopathic) medicine and is used together with conventional medicine, and alternative medicine is used in place of conventional medicine. Alternative medicine refers to therapeutic approaches taken in place of traditional medicine and used to treat or ameliorate disease.
The difference (as I've understood it anyway) is "alternative medicine" implies you contradict modern medical practices, which was the basis for the comment I replied to. And which (I guess not surprisingly) medical experts recommend against. So I think the kind of example you'd want is a medical treatment that modern medicine previously recommended against, but that ended up vouching later. Because this was all a response to a comment that suggested people had a double-standard and were moving the goalposts or something like that (hence the AI comparison). Merely showing the science went from "we don't know if X is a good idea" to "we know X is a good idea" isn't finding an inconsistency.
Exceptions may be in mental health. Somebody else mentioned psychedelic drugs, which might be FDA-approved for some conditions soon, but all official medical sources currently treat them as extremely dangerous.
However, I do think cherry-picking the effective treatments from naturopathy and TCM and turning them into mainstream medicine is similar in spirit to cherry-picking the successful approaches from 80s and 90s AI research and turning them into "just algorithms". But there are a lot more successful old-school AI techniques than there are effective alternative medicine treatments.
https://www.discovermagazine.com/health/the-doctor-who-drank...
Trump was mocked in the media for asking if getting UV light inside the body was a possible treatment for covid19, as if he was suggesting an alternative medicine treatment. And light therapy does have that long a history.
Thing is, this is not only a known treatment used in the past (only falling out of favor in the 50s), there are studies from the past few decades about how well UV light works on viruses, and it really is being investigated today for use on SARS-CoV-2.
I don't like this quote. It is mostly true, yet it is not completely true. But because it is mostly true so many treat it as if it is completely true and thus miss the edge cases where it is not true.
Take for example certain substances that are deemed by the US federal government to have no medical value. They cannot be medicines. Yet they are alternative medicines and even prescribed by doctors. Their are even states who go against the federal government by saying these substances do have medical value. And if you go back a few decades, you'll find a time where mainstream science largely ignored any medical benefits of these substances, likely due to pressure from the federal government, and yet at that time those substances were alternative medicines that worked.
That's not same thing as saying that there isn't anything which is currently considered "alternative medicine" that works. Rather, once an alternative medicine has sufficient evidence going for it (and is better understood), it becomes less mystical and is accepted as mainstream medicine.
Does the DEA and the FDA not play a role in what is considered a medicine or an alternative medicine?
>The fact that the medical establishment now accepts marijuana as a therapeutic tool is evidence of the quote: it was "alternative medicine" until there was sufficient evidence that it worked.
This is effectively making the claim that while there were problems in the past, the fixing of past problems can be taken as evidence that current problems do not exist. I'm arguing the opposite. That problems in the past, without fixing the systematic causes of those problems, is a reliable indicator that we are still at risk of those same problems.
Think of it like code. If your code had bugs but you fixed them, should you assume that A: my code had bugs but now they are gone and my code is bug free or B: the process by which I write code allows for bugs, and while I have fixed those bugs I haven't fundamentally changed how I write code so I am still at risk for having bugs. I find the latter the more reasonable approach.
>That's not same thing as saying that there isn't anything which is currently considered "alternative medicine" that works.
When people use that quote, I find that this is exactly what they are saying in the majority of cases. That if something is currently considered alternative medicine, it doesn't work because otherwise it would be considered medicine.
A claim that as our knowledge progresses alternative medicine that works becomes medicine is a much more reasonable claim, because it recognizes there is a timeline and since we aren't at the end of that timeline we allow for some alternative medicine actually work with the recognition in the future it will be refined into medicine.
And if someone wanted to make the claim that the majority of alternative medicine doesn't work, I would also fully agree. My issue is people taking a guideline that is generally true and applying it as if it is always true.
Part of the challenge is/was the line of thinking "surely if we can solve Hard Problem X, we'll have intelligence!" This turned out to be entirely wrongheaded, since a vast litany of Hard Problems X turned out to have plain old algorithmic solutions.
[1] I keep hoping this shows up online somewhere. It was shot using a machine room as the set, where mid-century modern furniture had been brought in for guest and host!
Until then we have Machine Learning or gradient descent or whatever else.
“Technology is a word that describes something that doesn’t work yet.”
Douglas Adams
Nick Bostrom wrote the following few lines in his book "Superintelligence: Paths, Dangers, Strategies" (which I absolutely recommend reading):
> There is an important sense, however, in which chess-playing AI turned out to be a lesser triumph than many imagined it would be. It was once supposed, perhaps not unreasonably, that in order for a computer to play chess at grandmaster level, it would have to be endowed with a high degree of general intelligence.
A better benchmark would be that a single piece of software that is not specialized is able to solve two of these problems.
As you want to support more use cases, you just need to plug in a new subprogram and modify the top level problem recognizer.
A chat bot sitting on top of a bunch of specialized systems, basically.
This is clearly still not general intelligence and probably not super useful, but hey.
The true benchmark for intelligence is a piece of software that is presented with some pattern and told to turn it into some other pattern and then figures out how to do it. This probably requires general and deep language comprehension(ie. get me a list of emails for environmental managers at US public water utilities. system needs to have some idea about google, some idea about the EPA and their web resources as a starting point for getting that dataset, some ability to google, read and understand web page data etc).
Obviously you can cheat by just having a giant database of information, but at some point you have to be able to answer questions that are not in the database.
AI used to be sold as human-like or even superhuman intelligence. Now, it just means "running software".
Which is way more cool and gives you more dreams (or money).
Whereas on the other end of the scale people are aimlessly using subtle AI-related techniques like neural networks and calling the result "intelligent" even if it is anything but.
Where's AI in Microsoft Office? Where's AI in spellchecking? Where's AI in route planning? Demand planning, maybe.
Should we consider the "if it works, it's no longer AI" point made now?
Route planning is essentially "the" fundamental AI problem. You need to get from A to B and there are 2^100 ways to do it. How do you pick the best one?
In contrast, your example of the human is like modern AI, specifically unsupervised learning. I'm sure I could get a human doing that kind of pattern matching quite good at it quite quickly.
If I didn't have a learning system I could still write an algorithm for it quite quickly, by computing the Levenshtein distance. But that algorithm would notoriously not understand the context of the sentence, probabilities of words, etc. And indeed Word/Android/iOS spell correct still gets it really wrong much of the time. But the AI used is quite different from old timey backtracking.
Works really well even if it is a bit repetitive.
The thing that I don't appreciate in this field is that companies seem more focused on discussing their "AI" than the actual problems they hope to solve. When I see this, my mind jumps to the assumption that these companies are either: 1) looking for easy valuation multiples. 2) looking for a way to impress VCs/investors. 3) they don't actually understand the problem they are solving. This is admittedly a big stereotype on my part, but this stereotype was built on pattern recognition over time.
How good was your input data and labels? ;)
If a path-finding algorithm that I can write on paper is AI, AI has completely lost all meaning. Let's not call graph traversal and sorting "AI" please.
A huge amount of stuff can and should be automated by mundane programming. Since that hasn't been done yet, AI isn't about to automate all of the stuff that shouldn't be automated by mundane programming.
Your comment is just a crystallization of what the parent comment is talking about: claiming something isn't AI because "pft that's such an easy, solved problem" _after_ it's solved defines away the possibility that AI can solve problems.
If anything the opposite has happened. In a painful attempt to push forward notions of success in AI almost purely mechanical tasks have been claimed to be AI, while there is virtually no progress on building machines that can think.
I mean sure you can claim all day that the navigation system in your car calculates you a billion routes per second and if that's intelligence my smart-toaster is probably more intelligent than everyone here together, but it completely misses the point, and the reason why people have expanded the term so much is because there has been so little progress on genuine intelligence.
Yes, there's two word senses, the theoretical and the colloquial. You're referring to the former, and I'm referring to the latter; the latter is a lot more relevant to this thread's topic, which is public perception of AI and its value. Wikipedia actually has a pretty good concise description of these two senses:
> Leading AI textbooks define the field as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals.[1] Colloquially, the term "artificial intelligence" is often used to describe machines (or computers) that mimic "cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving"[2]
(The [2] citation is of Russell & Norvig's '09 edition, substantially predating the recent mass-interest in AI)
> if that's intelligence my smart-toaster is probably more intelligent than everyone here together
The leap from "this is reasonably described as a step on the road to intelligence" to "being really good at this step means you're _really_ intelligent" is obviously nonsense, though I suspect you know that. The fact that a cat has the ability to orient itself and navigate home and am amoeba couldn't is a sign of relative intelligence; but if your cat has a better sense of direction than you, it obviously doesn't make it smarter than you.
Ok, then nothing being done today (neural nets, gan, etc) on computers is AI. What a dumb definition.
>claiming something isn't AI because "pft that's such an easy, solved problem"
Nope. It's not AI because instructions were directly written by a programmer of how to path-find. There was no input training where the program learned how to pathfind. It just had one method from day 1 that hasn't deviated since.
(1) Optimization is much, much bigger than graph traversal and sorting.
(2) Modern route-finding algorithms are to your on-paper-Dijkstra what a rocket ship is to your bicycle.
(3) I think you're under the same misconception I'm talking about: graph traversal is absolutely a fundamental of AI. Ask anyone what the main AI textbook is, they'll tell you it's Russell and Norvig: http://aima.cs.berkeley.edu/
The first topic they cover is graph traversal and search.
Let's not get ahead of ourselves. Modern path-finding algorithms are Dijkstra + lots of heuristics piled on top.
> graph traversal is absolutely a fundamental of AI.
I agree. But being a fundamental of AI does not make it AI itself.
Some good examples of modern approaches, though a bit dated now: http://algo2.iti.kit.edu/routeplanning.php
Relevant conf: https://icaps20.icaps-conference.org/
(ML techniques are increasingly being used to solve these problems, and graph algorithms are used in ML, but are not AI/ML.)
Sometimes the heuristics ARE the point.
A is fundamental of B does not mean B is A.
In particular, the best superhuman poker-playing systems use counterfactual regret minimization, which is literally just traversing a tree and updating small arrays of numbers at each node, augmented with some clever heuristics to make it scale to realistic poker scenarios. there aren't even any neural networks involved.
X is a component of Y does not imply X is Y.
I use lots of nails to build a house. A nail is not a house.
Physics once held that there was a substance called aether rather than vacuum, and chemistry that fire came from phlogiston, but we didn't need to rename the discipline when reasoning evolved.
The key difference being that physical theories based on aether never actually worked, whereas computer programs based on graph algorithms solved many problems they set out to solve.
Graph algorithms are not a subset of AI.
The algorithms of AI of decades past still work.
That's a big difference.
E.g. SHRDLU can be built today and you can have a conversation with it about its world of blocks.
Algorithms are artifacts of mathematics. To calculate distance between two points, we still use sqrt(dx^2 + dy^2); it doesn't go out of fashion due to advancements in topology.
I don't get this hate for new techniques being classified as AI. Just because they learn distributions instead of using classic prolog?
I knew what a neural network was, and understood it as part of AI, when Wham! was in the Top 40 charts.
The hate isn't for the "new" techniques, I think; just for the posers who claim to be AI experts because they know how to use some Python library or whatnot.
You don't have AI creds if you have no background in the symbolic stuff.
Having worked at the largest tech based Logistics company in India, I can say, we did rely on optimization and problem-solving, but none of them involved AI, they were mathematical models and not black box.
Of course this isn't new, but a broad focus on specializing on developing talent to create these models and methodology and tech to help would be.
I was going to say the same thing, but I worked at BMW and VW and spearheaded several initiatives with Corporate partners that relied more on optimizing via mathematical models/data sets within the warrantied parts/Takata airbag recall at BMW and the Tdi Diesel-gate buy program at VW. It entailed lots of data analysis and some trial by error on my part that eventually got us a favourable result, not AI.
AI can be useful, one day, but I'm returning back to Supply Chain analytics and Logistics and not much seems to have changed in those years. I submitted my proposal for a Supply Chain Analytics course as my final project that I drew up in 2017 at BMW and got 99.7% for my thorough, and more importunately to me, relevant analysis and execution of a scheme to optimize leadtime and overall turn over using the means and methods available back then.
A part of me wishes I could just run an algo/AI protocol with predictive modeling to do that all for me, as it almost got me fired several times trying to deploy it and I had to go over managements head and straight to the owners and corporate to get them to try it.
Luckily, I was able to negotiate a return to VW as result of my scheme, who were hemorrhaging Billions, were getting execs thrown in prison at the time so were way more receptive to ideas of cost cutting. Crazy days...
* AI initially inserted 'trail,' instead of a commonly used phrase 'trial by error' perhaps proving how autocorrect/spellcheck AI usecases are still not where they need to be to prove the point of the aforementioned post.
It's actually "trial and error", although that doesn't speak any better for autocorrupt.
Indeed. You got me there. But the point stands, as you mentioned.
It's only since 2013 or so that AI = ML = DL.
> if not for the absence of calculation-by-machine, Isaac Newton and Archimedes would have been doing AI.
From the Stanford Encyclopedia of Philosophy entry on Leibniz's Philosophy of the Mind [1]:
"He believed that such a language would perfectly mirror the processes of intelligible human reasoning. It is this plan that has led some to believe that Leibniz came close to anticipating artificial intelligence. At any rate, Leibniz's writings about this project (which, it should be noted, he never got the chance to actualize) reveal significant insights into his understanding of the nature of human reasoning. This understanding, it turns out, is not that different from contemporary conceptions of the mind, as many of his discussions bear considerable relevance to discussions in the cognitive sciences."
Wow... talk about changing the definition of something to fit your world view.
AI "gets a bad rap" because it is past its hype peak. The number of people realizing that slapping an AI sticker on your product is largely BS and doesn't impress any more is growing, and AI isn't magic to everybody that can solve every problem. It's the same overhype cycle which happens with just about everything.
You can add AI of course, but the basics are just math and statistics, e.g. take a look at this book: https://nlp.stanford.edu/IR-book/information-retrieval-book....
What? You mean this: https://en.wikipedia.org/wiki/Flood_fill
Who ever called that artificial intelligence?
I don't know about MS Office, specifically, though.
I thought the whole point was to get out of the business of having to write down procedures for everything, to have the computer approach novel problems like a programmer does.
Is this only a contemporary perspective? Seems like that's where the goalposts were at least 15 years ago.
That would have been uncomfortable, so the AI prophets have instead responded by changing the meaning of the term.
Literally the only things you mention using AI/ML techniques are "speech-to-text, asking Siri for answers, translating web pages" and search engines, which are external to business process. Internal search is usually a disaster because processes are document based rather than web/publication based.
There are ML applications used for business, like speech recog, pricing, marketing and so on, but not so much for business processes. Where are the AI tools for parsing legal contracts? Agents for finding and negotiating prices with suppliers? Filling out regulatory reports?
Your definition of AI seems to be particularly broad. There are several general and specific algorithms designed to handle the examples you gave and they have been working well for the last few decades, some even earlier and have been implemented on computers later. E.g. linear programming was designed to solve several classes of programs and was initially done on paper. Lumping all this as "AI" doesn't seem fair to me.
Is it? These have largely been done with MILP and related tools for decades, and those approaches have never been called AI. What AI techniques are you thinking of here?
Anyway, AI gets a bad rap because there's so much hype and snake oil out there that the signal is being lost in the noise. I'll be much better disposed towards it once the next AI winter hits. Like for instance, I have great respect for anybody selling an expert system at this point in time.
But most ML/Data Science people have no proper understanding of AI/ML, and when just traditional "coding" can solve the problem rather than throwing fancy statistical models and buzzword.
I'm not in business nor AI/ML, but in Robotics. And as a person in Robotics it's always the same experience working with AI/ML engineers: They first say they require large amount of data, then give great promises. (but never specific metrics, except maybe a percentage of success) Then they deliver a module that fails outside the perfect scope of deployment(works only in the lab at 1pm). This is ofc never specified in the delivery. Also crucially, it does not give a good indication of failure. The amount of ad-hoc you need to add after the thing is delivered is just staggering.
On top of this reality, most ML/Data science peoples response to this entire process is to point and blame the data, or the "well you guys is expecting too much from this!" when they had ample time to outline the scope, limitation and requirement before they even started collecting the data.
When you do "traditional coding" you are getting results that are easier to work with because you understand where they came form. Then it is easier to modify the model and understand the outcome.
Your background is robotics. As a robotics example, training AI model to keep bipedal robot upright might be possible but having proper control model that came from control theory, identifying the system, etc. will mean you can actually calculate the envelope of the model, the limitations, etc. and that the model is not likely to surprise you in catastrophic ways. It is a case where AI would be a poor replacement for traditional modelling.
To me, linear/logistic regression is AI, as are neural nets, as are pretty much any statistical model. These days "AI" seems to be roughly interpreted as "whatever a DS person deems as a solution to our business problem" at least in my very limited experience. It's all a bit silly since AI has come to mean so many different things that may not fit the traditional definition of AI from e.g. the 90's, but as long as people aren't pidgeon-holeing themselves into particular methods, I'm happy to go along with slapping an "AI" label on pretty much anything.
For as long as I can remember, AI has been a synonym for "hard CS problem" or even just "hard math problem." It has more use as a marketing term than as a technical term with precise meaning.
Yes, that's exactly what I meant. I kind of borrowed the term from aviation where this is used to describe conditions under which the behavior of the aircraft is known and predictable. Doesn't mean it is not possible to fly outside the envelope, you just don't get any guarantees.
Ah gotcha, totally agree.
> It has more use as a marketing term than as a technical term with precise meaning.
yea exactly; not a problem for me that this is the case (buzzwords are just a fact of life), but really what should be happening underneath any "AI" solution is finding and implementing the right solution to the problem. If the right solution is classical control theory or linear models, but the business leaders insist on deep learning because to them that's what AI means, that's where there's a problem.
* Read “not at all”
Academically we mostly specify AGI to mean things more advanced or speculative.
[0] https://www.controleng.com/articles/exploring-the-basic-conc...
Sorry but no, this rhetoric is often used as an easy escape of responsibility but I don't buy it. Sure, if you're talking about the neural nets used in Google translate or YouTube recommendation system, or any big tech company domain of problem that is based on vast amount of data with extreme size of state space and variability - fine.
But we are not talking about interpretability of the result but the reliability of it, and in robotics domain you rarely have to tackle that size of state space, but rather a highly noisy one.
Let's take your example of low level control of bipedal robot. At whatever stage of control/manipulation you're talking about, you always have to consider that the input will have some noise, and how to handle it. This is the reliability; you know how to(or how much) of the input noise you can handle, and how much you will leave (or amplify) to the rest of the system. Based on this you also know the limit of the system, so you can also get an indication of failure. You can make AI/Neural Network to consider the noise and also give an indication of failure, but you have to think about that from the data gathering step! Most control algorithms have this: "what input noise can I handle" question answered; The same requirement should be set to the data.
I assume you thought about Reinforcement Learning of sorts when you thought about control, but what most people don't realize is that those algorithms works only because you can use high precision sensor that reduce these noise level (combined with sensor fusion) to negligible level. But now you're relying on high precision, extremely expensive sensors. They almost always fail when you get out of the lab and have to suddenly consider cost. (and at this stage traditional control algorithm creeps back)
Let's take a real example. On my original comment I wrote "work in lab at 1pm" and I weren't sarcastic. Apparently not only did the AI solution (in Computer Vision) I were given only work in the lab, but at the exact lightning condition under high noon at the lab, with all the ceiling lights on all the time. We are now not talking about interpretability of the AI, but the reliability; This is easy to identify beforehand. You can also mitigate this (the lightning noise) by including it in the model (data). This should have been glaringly obvious if the person actually knew his toolset rather than just retraining a existing model using tensorflow magic.
They are probably not what most people would call ML, but they are simple and solve a number of common problems.
I wrote a tool that would go through and find "representative examples" of sessions for each tree path, then I could sit down with the UX designers and show them exactly what the user was doing. This was helpful as that group was the most vocal critics of our work, mainly because the data we provided began to drive UX design from senior management, and they weren't happy about it. But when I could sit down and show them exactly how the user was spinning their wheels, they were more likely to buy into changes (and offer effective solutions).
Plus, I could inform the UX team which design detail was the primary driver for some event we were testing for. An example would be if the user was applying more than three filters, they were less likely to complete their goal. Thus, the UX team was able to devise a couple of reworks to the filtering mechanisms, which improved customer feedback.
This tactic would probably be much more difficult if we used NN to solve the same problem.
If you're in sklearn, the method you want to use is the "inverse transform"
Maybe that's the problem. In lots of AI/ML problems you just CAN'T know ahead of time what can be deliver you need to spend the time and resources to do it and then see how well it works...
The problem imo is on the business side, most businesses don't know how to transform unpredictable progress into profit (even if average on a large timeframe that progress might be HUGE).
So ML/DS people need to overpromise in order to get anything approved, otherwise they'd just have to sit around and do nothing, and overall everyone would be worse too, bc that real but unreliable progress would never happen.
- in a highly competitive market your actions can be unpredictable by competitors: evolution shows this can work even in the most primitive ways - it's so damn hard to swat a fly because it changes direction randomly, being smarter than it doesn't help... you could get supercharged versions of this advantage with random-AI-progress as long as you don't constrain business direction, eg. if your AI research suddenly produces breakthroughs that shows you can have an advantage on smart adult toys by applying you aerospace technology capabilities, then be willing to get in that space and play to win regardless of how much that makes sense (tip: you'll probably want to stay privately owned, markets will likely hate this "random pivoting" as it would look from outside) -- as business you'd be "the fly that no one can swat"
- use what's knows to be good at turning positive unpredictability into overall profits (markets), but internally - have multiple projects & people competing against each other inside a company, cheating antifragility / resilience, and really allow projects to fail (as in make it clear that you could be promoted even if the project/team you lead fails, so encourage the king of risk that would be too high even for a startup) etc. - you'll average out the unpredictibility and be left with the progress + antifragility -- as a business you'll be "the cockroach that survives after radiation, eg. unpredictability, has killed everything else"
Hint: all these are already happening, since before AI... and what you'll see now will be that AI/ML will bring most benefits to large corporations with multiple lines of business and multiple independent departments/subsidiaries competing agains each other, and startups-inside-corporations that until now were "pale fakes"... and probably investors already groked this too, there's a lot of growth in AI/ML applied deep inside the guts of big Cos... when this plays out to real wins, result will probably be pretty dystopic: SMBs will be wiped out, lots independent startups too probably (as a small/lone player you can't turn unpredictable progress into profits), you'll get even more centralization, and more mass-surveilance bc now it will not offer only national security advantages but also massive immediate business advantages with AI feeding on the data...
EDIT+: That's I think the problem we need to solve is how to de-centralize and spread the benefits of applying AI to business... bc it will naturally promote centralization and closed gardens, unpredictable progress is mostly toxic to each individual player alone... we need some business-version-of-democracy++, pure capitalism will result in a marriage of mega-corps + pseudo-totalitarian-govs since these will be the entities naturally thriving in the new landscape.
There's an interesting whole vs. parts observation in there. Societally, we know exactly how to transform unpredictable progress into profits: markets. And they work very well, over the long term.
It's just that markets work by trying lots of new random things, most of which will be failures, and then creating an incentive to double down on the winners. Every individual market participant has an incentive not to innovate, because most innovations fail. But some of them do succeed, and the ones that succeed end up replacing the companies that never tried to innovate in the first place.
IME with 8+ years as a Data Scientist in internal and external consulting roles. What I like to do is propose a phase 1, phase 2, phase 3, etc. Phase 1 is almost always a mix of scoping and "Does it even make sense to do this."
Only a fraction of the teams/customers I work like appreciate and have a tolerance for this longer term strategy. Most of those projects are a success and people get value out of using a model in addressing their business problem.
Most people hear this and respond with "No, Phase 1 needs to solve our poorly defined business problem and show massive value in 3 weeks so we can boast about how innovative we are being. You need to AI this problem. AI it now. Do some of that AI stuff. Give me the AI." Most of these fail and I know they are doing to fail, even if any models turn out be useful.
I will say that there there Data Scientist/AI/ML people who really just throw models at stuff and don't think about the value of what they are trying to deliver, but a lot of them are typically inexperienced and just have new toy syndrome. It's not really any different than somebody who reads a bunch of books (e.g. leadership, business) and then gets overly excited about using what they learned. They'll grow out of it or leave the profession when reality hits.
* 1 month (or less) to prove that it makes sense to continue. We do start out with a pretty well defined problem definition though. Outcome is typically a bunch of jupyter notebooks proving that we get at least some predictive value that helps solve the problem. If no proof: no-go
* 2 months to upgrade it to something that works well enough to run a real pilot (e.g. shadow run, run with a few agents, in an A/B test, etc.). If no success: no-go
* Then hopefully in production within another 3 months. We're in a pretty regulated industry, most of this time is _not_ engineering.
The phases help set priorities as well. Doesn't always make sense to spend (much) more time on scalability if we don't know it works yet.
One of the things that really helps this happen is when the management structure (on both sides) understands a defined problem statement that both sides agree to is a requirement for a project to have any hope of success.
The above isn't specific to Data Science projects. It's just having a management structure who knows the right time to support things being pushed back.
Unpredictable progress is lethal to businesses. As an example, it does not matter how brilliant Steve Jobs was or Elon Musk is, they both need to show regular results that build confidence from shareholders.
About the only time you may not care for steady progress is if you have abundance of resources and clear goal like Bill Gates with his foundation. This guy may believe in something and keep steady stream of funds just on basis of his beliefs that something is right.
But the funds themselves must have been gathered some other way.
It is difficult to get funds for unknown payoff at unknown time in the future.
At peace, or in a quasi steady state, yes. But not in all situations. Eg. in war or other adversarial situations you need to both be making progress and be as unpredictable as possible, in the hope that in some indefinite future you can exterminate and/or enslave the other players.
As the world is becoming more and more war-like, these types of strategies can become more and more advantageous. And you don't need actual military war or even physical violence. Actually it's even better without these, since non-military non-violent war-type adversarial state can be sustained forever!
In an arms race type of business situation, if your progress is predictable it means you're not moving fast enough, and somebody else will outrun you! And moving more from competing on marketshare (fighting for more from the same pie) to competing on progress would be awesome too.
I absolutely love unpredictable progress, and I'm really optimistic about the direction the world is evolving to... endless war and without (most of) the death and violence part will be a great catalyst for high-risk-high-reward technological and scientific progress! And as volatility increases maybe more and more of the businesses that depend on "the world staying stable and predictable" will be wiped out and create some breathing room and opportunity for new more flexible and open players. If predicting the future is no longer on the table, you need to invest in true agility, eg. ability to quickly change direction and speed when you glimpse a new future, maybe replace most specialists with teams of nimble quickly-retrainable AI-augmented-expert-generalists etc.
And back to the AI side... we need to stop playing the "predictions game", you never want to actually predict what will really happen in the real world, you want to predict what could have happened and then immediately interfere and change the direction of things to invalidate that prediction (sabotaging other players that probably also made the same prediction)... hopefully to produce a change in the direction you want... but if that can't work, "using AI to inject volatility and invalidate others' predictions" could be a valid business strategy too...
I did not intended this to apply to anything else.
I may have made an error in my post, though. Making unpredictable progress is not lethal in itself if you can show steady progress. It should have probably been worded "lack of steady progress is lethal to businesses". It is ok to have some unpredictable projects as long as you can build trust with steady progress somewhere else in sufficient quantity.
Sure, this is true in some cases, but in most cases I think anybody with a fundamental understanding of ML could tell you yes/no by understanding or estimating whether your input data contains enough signal to explain your desired output. In many cases where ML "fails" it's relatively obvious that there is not enough signal to cover the output; in many other cases you can be quite assured a priori that ML will give you a decent solution without needing to test it first.
There will of course be issues if your practitioners are unable to distinguish these cases or if they are structurally not empowered to say so (e.g. if they are just ordered to make a model using data X to do Y). That is probably what happens in a lot of cases when business leaders blindly decide to "add ML" to their business.
The fact of the matter is that ML is very good at solving problems where an input signal strongly correlates with the output signal, and it's appropriate over other approaches when the mapping is hard to define, such as in recognizing images of birds. If you can apply some transformation to your business problem into something with this structure, you can apply ML; otherwise you probably shouldn't.
This is about as constructive as most political discourse today (disclaimer: I'm working with both robotics and AI/ML folks right now, and they're all terrific)
That would be great, but it's just not how data science works. You can't know the scope and limitations of a dataset that doesn't exist yet.
In general I agree that data science suffers heavily from the "when all you have is a hammer everything looks like a nail" effect. I see so many projects that attempt to use fancy models when they could be accomplished in an Excel spreadsheet in 15 minutes.
Basically they discover that training effective models takes too much time or too much data or both.
This article doesn’t mention a really, really straightforward factor for why AI hasn’t invaded these domains despite billions of dollars being dumped into them.
An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. This immediately destroys the upside as now you’re 1) doing the process manually anyway and 2) fighting the automated system in order to do so.
99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong.
In lots of business contexts, probably most, reducing variance is much more valuable than reducing mean expenses. Variance can halt downstream production, so the loss can be some huge amount of opportunity. And variance propagates through a supply chain, so your customers will hate variance in your output, as it may mean they have to ship the variance forward, which their customers hate. Plus if you allow your supply chain to get away with inconsistency, they can start to rob you with lower average quality and it will take you time to notice.
If a company has been bothering to do some process manually and they haven't outsourced it to the cheapest humans possible, then they care more about low variance than low cost. Pitching these businesses a solution that lowers cost at the expense of unknown high variance is very unattractive. Instead, you want to tell them "I can reduce your variance even further! Here's what that would cost".
But the argument still applies then - if only one incident occurs where a catastrophic mistake was not spotted by the model failsafe, and if later investigations show that the mistake could've been easily spotted by a second human, a human will be installed as a failsafe for the failsafe.
I think the general problem is the following: if a human makes a grand mistake, it can usually be attributed to a temporal lack of care, or just random bad luck, or happened because the person was having a bad day, or... it is also generally understood that making a grand mistake will be such a shock for the person responsible that this person will most likely never make the same mistake again. On the other hand, if a machine makes a grand mistake, the intuition of the general public, trained by centuries of experience with techology, is that this machine will make the same mistake again and again and again, when prompted with the same input. If the model is not designed to learn from mistakes in production, this will of course actually be true.
Surely this will depend on the cost of failure, and the cost of the human failsafe.
Spellcheck in an email client helps prevent the minor embarrassment of typos and spelling errors in emails, and few emails are so consequential that it's worth having them carefully manually vetted.
That is frequently true. Take the job of a lifeguard for instance. A single mistake can be catastrophic, and yet we know that people have trouble staying completely focused for hours on end.
AIs have no trouble staying focused and they can be trained to spot drowing swimmers pretty well even on a crowded beach.
Having a human lifeguard plus an AI that alerts the lifeguard when it spots something suspicious could lead to better outcomes than employing two lifeguards.
I disagree. I think you'll see the same thing as with drivers falling asleep while Tesla autopilot is running. The lifeguard will let the computer do all the monitoring since a low false negative rate combined with a low incidence rate means that most bodyguards will experience the computer being 100% reliable for weeks or months at a time. In fact it's not unreasonable that if the lifeguard sees someone drowning but the computer doesn't register it as such the lifeguard may question their own judgement based on experience.
Well, that is kind of the problem with AI. How do you fix a "bug" (if you can even call it that) in a model you cannot fully understand? Do you re-train it on the catastrophic mistake and somehow give it more weight? How can you be sure that this won't lead to any problems where previously there were none? How do you explain to a customer that your model now doesn't make mistake A anymore, but now mistakes B and C frequently occur? The only safe bet is to write some auxiliary code, which first uses the AI as a black box, and afterwards explicitly checks the result for this particular mistake. If this happens again, and again, and again, you need a human to maintain and extend this auxiliary code and also adjust it to changes in the underlying model, at which point I am quite certain just using a person of average intelligence instead of AI will be cheaper, more reliable and more flexible.
It's not like Google blocking your email or YouTube account, we're taking about your friggin bank account here.
I don't know how they're still in business and growing with such a process in place.
Undoing bad automation in finance often means reversing a very cheap edit on some database and pissing off a customer who’s so powerless that it won’t affect your business anyway.
Are they?
I doubt its customers are generally aware of the AI issue, or if they are, they must assume it is not out of the ordinary.
they might have a high valuation, but thats about all the financial success they have.
And even if the inputs are great, it's hard to understand what we can do with a 99% confidence prediction. The next great unlock that society is waiting for in the AI realm applies to industries where failure is not an easily acceptable outcome.
Take self-driving cars for example. Even if we can objectively prove that current AI models can drive on current roads in current conditions with a lower fatality rate than humans (this is debatable anyway, but let's assume) - what do we do when it knows it's not confident? If we assume the driver hasn't been paying attention during the ride so far, and a scenario comes up that the AI is uncertain about...the human now has likely mere seconds (at most) to capture their surroundings, analyze the risk, and take corrective action. If we assume the driver has been paying attention during the ride, then what was the point of the AI? Moreover: what if it thought it was confident, but it was still wrong. Who do we blame? How do we mitigate future instances of it? What were once societal problems we could blame on a fallible humans are now obscure technology problems we can't introspect. That's a scary place for a lot of people to be.
Basically, we've reached the level of AI where it can be used as a backup to humans and protect us from royally fucking something up. But the next big unlock will come when it can be the primary actor. It's hard to imagine how we'll get there without the AI actually understanding what it's processing in some real way.
Edit: rephrased the 3rd paragraph.
For example, Amazon Seller Central, Google, Youtube, can "outsource" their customer service to AI, because they are pretty much the only players, so customers have to suck it and deal with the frustration of a terrible experience by not getting help and not talking with a human.
With any other business if they get automated replies that don't solve a customer issue and they are unable to talk with a human, 99% of them say "fuck it" and go somewhere else.
This is one small realm of the relationship of AI and businesses. Then you have employees, suppliers, supply chains, finances, internal processes, so many multi-variable, fragile and nuanced systems that I doubt if they're not developed in-house, they'll probably do more damages than solve problems.
I can only speculate, but while it's true humans can fail when following a process, it's also true they can sometimes spot a potential failure even then there is no process to prevent it. They can come up with new processes, or ways to improve existing ones. They can also account for their actions, and managers can account for the activities of their team. All of this builds trust that the process can be improved in ways that can be well understood.
People are also scalable. You can implement controls like four-eyes on changes and critical metrics, so you're less exposed to one person's idiosyncrasies. It's hard to do that with AI.
Think of a DL powered, visually guided robot vs. a human on a production line.
One task might be to do QC inspection at the end of the line. Suppose something gets on the camera lens. In general, the DL system will keep chugging along and the accuracy will degrade. The human will notice this and clean his glasses.
If he sees something ambiguous as it passes on the line, he might give it a bit more attention or adjust the angle he's looking at it from. If he sees a series of the same anomalies, he will notice a pattern. Perhaps one of the machines up the line from him has started to introduce a new type of defect.
Suppose in assembly a worker has a sore muscle. He might adapt his motions to compensate, slow down, go to the doctor, take some pain pills, or take a day off. Unless programmed or trained to detect this, a robot will keep driving its failing motor harder until it breaks.
Like with self-driving cars?
Pharmacy - adding 1% more of a chemical compound to a pill could be lethal, but not otherwise detected for 3-12 months (pills sit on shelves for a long time). Figuring out why that happened will be very difficult.
Manufacturing - molding plastic parts requires very specific mixes of chemicals. Too much of one and it becomes brittle under certain temperatures, too little and it warps down under the warranty period. So if you manufacture the mold of a baby car seat, and you add or subtract 1% of a chemical, that car seat could break / shatter when it's involved in a collision. Terribly large lawsuits would occur because some C-level someone adopted a process to save headcount and reduce oversight.
Business processes are a lot like software - the first 80% takes 20% of the time and the remaining 20% takes 80% of the time.
There are plenty of places where human validation shouldn’t be necessary, but is.
"... and critically: there's no one to hold responsible for getting it wrong."
Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.[2]
1. "What's more, artificial research teams lack an awareness of the specific business processes and tasks that could be automated in the first place. Researchers would need to develop an intuition of the business processes involved. We haven't seen this happen in too many areas."
2. Including the ones who designed the "AI" system.
Because whoever does achieve the next unlock - should it happen - will receive an unimaginably large windfall. This is the classic intent of venture capital. In fact, I'd suggest that AI is actually one industry where VC is doing what it does best: taking extremely risky bets with a large potential upside.
> Could this be part of "AI"'s appeal? A dream of absolving businesses and individuals from accountability.
Presently, this seems to be one of its large detractors. If I have an employee do something stupid, I can say that an employee did something stupid. People might wonder why they were allowed to do that stupid thing, and what we're going to do to prevent it from happening again, but the explanation of the source is satisfactory. We're fallible, and we understand the fallibility of others (generally speaking).
AI is not that at all. If my automation does something stupid, I still have all the blame, and yet I have nowhere else to pass it off to. "We don't understand why our AI did this really stupid thing" is, frankly, not a satisfying response (nor should it be). Businesses employing AI certainly are not absolved of any form of accountability, and are arguably exposed to more of it (since they're not able to pass the blame on to another fallible human, and have to take direct accountability of a system they built but don't fully understand).
AI also exposes the possibility of systemic error where humans would be stochastic.
A human might only identify the right number of rentable units from a spreadsheet (to pick an example from this article) 97% of the time when an AI might do it 99% of the time, but even the same human will have a different 3% error on each day. The consequences of failure are more limited and more dilute.
On the other hand, the AI may work perfectly right up until a holding company redesigns their data tables for the 100th time, whereupon it misreads every financial report with much more concentrated ill effect.
When humans are wrong, the business’ “ego” can be saved by blaming the employee who made the call, sometimes firing them. But the process goes on with the same error rate.
But when software makes the wrong call, it feels like the business itself has done the wrong thing. With no way to externalize the blame for the decision, the blame gets placed on the decision to use ml in the first place.
It has nothing to do with accuracy or the repetitiveness of the mistake. It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes, because most companies are designed to absolve everyone at the top and blame everyone at the bottom.
I've worked in an industry (prop trading) where the costs of mistakes were real, and it is the most heavily automated industry there is. I've also worked in another part of the finance industry (hedge funds) where mistakes are paid for in other people's money, and it's super obvious that managers there are way more interested in passing off blame than fixing problems relative to how interested they are in doing these things in prop trading, and there is correspondingly less automation.
It's not about people in the population at large believing that machines are more prone to repeating mistakes than humans. Look at how many people are excited about the concept of self driving cars. It's about managers trying to convince themselves that this is true, so they can go on being unethical.
You can't fire a crappy AI and replace it with one with better judgement. If it does something weird or bad or inexplicable, you can't ask why it did that and tell it to how to be better in the future. Blame is a tool that organizations and people; you can't blame AI because you mostly can't teach AI.
I think this is the core of the problem. 99% isn't good enough. Even 99.9% isn't good enough when we are talking acceptable accepted error margins. Even if humans make more mistakes than the AI, telling our customer it was a human error is much easier for our customers to accept than telling them it was a program error without our threshold tolerance.
We see this with self driving cars. People's reactions to the machines is that the machines have to be far better than humans before humans will be okay with the risks involved. This also holds for financial aspects. Imagine your grocery store telling you that there is a X% chance of being double charged for an item and that is within acceptable error tolerance. Even if X is lower than the rate that human grocers accidentally double charge will people be okay with that as the planned error rate or will they demand perfection?
Most of these issues aren't about how many 9's of reliability there are, but whether or not some person is accountable. AI is not itself accountable, only the person in charge of it.
Many companies already give gift cards or whatever when a human mistake happens as a form of customer service, and there's not enough compensation around AI error rates to make them palatable.
The veracity of this statement greatly depends on whose mistake you are saying it was.
List of things by how hard they are to say (easiest to hardest):
1) Someone else messed up. We'll deal with him.
2) Our code is bad. We'll fix it.
3) I messed up. I'm sorry.
Wouldn't this itself be error prone? The operator is, after all, human.
Let's also not forget that most business processes are bespoke to each corporation. Finding a single process that can be successfully targeted across multiple companies is hard. Some people like AWS and Google are trying with Textract and some of these other AI-as-a-service products, but they're not having a lot of success. They still fail all the time.
(1) Inflated expectations from higher/middle management which trickle down the organization. AI is seen as a high-profile case which has to lead to success (and a larger budget next year for my dept.)
(2) Data quality issues. The data itself has issues, but the key issue is lack of metadata and dispersed sources. Lack of historical labels (or them being stuck in Excel or on paper) is part of this as well. Big data without any labels is mostly useless, contrary to expectations
(3) Most AI or ML projects are not about ML. In fact, they're mostly about automation or rethinking an internal or customer-facing process. In many cases, such projects could be solved much better without a predictive component at all, or by simply sourcing a 1 cent per call API. AI is somehow seen as necessary, however, without which our CX can never be improved. ("We need a chatbot" vs. "No, you just need to think about your process flow")
(4) Deployment issues and no clean ways to measure ROI leads to projects being in development indefinitely without someone daring to stop them early. This is also related to orgs starting 30 projects in parallel (2m lead times with one to two data scientists for each), which end up all doing kind of the same preprocessing and all lead to kind of the same propensity model. No one dares to invest in long-term deeply-impacting projects as "we want to go for the low hanging fruit first"
This in turn means that it may fail catastrophically when faced with adversarial examples or with examples that are drawn from a different distribution than that of the training set.
> Specifically in regards to adversarial input, humans are often the weakest link in terms of process security
I have yet to encounter a (healthy) person who looks at a photo of static and mistakes it for a cat.
More relevant however is that we humans can understand that we're faced with an optical illusion and we can make adjustments accordingly. We have formed the concept of an "optical illusion" and we just place "The dress" in that category. A machine needs to be specifically trained on adversarial examples in order to be able to predict them. Once you come up with a different class of adversarial examples it will continue to fail to detect them. There is no understanding there, just more and more refined pattern matching.
Does a machine that can match any pattern actually "understand"? I would say no. But these are already philosophical considerations :D
Broadly speaking, yes. At the same time that's not what happened in 2015. It produced so much polarizing content with people deeply entrenched in their believes. They might have recognized it as an optical illusion, but they refused to make adjustments.
> A machine needs to be specifically trained on adversarial examples in order to be able to predict them. Once you come up with a different class of adversarial examples it will continue to fail to detect them. There is no understanding there, just more and more refined pattern matching.
Moving away from image recognition examples, isn't that exactly what happens with humans predicting whether an email is a phishing attempt? I remember reading here on Hacker News this week about phishing tests at GitLab. It had a lot of comments about tests and training employees to spot adversarial emails. Some companies are more successful than the others. It is a complicated problem; otherwise we would have solved it already. But it's the same principle because phishers come up with different ways of tricking people. And some people will fail to detect them.
This doesn't mean that the kind of thinking that goes on in the human mind is the same as the pattern-matching that goes on in an ANN (for example). Think about how ppl learn to talk. It's not like we expose infants to the wikipedia corpus and then test them on it repeatedly until they learn. There are structures in the brain that have a capacity for language - not a specific language, but language as an abstract thing. These structures are not the same as a pre-trained model.
The truth is I don't know enough about cognitive science to properly express what I'm thinking, but I'm pretty sure it's not just pattern matching :D
(To your point, though, I agree that machines can make strange errors, raising trust issues. My experience is that ML is useful in cases like recommendations or search results where a person can interact with predictions rather than being a complete replacement)
https://henrikwarne.com/2020/05/19/artificial-intelligence-a...
But even that and other processes are difficult because a lot of these corporate enterprises have a bazillion different systems that don't talk to one another. Forget data science or ML, you really just need a unified data view. Typically the workflow is some use case comes, somebody, an analyst manually pulls data from some system via the GUI (because that is all they interact with). A model is built based on that data set and the project stops dead in its tracks from there on because it's impossible to get an API to query for that data from its source system. That is a technology and business process project and will rapidly blow up into a clustefuck.
The key competitive advantage of these so called "technology" companies is really this. The ability to expose any part of your data storage and pipeline to any other part of the organization as a API. Every piece of software is built with that concept in mind.
[1] https://en.wikipedia.org/wiki/Object_Linking_and_Embedding
https://www.theverge.com/2020/5/19/21260005/microsoft-office...
Business is about exploiting what exists. This is why the buzzword is "innovation", not "invention". Incremental improvements, not qualitative jumps. So nothing will ever be really considered "Artificial Intelligence" once it is boring enough for business.
Scheduling algorithms are incredibly useful for business. There was a time when this was considered AI, but that was the time when they didn't work well enough to be useful.
> why can't I get a computer to translate my colleague's financial spreadsheet into the format my SAP software wants?
Because you probably expect it to be 100% or maybe 99.999% accurate, and we can't do that. Imagine "AI" translating someones financial spreadsheet into a different format and dropping a zero somewhere. Oops.. but your test set accuracy is 99.8984%. Still not good enough. Just getting 1 thing wrong breaks everything. This is fundamentally different from clicking on image search and ignoring the false positives.
This suggests to me that lot of office jobs will be lost just by modernizing systems and making them spit out JSON
Consider invoicing, you may have hundreds of distinct sources, but none of them individually would warrant automation.
It is a problem of standardization. Any standard complex enough to cover all business cases would be too expensive to implement, or you could not get all stakeholders to adopt it, and so on. Accountants would not want to make themselves redundant, obviously.
Websites have been able to produce "ready to print" html for a while now, and html (even with a lot of restriction) would be hundred times better than PDFs.
Much of the world is full of PDFs that are bad scans. Upside down, sideways, green, or maybe in French because why not.
>making them spit out JSON
This doesn’t just require fixing your business but every other biz you interact with. Modernize your own stuff isn’t enough. A bit like offices still list fax numbers
Mistakes are tolerable, but you need to be able to recover from them somehow, and recovery from an AI mistake seems to be something that people like to pretend is unnecessary or impossible.
We currently live in a world where failures are tolerable but seem adamant that moving to an AI driven world means that failures are no longer tolerable. We still haven't lost the ability to make mistakes (because we still do many things manually) but we really seem keen on losing it.
School grades are in an environment where you learned it recently, have no access to references, and have not been doing it very long.
If you had to keep doing algebra for even a month on a full time basis, you will get a lot better than 90%.
Even in many highly critical human endeavors, there are many errors.
The key to success is not absolute error-free perfection, it is no critical errors in components that are severe enough to kill the project.
Every rocket launch has some issues, but the successful ones have issues where it doesn't explode or land in the wrong orbit.
In the spreadsheet example, dropping a critical zero will cause damage akin to the rocket explosion. But dropping an "O" in a label field is utterly trivial.
Humans understand the distinction, constantly make such judgements and focus on the critical areas in their moment-to-moment work and embed it in their work processes. These constant criticality judgements are not just binary, but refined scaled, and serve to apply resources where needed.
The AI systems do not have such a judgement layer, and apply the same degree of inaccuracy to every part of their domain. So, absolute 100% accuracy is required, as errors are no less likely in the critical components.
Example: my wife is an admin in a school office, and a ludicrous amount of her and her colleagues' time is spent on replicating data entry between a multiplicity of different incompatible systems. The Rolls Royce / engineer's solution to this would be to provide APIs for all these disparate systems and have some orchestration propagating the data between them, except of course that's never going to be remotely practical; instead, dumbly spoofing the typing that the workers do into the existing UIs is a far more tractable approach. My (admittedly not 1st person based) experience of these things is that they currently still require significant technical input in the form of programming and "training", but this fruit has got to be hanging a lot lower than any ML-based approach.
Also, if it's been running for months before it breaks, the humans that used to do the work are gone, or have forgotten how to do it.
Similarly, seen things like using a email as a trigger, when the source application has configurable web hooks.
Feels like there's an RPA culture of sorts to assume the things being automated only have human based interfaces.
Your first statement is a trusism, any solution can be a dangerous band aid, if applied incorrectly. RPAs solve real problems now, really the only correct measure.
Of the RPAs I have raised, I always watched them work, they just prevent mistakes. I would suggest taking a screen recording and verbally annotating it for posterity. Furthermore, nothing says your RPA has to run open-loop, one can put in checks to ensure that it hasn't gone off the rails.
I also believe that the leading edge RPA systems do take advantage of some real AI techniques. And that more AI will be deployed in RPA as time goes on.
A secondary issue with business data is that vast majority of the features are categorical, for example: vendor id, client id, shipper id, etc. These usually get hot-one encoded, and you end up with hundreds of features where there's no meaningful distance metric. Random Forest and XGB are about the only that produce somewhat rational models, but in reality, they are good because they approximate reverse engineering of business process.
And lastly, the hype far outweighs the possibilities, at least until the business are ready to re-engineer the processes, if it's not too late.
You couldn't do that as well with different stores simply because malls have different demographics (meaning any conclusions could be noise) and the costs of shuffling where things are located is high so you can't just test 50 different setups and see what works.
Xerox is a more apt example of the 'missed opportunity' narrative.
The big old-line brick & mortar chains just didn't think it worthwhile to build this sort of granularity into their systems, and are paying the price for it. I suspect that many executives who grew up in the 50s-70s think in terms of "Is this change good or bad?" vs. "Why is this change good or bad?" (Note that brick & mortar retailers who have embraced extensive data operations - notably Walmart, Target, and Safeway - are doing great. It's the Sears & JC Penneys of this world that are failing.)
I wouldn't be so certain of others ignorance. Retail stores have long done studies and applied consultants to problems on layout, music, pricing, etc.
Does it actually? Specifically, what has AI done for Uber?
What makes this hard, is that for all of these companies, machine learning models are essentially used in place of heuristics of varying degrees of complexity. The models are being used to incrementally improve heuristics that in some cases are tuned quite well. Couple that with the issue that a machine learning model is only one piece of an actual product improvement for these websites/companies (ie: now you have a prediction, what are you gonna do with it to effect the product?), and all of the sudden you have actual incremental improvements completely misaligned with moonshot expectations.
We've had plenty of rankers and recommenders for decades, the current wave of deep learning variants are improvements, but they're incremental from a high level. If your business wasn't successful/is failing using simple heuristics (sears, jcp, etc), a machine learning model isn't going to magically correct that.
The hardest part about programming is that you have to say what you want to happen clearly and precisely. You can't just say "I want a text editor", you need to say all sorts of specific things about how the cursor moves through the text and how you decide what text is displayed on the screen when there is too much to show all at once and how line-wraps work and whether you acknowledge the existence of "fonts", and what happens when you click randomly on every pixel of the display.
The program usually shouldn't be the spec, but you can't write the program without actually specifying everything that can possibly happen over the course of that program executing.
One of the things that makes AI/ML so hard is that we don't want to write a spec, most of the time. If we could write a precise spec that a computer could understand, we've typically already written the program we want. There are some cases where we can, like games or math, but most of the time, what we want to do is provide our AI/ML with a bunch of data and say "you figure out what I mean". "Label the pictures of dogs", "identify the high-risk loan applicants", and so forth.
Our AI/ML is actually solving two problems: first, it has to come up with a spec on its own, and then, it has to create a solution to it.
And here is where things get rough: we generally don't know what spec our AI/ML came up with. Did we train a model to identify dogs, or to identify dog collars? Does this model find high-risk loan applicants, or people of certain ethnic backgrounds?
The problem with many real-world and business applications for AI/ML is that the spec is really, really important.
We're just getting "42" and are stumped, because we didn't describe the meaning of the question nor the answer in a formal way.
To apply meaning we need "us" or the real world. And we need to decide its form.
It can definitely do that, but you might not like the cost/benefit analysis, depending on how many such documents you want to process. The costs are coming down steadily though as the tech improves. If you need to do millions of such documents, yeah a model will probably be worth it. But if you need to do a few hundred you probably should just do them manually.
The thing is, reality has surprisingly high resolution. When you give out a task like this to a person, they will likely come back to you for clarification about how you want to deal with some of the examples. Your initial requirements will be underspecified, or incorrect, in some details. When you are dealing with a person, these minor adjustments are pretty inconsequential, and so you don't really notice it happening. The worker might also have enough context to guess what you want and not ask, and just tell you the summary when they deliver the work.
If you're training a model, you need to work through all these annoying details about what you want, just as you would when you're creating any other sort of program. This adds some overhead, and places a lower bound on how many examples you'll need to have annotated -- you'll always need enough examples of annotation to actually specify your requirements, including various corner-cases. You need enough contact with the data to realise which of your initial expectations about the task were wrong.
So there will always be a lower scaling limit, where the automation isn't worthwhile for some small volume of work. The threshold is getting lower but there will always be a trade-off.
Why isn’t this generally solved, though?
If you tell me, "Go to ebay and get me a list of the prices of washing machines", that sounds simple, but then I'm faced with some washer/dryer combo or some hand-crank contraption. These are things you didn't think of. I can either ask you, or take a guess and hand you something that needs to be cleaned up later.
If I'm instead training a model, I need to encounter these tricky examples in order to ask you for a policy on them. If I'm collecting an unbiased sample of the training data, this could take a very long time. If there's some sampling strategy maybe it's faster, but there's still a minimum number of examples we need to think about, no matter what.
1. Unexplainable algorithms that cannot demonstrate fairness and biased algorithms - causing firms to be dragged to court for discrimination - where AI was used for decision making which impacted lives/careers (lending, credit, recruitment, medical procedure suggestion, financial modeling etc - just to name a few)
2. Biased algorithms resulting in small tainted outputs that could later snowball into a larger loss that get built over slow leaks over time. (Few AI based cloud app/infra monitoring systems ending up deciding the wrong scaleout factor/sizing - based on past history but not considering real situational context/need - resulting in a net loss over a larger time)
3. Some AIfied solution just outright denying users the level of control that's really warranted. ("full automatic , no manual" mode). This mostly happens where the buyer never uses it firsthand but buys based on brochure/ppt walkthroughs, and real users are disconnected from the decision making ivory towers. The risk ibeing these systems getting into the way, instead of aiding productivity, they end up being another JIRA - a hassle one could really do without.
I lived through the first AI winter. As effective as deep learning can be, problems like model drift, lack of explainability, and getting government regulators to sign off on financial, medical, etc. models are very real problems.
Two years ago I was at the US Go Open and during a social break I was talking to a lawyer for the Justice Department and he was telling me how concerned they were about the legal problems of black box models.
No, we haven't. I mean, we've made progress in those areas, but there's still a long way to go.
The best AI in Starcraft, AlphaStar, still can't beat the strongest players without relying on simply out-clicking them.
Driverless cars are still in the testing and development phase, none of them are smart enough yet for widespread deployment.
By "not that complicated", I mean a decent CRM system to track information about the organization is approaching peak operational efficiency for most businesses. Most inefficiency I see after that is people/political problems.
By "people don't know what AI is", I mean that business owners are unable to describe their business problem as a supervised learning problem. If you can formulate your business problem as a supervised learning problem, then you can probably solve it with AI (which, yes, is really just a marketing term for supervised ML).
But most business problems are really "order taking" or "production/delivery" or "moving things through a funnel" problems and thus AI isn't the solution, CRM or CRUD apps are the solution.
I now see Google sales dragging these people out to 'AI meetings', which are about Dialog Flow. (Meeting pitch email gets around, business manager reply all: 'WTF is this?') Later they summarized the meeting, as they understood it: "You just need to drag and drop the CRM on the 'Brain' and then you can book haircut appointments and open a bank account. We were like um, we develop and support software, they were like - just drag and drop you knowledge management API on it too." Uh... what KM system?
The complicated part is that in small and midmarket business, their knowledge is tribal and processes are folklore. There are processes that are followed though they don't exist, and business rules violated as they aren't aware.
Meanwhile, no one in sales puts useful information in a CRM system if they can avoid it, the least effort principle on administrivia is important if you are ever going to hit your ever growing number.
Finally, the management must still be spooging for SPOGs, I sill see marketing for them as some sort of nirvana. Meanwhile, they have been collecting data randomly about everything, all over the place, excel documents, FTP servers, and of course databases. Here, no one in IS/IT will admit: The servers were here when they started their job and they have no idea what is on them or if they are even needed anymore. They certainly aren't mentioning that to the incoming CIO, who gets focused on improving efficiency as per his executive mandate. So he/she gets to work with cloud migrations (more hilarity ensues)
I need to stop now. It's a rat hole...
Even though what we were doing was primitive and could have benefited from a lot more ML, I suspect that even then the best that you could have gotten would have been some sort of anomaly detection system that can catch a good share of the kind of fraud that you have seen in the past, but will never be very good at detecting an intrinsic change in fraudster behavior.
On top of this, especially when dealing with humans, you often are expected to be able to explain why a certain decision was made. Setting payments aside, think of predictive policing or sentencing decisions. In those cases ML is essentially guaranteed to build in all sorts of biases regarding somebody's race, gender, place of living etc.
The ethical committee required that prior to using any data, you have to make a static copy in another database. Their argument is
1. They don't want excel files flying around (which will happen regardless) and
2. To perform any analysis, you "obviously" have to have "structured data", which "obviously" means that you have to extract a csv from the base system (MongoDB) and put that into a RDBMS (redcap).
Go figure...
When I was on one I would by default let everything through unless I came across something where I thought I could help improve the experimental design. My colleagues on the other hand would go through them like little emperors and cause grief for the applicant just because they could.
I mean, what a weird combination of tech (un)savviness
AI is in the process of having a huge effect in business software, it's just not the type of business software most of us think of when we think of business software.
Many people think of horizontal business software like MS word, excel, quickbooks, salesforce, etc. Products like this will be hard to automate significantly with AI, since every company is using these products slightly differently. The products are intentionally designed to have as wide a TAM as possible, and so they are general purpose enough to do anything business related.
There is another very large group of business software that people don't as readily know about, and that is vertical-specific business software. These products are not designed to have a wide TAM, but instead to be tailored to specific industries, and provide a ton of value as a result. These vertical products are a perfect fit for automation with AI. The author says "Each business process is a chance for automation", and in these products, these business processes (and all their inputs and outputs) are structured and represented in software.
I am building AI systems at a vertical software company right now and am a big believer in the future of AI in these products. If you have ML expertise and are interested in working on such systems, feel free to email me your resume.
A spreadsheet has very little or no spacial component for a neural network to learn, and the location of an important number in a pdf probably has little to do with the number’s significance. Without a spacial component to do pattern recognition a lot of the recent advancements in machine learning like transformers or convolution get thrown out the window.
There are some machine learning problems that can be solved with more or better data, but I don’t think PDF to JSON is one of them.
If you could script new combat units in a video game on the fly or tweak the rules slightly, the human would slaughter the AI when an equally skilled human opponent would not lose so easily or even necessarily lose at all. You can see this in games like Galactic Civilizations where you can build your own units and unusual combos confuse the heck out of the computer opponent.
Same with cars. The entire approach is currently based around exposing the AI to every possible outcome. I remember a seminar on AI Safety where the vehicle AI had a problem with plastic bags in the air and it would swerve to avoid them. No human would have an issue with that.
I worked in innovation for a bank looking at automating all these kinds of things and even spent a few days doing the jobs (and this was eye-opening) and I was a developer, so not a manager looking at a job spec, but someone who would have actually done the work. 90% of the job could be automated, but 10% was dealing with wacky exceptions, many of which they had never specifically seen before. We had someone who had the job of taking PDFs and extracting tables of income and expenses. They were generally standardized PDFs, so that seems like something good to automate right?
Well, no. As tons of the financial advisors had added custom rows which the person doing the input had to interpret into another column. It was quite eye-opening that while the jobs were menial data entry, it was nowhere near as mundane as one might imagine as the guy was still making a judgment call on whether to classify "farm income" under a person's investment income category or whether to classify it as regular income for the purpose of investment advice.
I have a friend currently on a robotic process automation internship with another bank. Same issue. When the RPA dev people actually go and do these jobs, they realize that is frequently deviates from the approved job spec with the people in them making small but significant judgment calls.
It is not a lack of knowledge about what AI can do in either of those cases. It is not a lack of data as both banks have armies of people doing it and millions of clients. It is that for AI to do the job, all manner of other things would need to be standardized and reformed and if that were done, why use AI to solve the problem in the first place as a lot of it would simply be computational.
Then that data should possibly go to an executive's inbox to remind them of the need to standardize and reform.
Although most business people are not willing or capable of solving actual problems like standardizing things.
But anyway great comment that gets at the heart of the matter.
AIs will do the same to humans when trained against other machines, instead of being trained on human match data. Since the AIs will try out things most humans would think are illegal, thus not use them in regular matches. Like when Chamley-Watson first struck an opponent with his foil from behind his back.
A huge part of the business world is service over service over human interactions that are far removed from the core value production and are side effect of these very interactions. Sure at their core business must produce something, but apart from industrial or software processes, the enterprise world is mostly a giant social game, with success linked to the execution of sales, marketing and sometimes lobbying, so not much to AI-ify from that angle.
Edit: just to be clear on the tone it's not meant to be bitter, in fact after a hard time learning this world it can even be fun.
You don't even need AI to automate a lot of these tasks. Good old fashioned programming can automate anything truly menial better than AI can, but if you're going to solve a real problem through code there are only two ways to do it: 1) write the code yourself, or 2) spend millions of dollars hiring other people to do it.
Same is true for AI. In contrast, you can very often hire people to solve the same tasks for minimum wage, or if its a sufficiently digital task, even less than that, through a service like mechanical chimp.
AI isn't used to automate away menial tasks because the economics of it doesn't make sense. None of the problems raised by that article are difficult to overcome, it's just expensive to hire people who solve them well.
This has nothing to do with technology and everything to do with the current organization of society and its economy.
The title question is equivalent to asking: a robot cannot build a car, it's so useless for manufacturing?
Robots can build cars but we need to arrange them in an "alien dreadnought".
Businesses are not prepared to pay the price of full automation, what they expect is to put some open source AI run by a fresh graduate and fire all office clerks the next day.
That is to say - when you look at business case studies of the kinds of problems that businesses perceive they are going to solve, it's things like supply-chain "We figured out when it's going to snow, we should have snow shovels in stock!" Well, of course, and there are a whole lot of humans in your company that already know this, but they aren't being heard.
A lot of the places where AI has worked out, like spell checkers, various in-app automations - as the article and people in this thread indicate, are exactly the kind of problems more companies should probably instead focus their energy. For example, I think about various gyrations I've watched people do in order to format their data the way they needed for presentations. Not AI in the theoretical sense, but definitely time consuming tasks that exist in every business that would save gobs of time and money if they could be automated away. But, so long as their isn't clear profit motive good luck getting your project green-lighted.
The people who have access to people with software and hardware have found a lot of uses for the tech - I assume AI basically is Google Image Search.
I used to work for a really well-known medical dictation/transcription, documentation, and coding (in the medical billing sense) company.
They're using ML models all over for speech to text, document analysis, etc.
It enables some very real efficiency gains but it's not positioned the same as something like IBM's Watson and it's somewhat ridiculous AI claims.
Where I've seen it used well is as a piece in a larger system of automation. In the healthcare case, it's doing a first pass at transcribing an audio dictation so that a transcriptionist can then start with a 90%+ accurate document.
This is tough, their role shifts some (more editor/correctionist than true transcriber) and not everyone makes that transition well, but the end result is 2x+ efficiency gains.
As one of the other comments points out, technology can only change productivity when you change the process, sometimes radically. And that may mean restructuring the business.
If the AI is so good that it can actually be useful for anything, people will probably stop giving it advice on how to do what they want from a simple and simple and simple task; this is the key difference between the AI and humans at work.
In fact, when I asked about what people really want from a simple task and it wasn't really a good one, I was very disappointed because it was so complex. That is why I've come to believe it's really not that useful for anything other than a little bit more automation.
This is because you could rename deep learning as "over-parameterized statistics". And statistics is just about building some model of the data. That is the only thing "training" a model is for: discovering/optimizing a statistical distribution (a distribution is a generalization of a function). This means the entirety of deep learning is simply building highly complicated statistical models of a bunch of data.
It is unlikely that this is equivalent to general intelligence found in biology.
We could probably solve the AI problem if the entirety of all of research was not directed at deep learning. And it would also likely be far more valuable to any organization or individual.
But, that is just the 2 cents of some random HN commenter. So, I will keep dreaming.
Instead of a know-it-all, AI comes across as a guess-it-all (narrowing all results by the criteria programed) and as comments stated, AI is good at producing 99% maybes.
Looking past the hype, AI has thus far just added to the cacophony of noise among so called experts. (AI is not an expert)
The specific problem the author cites - digitizing data in PDFs - shouldn't even require AI. It can be solved better by just inputting data in digital form to begin with (like with a web form), and passing it around in a standardized machine-readable data format. But the commercial real estate industry is pretty backwards, and can continue to be pretty backwards because its core competency is ownership of the real estate, and digitization labor is round-off error compared to the profits generated by it. It'll take a major recession (and this current coronapocalypse may qualify) to create selection pressures to weed out inefficient firms, and until that happens there's no incentive for them to upgrade their processes.
AI alqays looks cool, but isn't very useful in practice. The past ten years feels like everything is keynote-driven development; get something nifty looking that demos well and try to shoehorn it into a business case.
It's really just a database developed through trial and error (aka "training") that we "hope" contains enough differentiating data points to produce a reasonable, weighted "best guess".
And apologists would that philosophy has made huge, unambiguous contributions. Only once this happens, those fields tend to be no longer considered "philosophy". Astronomy, physics, economics, and logic were all sub-domains of philosophy originally. Once they were formalized, with rigorous, specialized methods they moved into their own standalone fields. But it was philosophers who laid the foundation. Consequently when we think of "philosophy", there's a lot of selection bias, because it's basically the subset of open unsolved problems that remain.
I think there'a a close analogy here with what we think of as generalized "business problems". There are many specialized sub-fields like finance, logistics, marketing, industrial psychology, and accounting. All of those things used to be thought of as a generic part of business. But eventually domain-specific methods and technologies led to the point where specialized practitioners unambiguously out-performed generalist C-suite executives.
Think of techniques like Markowitz portfolio optimization, or five factory personality testing, or applying Benford's law to profile for accounting fraud. Those are all examples where something like AI/ML solved what at the time was a generic business problem. But afterwards it was now just considered a success of the respective sub-field that those techniques helped create.
The point I'm making is that formal rules-based processes (I won't use AI/ML here because it's so ambiguous, especially in a historical context) have had a long history of success in business. We just don't recognize it because we're begging the question. What we think of as "generalized business issues" is mainly those open problems that haven't yet succumbed to specialized formal techniques.
That is, if you think you have a better idea of what you want to do that isn't the right question. I would say that if you actually get better than an AI based version of a product then it's actually a useful tool and it can be improved to be used.
For example, if you say this is AI because a particular feature that is useful does not exist then you are actually talking about AI because someone is using it, just that it's useful.
But that statement only has a part of what you mean. That is, if your idea isn't useless in the sense that it is useful in a specific way, you are simply asking how you did it.
So to answer the question is: Why is AI useless for business?
And of course, we should always compare the recommendations with the usual baselines. e.g. i sometimes hate youtube recommendations, but what if the baseline was videos that are trending or are watched by users in my country? I would hate them more.
Not as sexy as a general recommendation engine, but useful nonetheless.
Another problem is that machine learning models are never a 100% correct and not easily interpretable, so they cannot be used for some critical processes. Good luck with explaining a customer why his account blocked due to a false positive made be the AI.
I think there is still a lot of potential for boring symbolic AI, in a lot of domains you can get results quickly, reliably and if the AI is wrong it's easy to debug.
They have been flooding many of the outlets with counterfeit emotional capital (social/MSM) and counterfeit intellectual capital (psuedo-science/education/politics) for quite some time now.
As long as the "I" or the "L" in those fancy acronyms know the difference, the quality of the data will not be in dispute.
Even without AI/ML, it`s been obvious how bad data integrity is in the most basic sense of the meaning.
Let's wait 5 years and see. I predict that all the business processes he mentioned will be automated (maybe with mechanical Turk oversight). In 10 years, most of the menial desk jobs will not exist.
Without a genera A.I. (which we are a long way from) how can the process work be done and what will it mean if it can be done? We’d like to think that we’d be more productive... we’d create new and better “things”... but if history is any guide, we’d simply focus on straightforward profit generating bullshit...
ML is used in business workflows all the time - to date, I have built several solutions that are being used for 53 clients, internal and external.
Here is what makes B2B ML hard: People have to trust it.
This isn't some movie-recommendation engine, which spams you with more bank heist movies after you watch one. B2C ML systems can get it wrong, and customers are generally forgiving, because it's a low stakes game. B2B applications are generally higher-stakes, because they impact business workflows, and if someone has decided to automate it, it's probably a high-volume, critical workflow. It has to be extremely accurate, and demonstrably better than the equivalent human system.
The problem has to be well-defined enough that an ML system can act with high-accuracy, but not well-defined enough that a rule-system could replace it. Don't use ML if a rule-system will do a better job. (For those scenarios, you can still put an ML anomaly-detection system to make sure the rule-system is still valid, and to guard against data input changes.) As just mentioned, the problem also has to be important enough and high-volume enough to warrant an ML solution. The percentage of problems that fulfill these criteria is not very large.
Now to actual ML development and deployment - the model is the tip of the iceberg. The rest of the iceberg is data acquisition, feature selection, data/feature versioning, automated training, CI/CD, model performance monitoring, et cetera. If ML is being developed inside a software development organization, this isn't a problem, most people will understand this. If it is being developed within an embedded BI team inside a business unit - they will generally not have support/runway needed to build the full system. The ML model might make it to production, but it will probably run naked, be brittle, and hard to retrain. A dramatic failure with business impact is just a matter of time.
There are a lot of low-code, no-code ML solutions that have been developed, or are being developed, and some of the supporting infrastructure as well, but, at the risk of sounding parochial/protectionist, you need a rock-solid, end-to-end, integrated, data management system that is fully understood by whomever needs to pick up the phone at 2AM. It's the interfaces that are hard, and chaining together a bunch of third-party black-box systems just means more interfaces and behavior you don't control. Choose and use these systems wisely.
So yeah, B2B ML is hard. But it's generally not due to lack of data, and transfer learning is generally not necessary. Understanding business processes is important, I agree, but that's comparatively easy. It's what consultants have been doing for decades. The hard part is choosing a problem where ML can add value, and then executing on it with enough integrity that people will actually trust it.
(1) Ok, clickbait might be harsh. But it is self-promotion, and the article itself is a collection of generic banalities. I feel it falls on the wrong side of the line.
Well the business people actually do understand AI, but their understanding of it is that it is a marketing tool they can use to sell to customers and/or investors. And in terms of doing that, it AI works very well.
Business needs to solve the prediction (i.e., regression) problem, which is a completely different kettle of fish.
P.S. Of course by "AI" I mean the 2020 definition of "multilayered neural network".
You might think about this a few times - if you are able to make an AI out of the box, maybe you have all of the necessary knowledge to get it out as well. Just as a computer can make a database out of a document. A programmer, however, could do all the necessary knowledge to get that system working. But most software engineer is like a carpenter - no amount of math or programming will change that. They probably also have all of the necessary knowledge necessary to make a car, so how about a car that can take a picture, and a car, that can run the calculations of the wheels
Ironically enough, after data robot did a layoff they also completed a large acquisition and hired more executives.
Because it doesn't make money? A big enough revelation?
It will be when AI systems will decide who to hire and who to fire and who to promote and demote, or what other companies to acquire or to merge with - profit will be increased, and almost everyone will hate it!
It will be when huge fusioned megacorps AI systems will gain monopolies and replace free markets with centralized planning systems that will actually outperform markets ("socialist planning" can't work because it can't work with humans ...bringing in "other" types of intelligences will change the game, and nobody will call it "socialism" bc it will not even try to benefit the people this time around - and there will be markets still, just likely HFT-style ones that will block direct human actors from playing in them) ...and most will hate it and likely wage war against the societies that will embrace it this way!
You'll see AI stops being useless to business, don't worry ...but it will come with many consequences and side effects, our society as it is can't handle it!
And if you do what the AI suggests, it will work and you will prosper.
But nowadays we find that the AI buzzword sells really well so we decided to lower and lower the bar until almost any algorithm qualifies as AI (also, any machine qualifies as a robot).
>Today's work in artificial intelligence is amazing. We've taught computers to beat the most advanced players in the most complex games. We've taught them to drive cars and create photo-realistic videos and images of people. They can re-create works of fine-art and emulate the best writers.
Today's work in ML is amazing.
> We've taught computers to beat the most advanced players in the most complex games.
Not true. You can spend a zillion dollars on self play to get an AI superhuman at games simple enough that you can simulate at many many times real life speed, but we're just now learning to do games like poker, which intuitively seems less intellectual than Go or Chess, but so does starcraft and that came after those other games. In ML, placing tasks in order of achievable to currently impossible can be really unintuitive for lay people.
> We've taught them to drive cars and create photo-realistic videos and images of people.
No again. We're getting there with cars, but it turns out that it's really really hard. Harder than playing superhuman chess! But people who play chess better than computers can drive cars better than computers. Weird, right? Again, in ML, placing tasks in order of achievable to currently impossible can be really unintuitive for lay people.
We can make photorealistic pictures of people, but we're sorta limited (it's complicated) to faces at high resolutions and just really really really recently getting them without weird artifacts. But the face is the most complex part of the body, right? So the rest should be easy!
> They can re-create works of fine-art and emulate the best writers.
This is soooo much of a nope, and you know what I'm going to say anways.
This xkcd is always relevant, even if the bar has moved. Maybe it's even harder because the bar is moving quickly. https://xkcd.com/1425/
> In CS, it can be hard to explain the difference between the easy and the virtually impossible.
In ML, we're really good at some tasks, so it seems like we should be good at adjacent tasks, but that's not how it works.
In our application, one key operation is that the user is required to classify a line item based on the text description. There's a huge code list of possible classifications, and the user has to pick one that is the most correct.
This is definitely a task that registers consciously. And, while most of the time it's fairly easy for trained users, there are often cases which require extra thought or research.
For example, a T-shirt of mostly cotton vs a T-shirt of mostly synthetic fibers should be classified differently. How would you know based on the description "Small Womens V neck Short"?
Of course, an AI/ML system that could reliably classify a majority, and reliably classify the rest as "unknown", would be interesting. Not sure how close we are to that though.