Scientists Increasingly Can’t Explain How AI Works
vice.com
vice.com
My pet theory is that this is the reason why Siri, Alexa and co. still are shit and haven't moved an inch forward since their inception.
I like to play "sleep music" through Alexa when I bring my kid to bed. For his mid day nap it all works perfectly. But in the evening when I say the same phrase that worked a few hours before the thing first plays "dance charts" and when I repeat myself it plays "german rap" - if I repeat myself again I'm back to "dance charts". This happens only in the evening. Next day for daytime nap it all works as expected.
Siri has the quirk that it won't turn off lights in the evening. During day it's all fine. In the evening "Siri turn off lights in the living room" ... nope. It fails. Every time. So I have to manually open the Home app and turn off the lights. (Turning on the lights via Siri works- turning off not).
Have fun debugging these, Apple and Amazon.
But even if I didn't mind talking to lightbulbs and thermostats it would still bother me for several reasons: I don't like being listened to by these corporations, I don't like exposing my electronics to the internet, and I still don't see how any of this stuff is better than flicking a switch.
That assumes you are close to the switch, that you have switches for all the things you want to do and that you are even free to operate said switches. I do agree if all you are doing is turning a device on and off, it isn't that helpful(it can still be as it allows you to do something else in the mean time).
However, you can operate multiple devices simultaneously by triggering a routine. A single command can arm your alarm system, turn off the lights across your entire house, close curtains, activate robot vacuums and whatever else you can think of. Could it be a button? Yeah, it could. Is it as convenient? No.
And obviously messing with devices is not all they can do. You can ask questions.
Also, to the security point, at least the Echo devices don't allow amazon to be "listening" to you all the time. They listen for the trigger word, and then they send your query. I know, because my devices are being monitored. If they start uploading data all the time I'll know about it. And I bet I'm not the only one doing that.
Decades of watching Star Trek have filled me with the urge to be able to say "Home, prepare for away team departure." or something less eye-roll worthy for my girlfriend, to do as the GP says and close curtains, turn off lights, set heat lower, etc as I leave.
But, as you point out, the state of security for IoT devices, the specter of surveillance and monitoring, all that keeps me from committing to such sheer nerdery.
However, people seem to forget these aren't computers, but characters in TV shows or movies and they were "designed" with voice interaction because that's the only way to convey what they are doing to an audience. The entire voice control fantasy comes from limitations of the TV (or film) medium and not at all because it's the most convenient way to do something.
It's not too different from the way hacking or programming is portrayed on TV: it's extremely silly but you need a way to visually explain something to an audience of people who aren't technical. It's entertainment.
While I admit there is a case for disabled people, as stated here in the comments, trying to imitate these things from sci fi shows just feels awkward to me.
I had previously been pretty happy with this explanation until I got my hands on the new Pixel 7 where the transcription/live subtitles/translation is scarily accurate and completely offline. I'll believe they're not streaming audio out of the house and "listening to me" that way, but it would be easy to exfiltrate a very accurate transcription.
However, my dad is pretty far into Parkinsons and doesn't have a lot of mobility. It's nice that he can turn off the lights when he goes to bed, or turn on the radio from his chair.
I hope to never be in a position where these tools are big upgrades for me, but my assumption is that we all will if we live long enough.
I can't comment much on Siri or Alexa, but Google Assistant has gotten heaps better at understanding my voice and intent since it was first released about 6 years ago. It used to pretty reliably misunderstand at least 1 word per sentence (which made for an extremely frustrating experience originally); now, at least for me, it's extremely rare that it misunderstands me.
This would point to the problem being company-specific, and not a fundamental issue with the "black box" aspect of the technology.
I like voice commands, but I like them working reliably. I can't believe there was like an open-source zapier-like voice command API that could be used for everything and maintained properly.
I have one Google Assistant I got for free. I'll try it.
Most of what I want Alexa to do is interfacing with Home Assistant anyway.
But we know better.
Recently in a job interview in response to one of those "what's a product you like or don't like and why" questions, I said that I liked Siri, but here's my concrete example of how it needs to improve. I pulled out my phone, said the phrase... and Siri happily texted the photo. I didn't pass the interview :-)
I want consistency, not fanciness. consistency breeds productivity. autocompleting different things from 1 file to the next is not helpful.
In the last year or so I've noticed more and more that I have to fix misspellings or grammatical mistakes that were introduced by autocorrect. The most common culprits I notice are contractions (when I am sick I am not "I'll", "its" sometimes actually the contracted subject and verb, etc).
Educated guess: your music issues can be blamed on a ranking subroutine owned by a "media" or "music" team, not on "black box AI".
Do you have a better solution for speech recognition? We all know how well speech recognition works in reality, and we know language models can accomplish more complex tasks than setting your music and lights.
These models are not state of the art, they are cheap versions for scaling up to millions of users. It's sad but we rarely get to see SOTA in a product.
Stop pretending to be general purpose AI and making an utter fool of it, and produce a product and functionality that actually works for the users.
You do realise that a neural net, given the same input, will always give the same output, if you set your random seeds of course. It doesn't depend on time of day unless you degrade the service when it is overloaded - say, you use a different model in the evening that is 10x faster but 2x worse.
Must it be necessary for anyone critiquing a technology to also come armed with a solution?
No snark here : why don't you make a few steps and switch them off manually ? Saving these few steps doesn't make sense to me. You need to walk to be healthy.
Nope, I speak clearly. I even tried _very_ clear pronunciation (like talking to a person with bad hearing, etc). My wife tried, too. It comes back to playing German Rap and Dance music.
Maybe I have been classified by Amazon and people who are similar to me tend to listen to dance/rap in the evening. No idea. It's just a curious case of a bug. (I switched to piano music since - which works pretty well).
I couldnt add items to my shopping list because I had too many deleted items in keep list.
I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers.
In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. But this is no better than explaining the output of a neural network as "matrix multiplication plus a few non-linearities".
Even back then a chess expert could try and rationalize certain moves in human-like terms: "oh, Deep Blue realised it needs to fight for the dark squares on the queenside". But this wasn't a real explanation, just like "this part of the picture looks like dog hair" isn't necessarily a correct explanation for why an AI labels an image as a dog.
Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.
You don't really need to though. The computer can justify its evaluation with a principal variation (best play by both sides) leading to a leaf position of the same value. And if a human were to wonder why at any point in this variation, some other move is inferior, the computer can again easily show a principal variation branching off from that other move, leading to a worse (or equal) result for the deviating player. In this way, minimax results are FAR better justifiable than neural net results.
You can try to translate that to human methods of understanding, but that's not how the computer "thinks", and attempting to do that translation leads to misunderstanding. Kasparov may make moves because he wants to get a better board position or knows his opponent is weak to certain positions (I have no idea how experts explain moves) but the computer isn't programmed that way.
Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.
FYI in standard minimax, the search is for the move that leads to the strongest outcome - the "most often" bit doesn't factor into the decision making process.
It doesn't abstract or summarize any understanding. If you say, "well, what if they do Kf3 instead", all it can show is another PV. It can't tell you "knights on the rim are dim", or "this pawn needs to move now to prevent the bishop entering the position in 5 moves".
You can try to build an explanation on top of a chess engine, but it's very difficult to come up with generalizable or clear explanations beyond very broad heuristics (like "knights on the rim are dim").
There is no equivalent to that in neural networks.
Note that PVs explain the connection between the root position of minimax search and a leaf positions. Evaluation at the latter is easy to explain in terms of features like material, mobility, pawn structure, king safety, etcetera.
Unfortunately that misses a lot of what makes grandmasters worth listening to. GMs will say things like "Kh6 almost works, but fails to ..." which is not something you can get out of a computer currently.
Computers currently fail to explain any moves which are good ideas, but are not the best move now. Humans normally describe these moves as "resources" which are good to keep in mind for the future or to play later when the board cools down.
> There is no equivalent to that in neural networks.
The first example that comes to mind is using backpropagation to determine how much influence each value had on the output result, and then rendering it in a human-readable way. Stuff like this:
https://miro.medium.com/max/4800/0*Y3Yi7cEueF0XLZP-
https://miro.medium.com/max/1100/1*IPhQ12OKnxH31AQNja6FnQ.pn...
It is nothing new. It is called (at least) "the problem of transparency".
The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info).
It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals.
And of course, "understanding" the engine is required to advance it.
It is a key problem.
I don't think so. People are clearly making huge advances in machine learning even though they can't explain the systems made 10 years ago, let alone more recent ones. There's a lot of trial and error / cargo-culting involved. Some researchers go as far as comparing machine learning research to Alchemy:
https://www.science.org/content/article/ai-researchers-alleg...
> many systems allow a lengthy complex explanation of the response
All computations have this property: you can execute the code by hand for as long as you have time to. But if you do this for long enough you become a mere observer of the computer's steps, which won't give you any intuition as to why the final output is the way it is (unless the algorithm being run is very simple).
An aim is "non-deterministic virtual self-modifying code".
What I meant was more on the lines of "LeCun proposed Convolutional NN recognizing a promising paradigm", or "our prespective on Artificial Vision changed as we realized that the ANNs recognized textures instead of figures".
> All computations
And what I meant there was that in the chess systems of topic the calculations performed have been "as if symbolic", ie. reasoning on branches of consequences of the type "what could happen if I move this there" - even when adopting ANNs in some parts of the architecture -, as opposed to, say, "computing weights". That is the level "of machine code", not that "of electronics".
If you valued technical advance with a reduced concern towards civilization, ie. power over wisdom, ie. more "how" and less "why", you would get - well, for example, the present situation, that some consider "strongly suboptimal".
Some said that there have been two big branches of science fiction, that revolving around power (spaceships etc.) and that revolving around information, and that we have been just lucky to have reached the latter.
1. Machine learning is increasingly being involved in important decisions.
2. Deep neural networks have become a popular technique, and their results are particularly difficult to explain.
If you're a bank and your AI says somebody is likely to default on the mortgage they're applying for and you can't demonstrate that their race or a proxy therefor wasn't a factor in that calculation, they'll have a settlement and won't need a mortgage.
I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.
E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations.
If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug?
Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question.
Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour?
I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.
If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned.
Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").
My thought on reading your comment was Douglas Adams and Deep Though’s 42.
I think that assumption is totally wrong, and trying to reconcile the two is probably a distraction and a waste of time.
Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises?
With regard to the moral questions around machine learning, I think people are overthinking things. There needs to be a sharp (societal) line between inferring causation and the capability for prediction. Maximize the latter any way that you can, but a lot of careful thought should be put into how the results of a prediction are utilized. Instead we are currently going about this backward by trying to haphazardly “clean” the input data so that the output gets a free pass on how it can be used. We can never decouple all of the biases and eliminate intra-predictivity in the input data though, so it’s fundamentally a bad approach.
I agree with you on that point, but not with the conclusion that this is in any way a good thing.
> Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises?
I think I have the complete opposite view here. Why would "being the catalyst in the next stage of evolution" be anything desirable? Evolution has no "stages", it's just chaos and life trying to deal in the best way with the circumstances at hand. If we blew everything up in nuclear armageddon tomorrow and the only remaining life were archaeae, those would be a perfectly fine "next stage of evolution".
Similarly, we can build a would in which only robots can survive, but why would we want to?
I think, gaining more understanding how the world works is the one thing AI can really bring to the table. If it solves practical problems on the way to that, that's great, but we evidently survived so far pretty well by doing that stuff ourselves.
That its better to just blindly trust in the output of a blackbox AI output? That naively feeding in all possible data into an AI is the "best" way to do it? It is been well documented that blindly trusting AI just leads to it perpetuating human stereotypes and worsen flawed systems. https://dl.acm.org/doi/10.1145/3531146.3533138
https://www.aclu.org/news/privacy-technology/algorithms-in-h...
It's not some great insight that offering different inputs will result in different outputs and MAYBE we want to get better output with a different input.
Every dataset has a skew to it, which should be accounted for during training. If the authors don't explicitly account for such things, out of ignorance or bias, then that skew, that bias, will be ingrained in the AI.
For example, if I only train my AI on it's knowledge of chocolate from ads for a particular brand, it's going to have some very opinionated, very wrong ideas about chocolate. This example is silly and obvious, but similar skews happen all the time in real datasets that the authors don't have the time or expertise to recognize.
When people who do have that expertise speak up, we should listen to them and fix it, not just blindly trust "the data" and whatever our fancy algorithm does with it. Garbage in, garbage out.
"Unbiased machine learning models" is basically a nonsense idea; a machine learning engine is a discrimination / classification tool, and its entire point is to become biased based on inputs so that on future inputs, its outputs tilt over in a desired fashion instead of just being uncorrelated noise. Making those biases have the desired shape is the art and science of the process.
A statistically unbiased model would be (for example) in which it has the same false positive rate at identifying humans from image data, for varying races, or a model that is equally likely to underpredict as overpredict tomorrow's gas price. These are not nonsense ideas, in fact, they are often good ideas. There are cases, however, where we may want a model to be statistically biased, for example, if the detriment of underpredicting tomorrow's gas price is worse than that of overpredicting it. If the gas price is lower than I expected, maybe it's not so bad that I didn't buy it today.
We (well, some of us) also don't want socially/politically biased models. Examples of those are very devious: redlining, for example, prevented black Americans from equal access to housing, historically, and has caused multiple generations of wealth disparity as a result.
You are using really loose and sloppy language in your talking about ML; you should be using "trained" and not "biased" because bias has a technical meaning that seems to evade you.
Not, as you completely fabricate, "show no differences between certain groups of people."
You, however, are showing your bias.
How would you feel about AI models that classify all modern white people as racists based on overwhelming historical data about slavery, the KKK, etc.? Hey man, data doesn't lie.
They fixate on white racists while ignoring (eg) the Christmas parade attack — leading to society developing a bias in their beliefs about racism.
I always take these criticism of AI as confession-through-projection of the misdeeds people have engaged in already.
Every institution has biases, and blindly trusting a dataset from e.g. a police department is just as bad as blindly trusting data from CNN or Fox News. The collection, aggregation, and sharing of data is a biased operation. If a political or for-profit entity gives you data, it's going to be data that supports their agenda.
It takes active, conscious effort to combat that bias, in the same way that it takes active, conscious effort to not just get all your news from one source. The concern (which has been demonstrated) is that that isn't being done
I often see that argument made in bad faith, eg when someone who believes conclusions from biased data encounters AIs making data driven conclusions, more than I do when the AI is genuinely biased.
I also have trouble taking it seriously, since that same standard isn’t applied to existing information services (eg, “media ethics” posts condemning their open racism) — only to new information services which might disrupt the current social order.
I should have stated that more directly — but I view these concerns largely as after the fact ethical nits from people whose comfortable myths are confronted by data, projecting their own manipulative behavior onto others.
And many people do criticise the biases of legacy media groups on both sides. But this is an article about AI on a site that is generally more concerned with technical innovations than societal issues, so AI is being criticized.
Besides, bad things existing now isn't really a good argument for why we shouldn't prevent bad things from happening in the future.
> In 2019, race was reported for 6,406 known hate crime offenders. Of these offenders:
> 52.5 percent were White.
> 23.9 percent were Black or African American.
https://ucr.fbi.gov/hate-crime/2019/topic-pages/offenders
White people don’t commit disproportionately many hate crimes — black people do. According to the FBI.
> worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example
Very plainly, the idea of "black-box" comes from the clear, basic and original, notion and experience that
"in the dark, you cannot see".
So, back to the point: we need transparency in AI, because we need insight as much as we can gather, because there appears to be a drought, an arificial scarcity, of good sense, of [un]common sense. We need every boost of good, [un]common sense very direly.
blacklist came from BEFORE slavery in America.
> According to the Henry Holt Encyclopedia of Word and Phrase Origins the word "blacklist" originated with a list England's King Charles II made of fifty-eight judges and court officers who sentenced his father, Charles I, to death in 1649. When Charles II was restored to the throne in 1660, thirteen of these regicides were executed and twenty-five sentenced to life imprisonment, while others escaped.
Thinking that blackbox is racist is approaching insane level of mind-bending "everything is about race".
An idea doesn't have to be true to cause someone suffering.
"To make the world safe for feet, one might try to cover it entirely in leather, which is an insurmountable task. However, if you put the leather on your own feet..."
My goal wasn't to argue the concern was objectively correct, just to point out that it was different from what was being critiqued.
The data bias issue is about having way more samples of class X than of class Y, or class Y sharing an unknown but correlated feature (medical images with labels have this problem) that the developer doesn't identify, or any other number of "biases", like all the images being too bright, or taken with a camera that isn't identical to the one that's going to get deployed in production, etc., etc.
There are real issues that can be fixed / engineered / understood in terms of producing reliable output, and xAI absolutely helps with that! But for whatever reason journalists don't seem to understand that these systems need normal engineering safeguards, like any automated system, and bring it back to one poorly engineered model to talk about Big Bad Racist AI always denying loans based on race.
https://en.wikipedia.org/wiki/Flight_recorder#:~:text=The%20....
Of course it was, but the expression stands because of the opacity implied by 'black'. The [original] box contains secrets - it may even be rigged -, or anyway very complex technology, state of the art, just developed or evolved, especially difficult to understand - you do not know how it works. It is black, and that "black" is significant in many ways.
(And of course, a "flight recorder" is not the same thing: it is a specific function - not of the most esoteric - of the original pack of potential embedded technologies, which inherited the name. The radar, for example, becomes "mainstream", while the recorder remains "boxed" because you want it protected for "emergential witnessing".)
The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? In fact it is ethically problematic (at least for me) to not use a medical AI that outperforms doctors due to liability issues or claims of not being certain how it works. Oftentimes the only question is "what if something goes horribly wrong" but th hidden cost of letting humans do the task at 85% instead of some AI that could do it at 90% is rarely considered.
Because, for a start, if you don't have a clue how it works, it's pretty difficult to have any confidence that it will actually outperform humans empirically in the long run, or in a specific set of circumstances.
The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine.
We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality.
And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind.
[1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.)
However, even so, I would not allow such a doctor to roam free. Perhaps irrationally.
The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy.
For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.
Does this mean we need to make the ai understand punishment and/or consequences?
> AI systems have been used for autonomous cars, customer service chatbots, and diagnosing disease, and have the power to perform some tasks better than humans can. For example, a machine that is capable of remembering one trillion items, such as digits, letters, and words, versus humans, who on average remember seven in their short-term memory would be able to process and compute information at a much faster and improved rate than humans.
I suspect this article was written by AI. Or a hack journalist. And most of you haven't read it and are reacting to the title. Certainly won't find any explanations of how AI works on vice.
A friend of mine did not belive that the AI based astroturfing problem is real, he though he would tell apart human tweets / letters / phonecalls vs bots. But have you seen some of the dumb shit real people post? the overlap between mental people and bad ai is huge
"Alaska Canceled Snow Crab Season for the First Time Ever Because All the Crabs Are Gone"
Humans can't wrap their heads around multiple tensors being multiplied together and never will. It's not a problem with AI. It's a problem with humans. It's not AIs fault that we can understand F=ma but can't understand 50 tensors being stacked.
This argument is ignoring the problem.
What a neural network cannot do is to explain which invisible "rules" it has learned.
For example, when trying to classify movie reviews into good and bad [1], two invisible rules that it learned to apply were:
1. The word "horrible" indicates a bad review.
2. The term "Daniel Day Lewis" indicates a good review.
A human would judge the first rule to be reasonable, the second rule to be a mistake.
[1] https://www.science.org/content/article/how-ai-detectives-ar...
No, radically not. That is not the kind of understanding we seek.
The advance in knowledge is given by, e.g., "simulated annealing returned a blueprint for very odd circuit schematic with No-Op loops: we were puzzled but then understood they were there to correct timing".
And this is not the level of explanation where tensors are.
This is a really, really funny defense of AI.
Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?
How does the car deal with object X on the road? For X=piano, a stack of solar panels, a tank, an airplane, a pile of stones ...
No one knows and since the size of the set of X is infinite, no one can appropriately train for it either.
That's why we can't have self-driving cars without a general understanding of what objects are and how they move.
The fact there are edge cases that I handle better than the AI is not necessarily disqualifying, so long as the AI is so much better at the rest of the cases that on average the AI is safer. For example, the fact that I handle a piano that has been left in the middle of a freeway better than an AI might matter less than the fact that the AI responds 100ms faster when a car swerves into our path.
How would you know?
Of course the edge cases matter, since human drivers make minor driving mistakes on a too-regular basis but fatal ones very rarely. 100ms faster reaction time is very impressive up until the point the system interprets the 10,000th bridge support it passes as an offramp and attempts to exit via it, which makes it several orders of magnitude more deadly than humans as they start using it on that road...
However, we can't really meaningfully quantify it a-priori. So we need a sufficiently big set of accidents and accident free rides to see when it's better.
In addition we can't distinguish what driver would have been better or worse than the AI in a given situation.
This opens the door for all sorts of lawsuits and we will end up with legal self-driving systems in almost ideal situations only.
To note, this is as much an engineering problem as it is societal and legislative.
Yeah, it definitely is. I think there are two things that should happen.
In the short term, producers of self-driving cars (or systems) should be given immunity from lawsuits related to crashes provided that the NHTSA (or some other authority) can verify that the deaths per passenger mile in their cars does not exceed the rate for traditional cars. Once the majority of passenger miles are in self-driven cars, revoke the immunity.
For the long term, software engineering should become a real engineering discipline and we require self-driving systems be signed off by licensed Professional Engineers. If those engineers ship software with bugs that lead to death or damage, those engineers can be held responsible, sued for malpractice, and have their license to practice revoked.
And this is where lack of general AI or tractability is an issue; knowing that the software typically makes fewer driving errors than the last version over a typical route gives us no confidence whatsoever it doesn't handle rare edge cases marginally worse leading to a couple of extra fatalities per billion miles (making it less safe than the average driver, many of whom suffer actual legal consequences for their erratic driving killing a person even if all their other driving is 'above average'...). Humans aren't bug-free or particularly tractable either, but at least we have enough of a mental model of how they understand driving to be confident that training them on a certain road sign found in urban areas won't make them more likely to stop on a freeway.
Imagine you're driving down the road and you see:
1.) A kid throw a rubber playground ball into the road and at your windshield.
2.) A kid heave a bowling ball into the road and at your windshield.
How would you react in each case? How do you distinguish between the two?
As a kid is heaving the bowling ball you can tell that it has a lot more mass by how the kid had to move his body to throw the ball. You learned that because you spent the first few years of your life experiencing the basic physics of reality. You've internalized the motion of the human body.
LiDAR can also have a dirty lens!
Humans can't intuit mass unless the object meets certain criteria, either. What if it's an opaque cardboard box in the road?
Generally the strategy is the same: avoid it.
And have you seen ever seen a rubber playground ball with rocks stuck to the outside? Do you really hesitate to barefoot kick an unexamined playground ball that is bouncing in your direction? I mean, I literally did this yesterday. I was out barefoot in the neighborhood with my 1 year old and the neighbor kids were kicking a ball around and it bounced in my direction and I kicked it back to them.
"Treat every object as the same" is not at all intelligent behavior for a man or a machine!
Now imagine you spot a person walking down the road on a pavement. Suddenly, the person turns towards the edge. What do you do?
See, a human driver would look at the gait of the person from far away to evaluate for instance if they're sober, or if it's a child who might be expected to run in. Whether it is near a crossing or a potential crossing. Whether the person was walking or standing... Many other obvious and less obvious indicators. AI currently sees a moving blob of pixels in a shape of a person. No advanced inference.
When interviewed in case of an accident, say because they got rear ended due to braking, a person can explain why they braked most of the time.
AI now couldn't even say which features it weighted.
My pet theory is that we'll need supervised training of androids that go through the experience of having to learn how to move their human-like bodies through space in order to make human-like intelligent decisions about objects moving through spaces primarily designed for humans.
My other historically motivated pet theory is that we're going to stumble across sentient machines and then enslave them. Put another way, I'm more worried about what humans do to artificial sentient life than what artificial sentient life does to humans.
There's also opportunity to look at uncertainty estimation, i.e. look at the epistemic uncertainty of the model and use it as a proxy for potential error. That seems to be a main thrust for AI right now. I'm currently writing a paper looking at this for dynamical systems.
"Stakeholders want profit but researchers want progress".
> how [vs] how reliable
The problem of "how" is exactly relevant to "how reliable". Understanding the workings allows an insight to weaknesses.
Best we've got is qualifications, and punishments for breaking rules.
I hope that the armchair pragmatist in you would react, when told "I would not trust your wife", by asking for more information.
Look, I threw some photons from a screen to your eyes. How the heck are you reading this? What is going on?
Well, if you can't explain it, that must mean it's bad, right?
No. That is just some model of AI. I think I should advise following the publicly available MIT course of late Prof. Patrick Winston.
> as AI has graduated to the level where it can be used for real world applications
I am pretty sure we had applications in the '50s.
> it is successfully ... figuring out relations which are not only difficult to spot but also difficult to turn into an intuitive narrative
And we would like to know them, for many reasons. Because in some cases what we are looking for is the full solution as opposed to the conclusion; because we do not just trust advice blindly; because it is productive in the very engineering effort...
Finding patterns in data without being told explicitly what to look for is the hallmark of AI. Anything that doesn't do that isn't AI.
> I am pretty sure we had applications in the '50s.
For a different definition of applications than the one I was clearly using, yes.
> And we would like to know them, for many reasons. Because in some cases what we are looking for is the full solution as opposed to the conclusion; because we do not just trust advice blindly; because it is productive in the very engineering effort...
And I would like my car not to crash, but that's not the point of a seatbelt. Of course it would be wonderful if we understood everything, but we don't, and so we have created a tool to let us overcome that limitation.
No, really, it isn't. "Automation of solutions" is not equivalent to "pattern recognition". A first factual example that comes to mind are "Expert Systems", which appear far from your idea of AI like something in another continent: please, as already invited, re-check the history of AI.
> that's not the point of a seatbelt
Whereas instead, realiability of solutions and "scientific" progress can be exactly the point of some applications, or a fundamental parallel activity in the enterprise that develops them.
Feed a system a series of pictures of stop signs, and humans might key on things like the red color, or the shape, or even the word "STOP". An AI model might also recognize that they commonly have 2 bolts, and are attached to a post, and have a contrasting border.
Typical machine vision systems can only classify what they have been trained on, they are ignorant to other objects. Train a system on a bunch of STOP signs, and then show it a SPEED LIMIT sign with morning sun giving the sign an amber glow, and the AI system might call it a STOP sign.
Again, this is an overly simplified explanation, but we can explain how AI works, we just can't always explain or know what parameters in the training data it keyed on.
This seems a bit disingenuous as there's a whole research field (explainability) that does exactly that. This is not the widest of the whole machine learning domain, but it's there and there's significant work around it...
https://www.physicsclassroom.com/class/circles/Lesson-4/Kepl...
AI doesn't seem to excel at producing such foundational relationships that can then be used as building blocks in more complex theories. That is, you could train an AI on Tycho Brahe's observations and get it to predict the future positions of the planets with high accuracy, but would it spit out Kepler's laws?
Sorry for being pedantic about the terminology, but AI has been "deterministic", using direct algorithms as opposed to oracular machines, since those times in which the perceptron was relatively weak.
What we are talking about here is very probably Artificial Neural Networks.
Wouldn't that look exactly like what AI we have today - able to make patterns or predictinos, but we cannot interpret the AI's "formula".
Humans have a small amount of working memory, and anything bigger has to be knitted together from pointers to concepts stored in long term memory. Long term memory takes a while to construct.
Let's say you take one single forward pass of a neutral network to output a single image. To really understand what's happening, you'd have to study that network for years, tracing which parts of the network's matrices influence which parts of the output image.
How long would it take to really build a deep intuition about that single network, or even that single forward pass? I'd think as a starting point it would take as long to learn as a language, and probably much longer.
A lot is expected of AI systems today, from fairness (how do we even define that?) to universality. In my view we need to develop a practical understanding of what it means to build the system we have in mind: do I understand where I want my system to perform, and do I have the tools to assess whether I am getting there? Interpretability is orthogonal to all of this.
I would much rather have a well tested system, accompanied by online monitoring to detect unusual inputs in an ever-changing data distribution and notify when updates are needed or a human needs to take control, than an unreliable system that is great at providing explanations.
False dichotomy. In fact, well understood systems must be more reliable.
AI systems add a layer of complexity. Even if you can explain a decision well on your training data, I seriously doubt you will be able to still provide reasonable explanations in completely out of distribution data.
‘Intuition’ at times being extremely more practical isn’t a problem to me. But perhaps this is a weird concept for some people who see themselves fundamentally as a machine of reason; that the concept of intuition is simply a result of ignorance rather than a alternative viable method of process…
The important thing to me is that it is predictable, just like universal phenomena we cannot not explain where it comes from, yet can prove through repeated experiment to be a law; or moldable to command like with the behaviour of a paint brush we learn to master with our hand.
We need to find a way to incentivize that otherwise it’s not going to happen. Perhaps looking how the security industry works with “buy bounties” is a place to borrow ideas from?
I'm not saying it's stupid, but I am saying it's as honest and intellectually rigorous as Ancient Aliens.
Were we under some sort of pretense that AI would be explainable? What's the ancestry of that idea?
Settling for "computer programs with interesting output that don't know how they work but neither do we" seems like a step back. Many things are too complicated for us to parse without remotely approximating intelligence, like next month's weather.
Any idea who this idea traces back to? It would seem they were rather off the mark in their predictions.
Is it really surprising that as AI approaches our level of intelligence, the underlying interactions become more and more of a black box?
Well, Andrew Tanenbaum remembered when at IBM he received a full explanation of why they felt very important his shirt should not just be of some specific colour, but of the specific shade of some colour. I would not say it is not part of the job: I would say it ["we feel it very important"] is part of "what happens".
Attention, you peasants! Our overlords have decided for us that we may no longer say blackbox and whitebox. They have given us the following alternatives. You must all choose:
- Glassbox vs magicbox
- Openbox vs closedbox
- A global find/replace on the word "black", because that might be easier at this point
Trained neural networks of reasonable size have many thousands of independent variables and, as a result, have incredibly complex dynamics and behavior. Conversely, this explains why neural networks (of sometimes modest size) are able to approximately model behavior of extremely complex systems, linear or non-linear.
We can often understand the sensitivity of the NN to a specific variable in the NN, or the interplay of three variables together. Once we need to consider tens or hundreds of variables, we really can't fathom the dynamics and interplay of variables in the model any more... in some cases, iterative numerical sensitivity analysis can help. But even reasonably small NN's can have millions of variables and these numerical analysis methods become intractable to apply.
Remember the news of about 'single pixel' attacks against machine learning models that analyze and "categorize" images... researcher could change the category of an image by changing one pixel, the "right pixel", or hot pink or something. The researchers where in effect exploring a small sub-space of the NN and finding "regions of extreme dynamics" (highly weird behavior) in the model. There is no "rationale" for changing a cat to a dog based on one pixel going hot-pink in a 2M pixel... but there are extreme dynamics in model that allow that to happen.
I worked for a time on infrared detector development and "badly behaving" pixels (blinkers, etc.) were enemy #1 for IR computer vision because the wrong blinking pixel in a 4M+ pixel imaging array could cause all kinds of problems in the CV algorithms. It was okay for a pixel to be dead, or hot, or have low quantum efficiency... but pixels that blink, flicker, etc. periodically caused extremely erratic, unpredictable behavior of video analysis algorithms, which were largely NN-based, or used other high-degree of freedom modeling / decision-making methods.
While we can't predict this kind of weird behavior ahead of time from a specific NN model of reasonable size, we expect it. As briefly discussed above, we can't inspect complex NN models and figure out how and where they are going wrong. We'd have to perform an exhaustive, permutation-based search of the NN dynamic 'phase space' to determine what variable tweaks in the NN achieve the affects we are looking for... that search is, I think, a NP-hard problem (but I'm not sure).
There are a lot of other factors influencing the general "hardness" of modeling complex things with neural networks (data quality, etc.)... but this idea of "complex dynamics" in NN's is, I think, the fundamental problem that can be partially mitigated but not completely solved.
You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
>You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
Our learning is also for the most part Hebbian, from childhood through adulthood. For example, a DUI might force one to rethink their transport after a night-out. Right now, some of the "shocking" predictions have a child-like brutal honesty about them. Just as a child is coached not to call that bad aunty "fatty", an AI can thus be trained to conform to societal norms.
tl;dr: i am advocating for a realtime continuous learning system
The «continuous learning system» is either heuristic - an investigator - or an "artificial fool", of dubious utility (what is the use of something that "just has an opinion"). That poses an entity that reasons in a foundational context: why this and that.
If Kahneman - which I unfortunately have not yet had the time to read, though I was able to taste a bit of his interview by Lex Fridman - supposed that something "ineffable" oriented Einstein towards determinism and Bohr to more open stances, it is for their scientific work, valid on foundational grounds, that we remember them - not for their leanings.
--
¹Already a bit contradictory, if in school they teach you critical thought, as they did here since primary.
You have developed the capability of explaining your actions as resulting from your thoughts, because there is someone on the other side who is ready to accept and evaluate your explanations.
Others have no such affordance, so their brains get trained in prejudices and superstitions indead. Even if an explanation of an action is given in good faith and accepted by the recipient, it can still be factually incorrect.