Hacker News Reader

Comment by colllectorof | Hacker News Reader

Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical.

Also, people should have the right to know when machine-learned models are used to make decisions about their lives. They should be able to ask why a particular decisions was made and get that information.

This is real AI ethics.

View on HN

> Models that are used for anything important should be explainable. That is, you should be able to get a definitive answer as to why a particular result was achieved in any particular case. If a model does not have this property, it should not be used for anything critical.

What's the current standard for anything "important"? Most things important decisions are biased and not explained. Judges have biases and are allowed some discretion when sentencing. I'm sure police officers have biases as well. Same is true with just about any person making a judgement. There are laws (for good reason) that prevent certain types of biases, but it's naive to believe that the status quo on important decisions is great.

A model is better in a number of ways. First it is based on actual evidence. Although that can be manipulated, it's a lot easier to observe control than the life experiences of an individual. And you can guarantee that certain factors won't play a direct role at least by excluding them as inputs. It's much harder to tell a person to ignore some factors.

I think algorithmic, objective decision making on important decisions is very much preferable to what we have now.

I disagree. Having humans as deciders in key positions of society is a feature.

* having judges that live within a changing society is what allows laws to be changed in accordance to what the state of public debate is at a given time. For some it might be too slow or too fast, but it kind of works, if you think over the centuries.

* it is harder to manipulate all judges in the land than changing one algorithm or its goals. The distribution and fragmentation of power is a feature and not a bug. It makes some things weird and others slow, but ultimately it helps making a dictator take over hard.

Any new proposed system must be able to proof it has similar resistance against fundamental change from democracy into dictatorship, while still beeing able to slowly shift where it counts.

As a system the things we have are not bad and in western societies they managed quite well. I think any modern day proponent of alternative systems of governance should realize their hubris in the light of the things we already have working. The stuff we got is not just some service that could be run better, it is a way of preventing us from tearing ourselves and each others appart.

As desireable as algorithmic decision making sounds, it needs to be objective and deterministic. And it needs to be able to factor in circumstance in order to stay humane. But if it does so all the time it will be abused. In the end it would have to produce decisions that are accepted by humans after all.

I think for topics of high importance it would be best to have informed decisions by humans who educate themselves about the matter and bring forth arguments. For smaller decisions algorithms might work, as long as they are transparent

> * having judges that live within a changing society is what allows laws to be changed in accordance to what the state of public debate is at a given time

I agree that the system needs to have a feedback loop and the law should evolve over time. But IMO there are better ways to achieve that than introduce a biased decision maker. You can rely more greatly on legislation or bring in some evolving value system into your model.

> * it is harder to manipulate all judges in the land than changing one algorithm or its goals. The distribution and fragmentation of power is a feature and not a bug. It makes some things weird and others slow, but ultimately it helps making a dictator take over hard.

Fragmentation of power is generally a good thing. But the idea of having many judges as fragmentation of power is not actually fragmented. It would be fragmented if they all heard your case and sentenced you accordingly with some aggregation method. Or if they have a small jurisdiction. But in many cases, one judge (e.g. your sentencing judge), has all the power (ignoring appeals).

Let's assume I wanted to influence a sentencing judge. I could learn his biases. I could hire a lawyer with a good relationship with the judge. I could give him some financial incentive (e.g. kick backs from a private prison or campaign donations). I could manipulate jury selection using racial biases. I could look at the statistical properties of sentencing during different parts of the day and try to manipulate that. Less common in the US, but I could intimidate or threaten the judge.

How would that play out if it was an algorithm? I could hack into the administrating body and somehow retrain the algorithm? I could try to subvert some data scientist working on the algorithms and have them introduce slight biases into the algorithm that would eventually favor me?

I'd take my chances with the human judge. And that may be a good thing, but I don't think humans are harder to manipulate than algorithms.

> And it needs to be able to factor in circumstance in order to stay humane.

Again, I think it's a lot easier to tell an algorithm consider these new factors with the explicit goal of increase/decreasing X.

The word "objective" is problematic here. Normally people mean by this term "explainable by some quantitative criteria". In the case of AI this is technically true, but seems to lack substance. the workings of a neural networks for facial recognition could be decomposed to some god-awful switch statement involving all the pixels of the input picture (after all, this is exactly the structure of the resultant machine instructions), but how would one go about justifying any of the individual branches of such a structure? In other words, could you comment the code? I'm not 100% disagreeing with you, I just think it is problematic to call a statistical process "objective".

Modern facial recognition uses landmarks, much like humans do. First it identifies a face, which is essentially a flat 2-d plane. It's able to identify locations of the nose, ears, mouth, chin, etc. It then looks for relationships between these features (e.g. how wide is the mouth, how far apart are the eyes, etc).

It's a much simpler problem than the general image detection. Imagine how many different types of dogs you can encounter at various angles, sizes, colors, perspectives. For that it's much more of a black box as your model would have to support detecting a dog from behind, side, front, above, sitting down, walking, jumping, standing, dogs with long tails, short tails, long hair, short hair, etc. Now think about a face. Most faces are very much alike.

https://github.com/ageitgey/face_recognition

AI people have spent a lot of time trying to get systems to reason about the real world. These studies are lumped into the categories of common-sense knowledge and naive physics. Both have not got so far because it's proven hard to encode all the factors that go into real world decisions. Current AI models are a partial take on a part of the real world, this is why you get poor recognition rates if you feed photos taken from odd angles or with unexpected objects in them into deep learned systems.

So the algorithmic objective decisions of current systems are actually partial and selective. We must be very careful not to attribute powers to them that they do not have. They can provide useful tools, but they are not the locus of decision making; that rests in the place that they were created, and may be distorted by either accident or design.

You know what AI systems had the ability to explain their decisions? Expert systems.

Expert systems were the big success of GOFAI, but they fell out of favour in the last AI winter, at the end of the '90s or so, clearing the way for probabilistic inference and statistical machine learning.

Since then it seems, we took one step forward (with accuracy in classification) and one step back (with the loss of the ability to explain decisions).

Who knows, maybe a new AI winter will wipe out the statistical machine learning dinosaurs of today and leave a clear field of play for the AI Mammals of tomorrow.

I work at this company:

https://en.m.wikipedia.org/wiki/Cyc

It's a survivor of the AI winter, and what we have is basically a generalized expert system. Every conclusion incorporates cross-domain knowledge and can fully explain itself. It's been a long, slow trudge of research and development over the past couple decades, but we're starting to poke our heads out and ride the current AI hype wave. We like to say that Watson is our marketing department.

> We like to say that Watson > is our marketing department.

Did you forget a /s there? For me the marketing around Watson, with the unrealistic and not scientifically backed assumptions about the state of AI they put out to the general public, epitomize what's wrong with how the world sees our field.

More on topic: the underlying problem with the original post here seems to be one of selection bias in the training data. Somewhat of a remnant of human decision making that, likely unintentionally, ended up in creating biased decisions in the end. While the system you linked seems to potentially create nicely decomposable decisions, is there anything that inherently prevents "bias" in what it learns? The approach seems to face many of the same problems modern ML systems in the Linked Data / Semantic Web space would given unrestricted learning from the web.

>> Did you forget a /s there? For me the marketing around Watson, with the unrealistic and not scientifically backed assumptions about the state of AI they put out to the general public, epitomize what's wrong with how the world sees our field.

IBM's marketing Watson as a medical application is one thing. The system itself is quite another. And the system itself remains the most advanced NLP system created so far.

Note that I say "system". Most NLP research consists of testing the performance of various algorithms against very specific benchmarks, but very little work goes towards creating a unified system that can integrate multiple NLP abilities. Watson on the other hand is exactly such a unified system. It integrates statistical machine learning, symbolic reasoning (frames, fer chrissake, frames! In this day and age!), pattern-matching (with Prolog) and so on and so forth. Far as I can tell there isn't anything like it anywhere - but of course, I don't know much about what Google, Facebook et al are doing internally.

Sure, I specifically meant the marketing though. I kind of get why they do it but I'm not sold on the idea that it's helpful to present AI as an infallible silver bullet.

In terms of systems I'm not sure why IBM would stand out, Microsoft, Amazon and Google all offer relatively coherent pipelines for the engineering side of NLP and all of those are backed up by accomplished research teams. I'm sure you can find everything from total horror stories to beautiful testimonials for all of these platforms.

Unfortunately it's difficult to tell what Google etc are doing with their AI systems, but I get the feeling that Google and Facebook in particular have a culture of focusing on end-to-end statistical machine learning to the exclusion of everything else. I base this intuition on the way google translate is trained end-to-end and without any attempt to integrate grammatical or other background knowledge (which causes it to make mistakes that a rule-based system would easily avoid, e.g. getting the gender of nouns wrong when translating between two languages that have gendered nouns via English, that doesn't, etc).

I think these two companies in particular would be very unwilling to design and build a system like Watson, integrating symbolic techniques alongside statistical ones. They probably recruit so much for statistical machine learning skills that they don't have the know-how to do it anyway.

The funny thing is that, like many large corporations, they probably have ad-hoc expert systems except they don't call them that. Pretty much any sufficiently large and complex system that encodes "business rules" is essentially an expert system. But, because "expert systems failed" companies and engineers will not use the knowledge that came out of expert system research to make their systems better.

That AI winder really got us good.

The joke may not have translated the way I wrote it; IBM makes promises about Watson's intelligence that it can't live up to, and we've had clients in the past who've come to us because we can do (some of) what Watson advertised but couldn't deliver.

The main way our system would combat bias is by shedding light on it. No human-built system could be completely impartial, but ours will say "I decided X because of A, B, C, and D", and if people decide that C is biased then that piece of knowledge can be adjusted accordingly.

That makes more sense than what I read into it, apologies for that (and glad to hear that since I've had similar discussions). Thanks for clarifying.

On an intuitive level, this actually seems like a more promising approach than "from scratch" learning systems if we want our AIs to "understand us".

I know about Cyc, of course. Godspeed [and speed us away] :)

This seems like a fundamentally flawed and ultimately doomed approach to creating general agents. It's like you're trying to build a house, except only one molecule at a time.

The intent is to seed it by hand with enough knowledge that it will be able to do true NLU and learn new knowledge itself from the web. It's already shown some promise against these sorts of problems, which ML really struggles with:

https://en.m.wikipedia.org/wiki/Winograd_Schema_Challenge

That doesn’t really make any sense though. Maybe I’m not fully understanding what you’re saying here? Can you clarify how exactly you would turn a connected graph of data points into something that can learn arbitrary new knowledge and assimilate it into its understanding?

Because it seems like currently, this process is an entirely human driven manually done task wherein you have people making the decisions on how to connect and add new relations/knowledge to each other.

What you suggest, using previously acquired knowledge to learn new concepts, is par for the course in my field of study, Inductive Logic Programming. ILP is a set of algorithms and techniquest that learn logic programs from examples and background knowledge, where the examples and BK are themselves logic programs.

Initially the BK comes from some existing source- it can be a hand-crafted database of a few predicates deemed relevant to the learning task or a large, automatically-acquired database mined from some text source, data from the CYC Project of course, etc. In any case, because of the unified representation, learned hypotheses (the "models") can be used immediately as background knowledge to learn new concepts.

Edit: I don't know if the Cyc project uses ILP. But what the comment above says is doable.

In our current NLU experiments we use an off-the-shelf open source parser for the initial parse (Parsey McParseface, from Google). We then take the grammatically-tagged words and our system maps them to concepts it knows about. It theorizes about multiple possible interpretations of the sentence (pronouns in particular), judging their likelihood using real-world knowledge. A Winograd Schema can be solved because Cyc knows things about the concepts; it knows that a dog has four legs, and what it typically weighs, and that it's a mammal (and shares further traits of mammals), and knows that it couldn't interbreed with a cat because they're different species and the act of procreation requires two organisms of the same species. So far this information has been hand-encoded, but this has been going on for decades, and we have a lot now. Going past that:

1) We're able to map outside data from a DB into Cyc's knowledge format, rather than hand-encoding it. This knowledge is inherently not as rich as the rest, but it can obviously be useful anyway.

2) At some point we hope to reach a critical mass of knowledge that will allow Cyc to simply "import" a Wikipedia page by parsing and understanding it. It will interpret a given sentence into its own understanding, then assert it as true and do reasoning based on it down the line.

An expert system will never be able to process or understand new things unless given explicit solutions.

So you’re basically saying that it is possible to enumerate and solve for everything possible by manually iterating through every single possible edge case that exists. There’s a reason why Cyc has spent over 30 years and only has been able to get this far. You’re fundamentally limited by human constraints. The only realistic way of achieving a general purpose learning system is by teaching a system how to learn and then letting it figure things out on its own. Patently some method involving reinforcement learning.

By the way, it’s not like RL is some newfangled thing. Much of it started during the 80’s, as it concurrently developed with the other purported method of developing intelligence, which was through expert systems.

If you're interested, I highly recommend checking out a lecture that Demis Hassabis gives talking exactly about this issue: https://www.youtube.com/watch?v=3N9phq_yZP0

>> An expert system will never be able to process or understand new things unless given explicit solutions.

(Not the Cyc person).

Your comment is arguing for an end-to-end machine learning (specifically, reinforcement learning) approach. However, modern statistical machine learning systems have demonstrated very clearly that, while they are very good at learning specific and narrow tasks, they are pretty rubbish at multi-task learning and, of course, at reasoning. For breadth of capabilities they are no match to expert systems that can generally both tie their shoelaces and chew gum at the same time. Couple this with the practical limitations of learning all of intelligence end-to-end from examples and it's obvious that statistical machine learning on its own is not going to get any much farther than rule-based systems on their own.

Btw, reinforcement learning is rather older than the '80s. Donald Michie (my thesis advisor's thesis advisor) created MENACE, a reinforcement learning algorithm to play noughts-and-crosses in 1961 [1]. Machine learning in general is older still- Arthur Samuel baptised the field in the 1959 [2] but neural networks were first described in 1938 by Pitts and McCulloch [3]. Like Goeff Hinton has said, the current explosion of machine learning applications is due to large datasets and excesses of computing power- not because they're a new idea that people suddendly realised has potential.

_______________

[1] https://rodneybrooks.com/forai-machine-learning-explained/

[2] https://en.wikipedia.org/wiki/Arthur_Samuel

[3] https://en.wikipedia.org/wiki/Artificial_neuron

The future of generalized intelligence will not be based in brittle datasets, but purely based off repeated self-play, allowing for the bootstrapping of an infinite amount of possible data.

Rule based systems fail catastrophically the moment they encounter something that has not been codified into their rule set. I point to Chess as a prototypical example with two AI engines, StockFish and AlphaZero. StockFish, a manually created meticulously designed over the course of decades expert system, is handily defeated by AlphaZero, a reinforcement learning based system that trains purely through self-play.

If you look at any of the sample games between the two AIs, you can see a distinct difference in style between the two. In colloquial terms, StockFish acts far more “machine-like” whereas AlphaZero plays with a “human grace and beauty” according to many of the grandmasters that commented on its play. These of course are purely due to the fact that StockFish has certain inherent biases caused by brittleness from its codified rule set, which causes it to make sub-optimal moves in the long run, whereas AlphaZero is free from the constraints of any erroneously defined rules allowing it to do things like sacrifice it’s pieces as a strategy. Meanwhile, because StockFish codes in the value of losing a piece as giving negative points, it inherently has to overcome this bias every time it might choose to make a move in this manner, pushing its search space to find moves where it doesn’t have to be sacrificing pieces which is more optimal under its rule set.

Statistical machine learning models are just as brittle as hand-crafted rule-bases, when it comes to data they have not seen during training. They are incapable of genearlising outside their training set.

>> The future of generalized intelligence will not be based in brittle datasets, but purely based off repeated self-play, allowing for the bootstrapping of an infinite amount of possible data.

How will general intelligence arise through self-play, a technique used to train game-playing agents? There's never been a system that jumped from the game board to the real world.

OpenAI's GPT-2 language model (build using a deep neural network) recently achieved 70.7% accuracy on the Winograd Schema Challenge, which is 8% better than the previous record[1].

I've never used Cyc but I have used OpenCyc and I'm familiar with some of the applications of Cyc. It's interesting when it works.

[1] https://openai.com/blog/better-language-models/

Well, there are some statistical learning models that are explainable. Decision trees can be induced and can "explain themselves", just like in expert systems. Additive models are self-explanatory.

Not sure I'm sold on LIME and other similar approaches, though. Seems like a lot of deep learning people are all too happy to substitute "intepretability" for actual explanations.

Decision trees are not a statistical model. The learning method is statistical, if you will, but the model itself is a disjunction of conjunctions- a symbolic model (and more specifically, a propositional logic theory).

Decision Trees are in fact an example of the early years of machine learning where the trend was towards algorithms and techniques that learned symbolic theories. I believe the effort was driven by the realisation that expert systems had a certain problem with knowlege acquisition [1] which drove people to try and learn production rules from data.

I digress- I mean to say that decision trees are explainable because their models are not statistical.

To be honest, I don't know much about additive models.

_______________

[1] https://en.wikipedia.org/wiki/Knowledge_acquisition

Yup. That's what I learned first. Mycin being the first, success story I read about.

https://en.wikipedia.org/wiki/Mycin

I tried some quick DuckDuckGo-ing to see if anything new turned up. Drowned out by unrelated stuff. I did find what looks like a great, quick overview of expert systems for folks unfamiliar with them. Might be new, default link I share about how they historically were perceived. What you think of this one?

https://www.tutorialspoint.com/artificial_intelligence/artif...

The tutorial is a bit old.

I think that advances in probablistic reasoning & modelling, such as practical Bayes networks should be included, and the mechanics of resolution have improved massively with the introduction of answerset systems - this gets over the problem of commitment that kiboshed gen 5.

https://en.wikipedia.org/wiki/Answer_set_programming

https://en.wikipedia.org/wiki/Probabilistic_programming_lang...

Can you explain how you, as a human being, understand handwriting? Can you explain it in detail without handwaving some stuff away as "now I detect the pattern"? Because I'm pretty sure you can't, but you use it for basically everything in life. So, should a person's eyes not be trusted when it comes to "something important"?

>So, should a person's eyes not be trusted when it comes to "something important"?

A documentary called "Murder on a Sunday Morning": https://www.youtube.com/watch?v=LFLbptkb1eM

A woman was murdered in front of her husband. The man saw the attacker up close and was the only eyewitness.

He accused a completely innocent man, and it took half a year of jail and court-related turmoils for this to get cleared up.

>Can you explain how you, as a human being, understand handwriting?

We explain how to understand handwriting to every single child that goes to school. It's not the answer you want, but it's the answer that actually matters here.

Trying to equivocate AI and human cognition in this way is completely disingenuous.

Human reasoning is not 100% reliable, but we know very well in which ways it's unreliable and how to deal with it. We have shared biology and millennia of experience trying to empathize and communicate with others.

We do not explain how to read handwriting, we teach by example - and that is exactly how ML works.

And your last point is wrong. ML models are studied and understood much better than human reasoning.

>We do not explain how to read handwriting, we teach by example - and that is exactly how ML works.

Teaching children to read is an interactive process that has pretty much nothing in common with data steamrolling in modern machine learning.

>And your last point is wrong. ML models are studied and understood much better than human reasoning.

Is that why new ANN architectures are almost universally constructed by trial and error?

But if you were to present handwriting as evidence in court, if the veracity of the evidence were in question, you would need to present an expert whose job it is to explain the methodology behind the analysis.

Given that eyewitness testimony is consistently unreliable, biased, and internally inconsistent, no, they shouldn’t. Human judgement is the best we can do so far, but it is hardly a worthwhile goal, especially with systems that take us from affecting individuals on a person-to-person level to influencing the interactions of millions of people with their government.

Algorithms can be explainable, and I agree that anything that affects your standing in the eyes of government should have that as a minimum requirement.

The best work on this was done by Brendan Lake and Josh Tannenbaum. https://www.sas.upenn.edu/~astocker/lab/teaching-files/PSYC7...

There's a recent review that they published.

https://arxiv.org/pdf/1902.03477.pdf

> Can you explain how you, as a human being, understand handwriting?

I do not want to speak for the parent but I think you might be on a different level of abstraction when talking about "explaining" things. Consider this for example: someone writes a letter to your boss that you should be fired. That someone does not need to explain the process of writing but probably should be required to explain why you should be fired.

This is such an interesting topic and I really don't know where to stand on it. On one hand, this is an argument that systems such as laws, lending practices, etc. should be based on repeatable and explainable rules. Rules will end up with some bias, but the bias can be dissected and argued about until we reach some consensus that the bias is acceptable/desiresble. In the case of lending, maybe we consider credit history a fair input metric but zip-code an unfair metric, for example.

On the other hand, systems such as laws and lending, which are inherently about social interaction, do tend to have some unspecified human element. Judges interpret the law and apply it to specific cases, underwriters have latitude to make exceptions under certain vague circumstances, teachers may regrade a paper upon realizing they misspoke in a lecture. This is a feature, not a bug--if our social systems have no room for empathy then there is a big problem.

So now that AI is "unexplainable", how is this worse than the unexplainability of human systems? You can ask a human why they took some action, but their explanation tends to be incomplete, wrong, a lie, or maybe they don't recall. I once was given the opportunity to lease a top-floor apartment in my building because I had a daily chat with the receptionist. All fair housing laws were followed, I just happened to be the first to know because I had such frequent interaction. If you were to ask the receptionist why I got the apartment she probably wouldn't say "zwkrt tells me bad jokes and we share pictures of our pets", but that probably is part of the reason.

I think fundamentally we understand that humans share a way of life and a set of ineffable values, and we believe that computer-generated results do not share these values. Unfortunately I do not have a conclusion, just a formulation of a problem.

The present systems have more attributes than “they somehow work, but slow”. Think about separation of power and decentralization. Slowness is a feature here. If you want to radically change a society where decisions are carried out by any number of individual actors, you need to take them with you by either using military force, money or convincing them otherwise. In any case this is a project that could also fail. With an algorithm, you could change it today and it would be in action without many even noticing it.

I could get behind a simple and transparent tax system, where you just see in realtime what money you give while beeing sure that big company has to do the same, without gaming the system.

But I am not sure the system that decides on these rules should be another system.

The dispute here is about gender classification. If you can give a consistent explanation of how your own visual system works for this task, then perhaps one day we’ll be able to program a computer to do the same.

I can't, but human visual perception can be a subject to introspection to a high degree, and can be top-down overriden. That is, I can name at least some of the reasons why the person I'm looking at looks male/female to me, and in cases that are not obvious, I can step what I see through high-level, explicit reasoning, and the result actually informs my perception.

Also, to the benefit of my perception works the fact that human beings have shared brain architecture, and the way I see stuff is the same as everybody else, so whatever half-assed explanation I could give, it's intimately understood by other people. In contrast, ML models are completely alien to us.

The problem is that most machine learning model choices are humanly indecipherable.

You can understand how the model works and the math behind it but you will be hard pressed to understand the exact path behind a particular choice.

Decision trees are one exception to this. Most humans can understand those.

Do you put the same requirement on natural neural nets?

Is the decision reached by human members of a jury, for example, explainable in the way you mean?

Reply on news.ycombinator.com