Doug Lenat has died
garymarcus.substack.com
garymarcus.substack.com
I worked at Cycorp (not directly with Doug very often, but it wasn't a big office) between 2016 and 2020
An anecdote: during our weekly all-hands lunch in the big conference room, he mentioned he was getting a new car (his old one was pretty old, but well-kept) and he asked if anybody could use the old car. One of the staff raised his hand sheepishly and said his daughter was about to start driving. Doug gifted him the car on the spot, without a second thought.
He also loved board games, and was in a D&D group with some others at the company. I was told he only ever played lawful good characters, he didn't know how to do otherwise :)
Happy to answer what questions I can
Emphasis on needlessly. Some bluntness is a good thing, even if it makes people uncomfortable. The alternative usually is prolonged awkwardness and beating around the bush, which likely will make people uncomfortable anyways. Just get it over with like an adult. Don't drag an uncomfortable thing out or even risk people not getting the message at all the first time around.
The socially acceptable thing to do is wait for them to catch up but that gets old.
A article shared on HN recently: https://www.bihealth.org/en/notices/intelligent-brains-take-....
Don't get me wrong other intelligent people are on board with me during these episodes, and they are sometimes able to present their specific concern with the solution and that is valuable. I've been wrong before of course. On the other hand the "think of all the complexities" crowd just spout nuggets of wisdom like that until they are able to comprehend the solution, and that takes about 15 minutes in my experience.
That internship was life-changing for me, and I'll always be grateful to him for taking a wild bet on a literally a kid.
Doug was a brilliant computer scientist, and a pioneer of artificial intelligence. Though I was very junior at Cycorp, it was a small company so I sat in many meetings with him. It was obvious that he understood every detail of how the technology worked, and was extremely smart.
Cycorp was 30 years ahead of its time and never actually worked. For those who don't know, it was essentially the first OpenAI - the first large-scale commercial effort to create general artificial intelligence.
I learned a lot from Doug about how to be incredibly ambitious, and how to not give up. Doug worked on Cycorp for multiple decades. It never really took off, but he managed to keep funding it and keep hiring great people so he could keep plugging away at the problem. I know very few people who have stuck with an idea for so long.
Also - what exactly did you do in the internship as a 17 year old - what skills did you have?
My first summer, I was an ontologist, which was a unique role that only existed at Cycorp where they hired people to literally hand-enter facts like "A cat has four legs" into Cyc using formal logic. My second summer I programmed (poorly) in Lisp for them.
Could you say more about that? How could you tell?
1) Cyc's reasoning fundamentally did not feel "human". Cyc was created on the premise that you could build AGI on top of formal logic inference. But after seeing how Cyc performed on real-world problems, I became convinced that formal logic is a poor model for human thought.
The biggest tell is that formal logic systems are very brittle. If there is any fact that is even slightly off, the reasoning chain fails and the system can't do anything. Humans aren't like that; when their information is slightly off, their performance degrades gracefully.
2). Imagine a graph where time/money was on the x-axis, and Cyc's performance was on the y-axis. You could roughly plot this using benchmarks like SAT scores. It was clear if you extrapolated this that Cyc was never going to hit human-level performance; the curve was going to asymptotically approach something well below human-level performance.
As a side note, if you look at the performance of LLMs, I would argue that you get the opposite result for both criteria.
It was a fun time with a small core team (mainly me, guha, and Doug) but over time I became dissatisfied with some of the arbitrariness of the KB. By the time I left the Cyc project (for my own reasons unrelated to work) I was somewhat negative towards the foundations of the project, despite the tight relationship we’d had and the fact it ran on my code! But over time I became smarter and came to appreciate once again its value. I had too much of a “pure math” view of things back then.
As I moved on to other things I lost touch with Doug and Mary, and I’m sorry for that.
One day, the rapid advancement of AI via LLMs will slow down and attention will again return to logical reasoning and knowledge representation as championed by the Cyc Project, Cycorp, its cyclists and Dr. Doug Lenat.
Why? If NN inference were so fast, we would compile C programs with it instead of using deductive logical inference that is executed efficiently by the compiler.
This is the definition of a strawman. Who is claiming that NN inference is always the fastest way to run computation?
Instead of trying to bring down another technology (neural networks), how about you focus on making symbolic methods usable to solve real-world problems; e.g. how can I build a robust email spam detection system with symbolic methods?
Perhaps LLMs can automate in large part the manual operations of building a future symbolic knowledge base organized by a universal upper ontology. Considering the amazing emergent features of sufficiently-large LLMs, what could emerge from a sufficiently large, reflective symbolic knowledge base?
I’m making a sloppy version that will contain all the symbols needed to run a multi-unit building.
> This is the definition of a strawman.
(Actually, it is an example of a strawman.) Anyhow, rather than a strawman, I'd rather us get right into the fundamentals.
1. Feed-forward NN computation ('inference', which is an unfortunate word choice IMO) can provably provide universal function approximation under known conditions. And it can do so efficiently as well, with a lot of recent research getting into both the how and why. One "pays the cost" up-front with training in order to get fast prediction-time performance. The tradeoff is often worth it.
2. Function approximation is not as powerful as Turing completeness. FF NNs are not Turing complete.
3. Deductive chaining is a well-studied, well understood area of algorithms.
4. But... modeling of computational architectures (including processors, caches, busses, and RAM) with sufficient detail to optimize compilation is a hard problem. I wouldn't be surprised if this stretches these algorithms to the limit in terms of what developers will tolerate in terms of compile times. This is a strong incentive, so I'd expect there is at least some research that pushes outside the usual contours here.
I have two concerns. First, just after pointing out a logical fallacy from someone else, you added a fallacy: the either-or fallacy. (One can criticize a technology and do other things too.)
Second, you selected an example that illustrates a known and predictable weakness of symbolic systems. Still, there are plenty of real-world problems that symbolic systems address well. So your comment cherry-picks.
It appears as if you are trying to land a counter punch here. I'm weary of this kind of conversational pattern. Many of us know that tends to escalate. I don't want HN to go that direction. We all have varying experience and points of view to contribute. Let's try to be charitable, clear, and logical.
Well done, sir, well done.
What's the point of all that data collecting dust and accomplishing not much of anything?
This is hugely problematic. If you get the premises wrong, many fallacies will follow.
LLMs can play many roles around this area, but their output cannot be trusted with significant verification and validation.
That's one of the principal features of Cyc. It's carefully built by humans to be (essentially) logically sound. - so that inference can then be run through the fact base. Making that stuff logically sound made for a very detailed and fussy knowledge base. And that in turn made it difficult to expand or even understand for mere civilians. Cyc is NOT simple.
By contrast, LLMs for now are embarassing. With inconsistent nonsense provided within one answer or an answer not recognizing the context of the problem. Say, the work domain being a food label and the system not recognizing that or not staying within that.
This is unpersuasive without laying out your assumptions and reasoning.
Counter points:
(a) It would be unethical for such a knowledge base to be put out in the open without considerable guardrails and appropriate licensing. The details matter.
(b) Cycorp gets some funding from the U.S. Government; this changes both the set of options available and the calculus of weighing them.
(c) Not all nations have equivalent values. Unless one is a moral relativist, these differences should not be deemed equivalent nor irrelevant. As such, despite the flaws of U.S. values and some horrific decision-making throughout history, there are known worse actors and states. Such parties would make worse use of an extensive human-curated knowledge base.
I'd recommend that more people take a look and compare its approach against others. https://en.wikipedia.org/wiki/CycL is compact and worth a read, especially the concept of "microtheories".
Neural and symbolic AI will eventually merge. Symbolic models bring much needed efficiency and robustness via regularization.
If you read PAIP today, the most likely reason is that you want a master class in Lisp programming and/or want to learn a lot of tricks for getting good performance out of complex programs (which used to be part of AI and is in many ways being outsourced to hardware today).
None of this is to say you shouldn't read PAIP. You absolutely should. It's awesome. But its role is different now.
Other parts like coding an Eliza chatbot are indeed outdated. I have read AIMA and followed a long course that used it, but I didn't really like it. I found it too broad and shallow.
My hunch is it emerges naturally out of the hierarchical generalization capabilities of multiple layer circuits. But then you need something to coordinate the acquired labels: a tweak on attention perhaps?
Another characteristic is probably some (limited) form of recursion, so the generalized labels emitted at the small end can be fed back in as tokens to be further processed at the big end.
https://www.youtube.com/watch?v=3wMKoSRbGVs&pp=ygUabGV4IGZya...
The best episodes are ones where the guest drives the interview and has a lot of interesting things to say. Fridman's just useful for attracting interesting domain experts somewhere we can hear them speak for hours on end.
The Jim Keller episodes are excellent IMO, despite Fridman. Guests like Keller and Carmack don't need a good interviewer for it to be a worthwhile listen.
(1) generic (e.g. swerves the thread toward larger/general topic rather than something more specific);
(2) flamey (e.g. provocative on a divisive issue); and
(3) predictable (e.g. has been hashed so many times already that comments will likely fall in a few already-tiresome hash buckets)
- is the bad kind of offtopicness: the kind that brings little new information and eventually lots of nastiness. We're trying for the opposite here—lots of information and little nastiness.
He said they learned after ~5 years that this was an order of magnitude off -- it's more like 10 M things.
Is there any literature about this? Did they publish?
To me, the obvious questions are -
- how do they know it's not 100M things?
- how do they know it's even bounded? Why isn't there a combinatorial explosion?
I mean I guess they were evaluating the system all along. You don't go for 38 years without having some clear metrics. But I am having some problems with the logic -- I'd be interested in links to references / criticism.
I'd be interested in any arguments for and against ~10 M. Naively speaking, the argument seems a bit flawed to me.
FWIW I heard of Cyc back in the 90's, but I had no idea it was still alive. It is impressive that he kept it alive for so long.
---
Actually the wikipedia article is pretty good
https://en.wikipedia.org/wiki/Cyc#Criticisms
Though I'm still interested in the ~1M or ~10M claim. It seems like a strong claim to hold onto for decades, unless they had really strong metrics backing it up.
> how do they know it's even bounded? Why isn't there a combinatorial explosion?
I don't know - I'm in middle of watching the interview too, but he's moved on from that topic already. I'd guess the 10M vs 1M (or 100M) estimate comes from the curve of total "assertions" vs time leveling off towards some asymptotic limit.
I suppose the reason there's no combinatorial explosion is because they're entering these assertions in most general form possible, so considering new objects doesn't necessarily mean new assertions since it may all be covered by the superclasses the objects are part of (e.g. few assertions that are specific to apples since most will apply to all fruit).
In other words, I think it would be so highly useful to have a browseable corpus of arguments and conclusions, where people could collaborate on them and perhaps disagree with portions of the argument graph, adding to it and enriching it over time, so other people could read and perhaps adopt the same reasoning.
I play around with ideas with this site I occasionally work on, http://concludia.org/ - really more an excuse at this point to mess around with the concept and also get better at Akka (Pekko) programming. At some point I'll add user accounts and editable arguments and make it a real website.
You can see some of the explanation at http://concludia.org/instructions .
Intuitionist or classical?
I thought of it while daydreaming about how to converge public opinion in a nation with major political polarization. It'd be a sort of structured public debate forum and people could better see exactly where in the hierarchy they disagreed and, perhaps more importantly, how much they in fact agreed upon.
Would be great for brainstorming.
In any case, at the time Lenat starting working on Cyc, we didn't really have the compute required to do NN models at the level where they start exhibiting what most would call "common sense reasoning," so it makes total sense why he started out on that path. RIP.
"Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc"
Lenat's last paper (July 31st, with Gary Marcus)
https://news.ycombinator.com/item?id=37354601
This may disabuse you of two ideas:
1. that NN models (LLMs) exhibit common sense reasoning today
2. that the approach to AI represented by Cyc and the one represented by LLMs are mutually exclusiveResponse: There is no widely available historical information to suggest that Abraham Lincoln's maternal grandmother had an unusual number of thumbs. It would be reasonable to guess that she had the typical two thumbs, one on each hand, unless stated otherwise.
So it is essential to clarify architecture when making claims about capabilities.
I'll start simple: Plain sequence to sequence feed-forward NN models are not Turing complete. Therefore they cannot do full reasoning, because that requires arbitrary chaining.
The mildly entertaining YouTube video below discusses this. https://youtu.be/QrSCwxrLrRc
[1]
"Over the last year, Doug and I tried to write a long, complex paper that we never got to finish. Cyc was both awesome in its scope, and unwieldy in its implementation. The biggest problem with Cyc from an academic perspective is that it’s proprietary.
To help more people understand it, I tried to bring out of him what lessons he learned from Cyc, for a future generation of researchers to use. Why did it work as well as it did when it did, why did fail when it did, what was hard to implement, and what did he wish that he had done differently? ...
...One of his last emails to me, about six weeks ago, was an entreaty to get the paper out ASAP; on July 31, after a nerve-wracking false-start, it came out, on arXiv, Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc (https://arxiv.org/ftp/arxiv/papers/2308/2308.04445.pdf).
The brief article is simultaneously a review of what Cyc tried to do, an encapsulation of what we should expect from genuine artificial intelligence, and a call for reconciliation between the deep symbolic tradition that he worked in with modern Large Language Models."
> my content would probably just add noise to the internet
Maybe, but there is worse noise out there for sure. :) Anyhow, some unsolicited advice from me: don't replay this quote to yourself any more than necessary; it isn't exactly a motivational mantra masterpiece. Share what you think is important.
Why? Even small, "improbable" improvements to knowledge can to matter. Given enough of them, statistically speaking, we can move the needle. Yeah, and we need to be able to find the relevant stuff; a big problem in of itself.
I feel it's appropriate to link a blog post of mine from 2018. It's a quick recap of Lenat works on the trajectory that brought him towards Cyc, with links to the papers.
http://blog.funcall.org//lisp/2018/11/03/am-eurisko-lenat-do...
Wikipedia's overview: <https://en.wikipedia.org/wiki/Cyc>
Project / company homepage: <https://cyc.com/>
It's failure is no shade against Doug. Somebody had to try it, and I'm glad it was one of the brightest guys around. I think he clung on to it long after it was clear that it wasn't going to work out, but breakthroughs do happen. (The current round of machine learning itself is a revival of a technique that had been abandoned, but people who stuck with it anyway discovered the tricks that made it go.)
I believe both approaches are useful and can be combined and layered and fed back into each other, to reinforce and transcend complement each others advantages and limitations.
Kind of like how Hailey and Justin Bieber make the perfect couple: ;)
https://edition.cnn.com/style/hailey-justin-bieber-couples-f...
Marvin L Minsky: Logical Versus Analogical or Symbolic Versus Connectionist or Neat Versus Scruffy
https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...
https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...
"We should take our cue from biology rather than physics..." -Marvin Minsky
>To get around these limitations, we must develop systems that combine the expressiveness and procedural versatility of symbolic systems with the fuzziness and adaptiveness of connectionist representations. Why has there been so little work on synthesizing these techniques? I suspect that it is because both of these AI communities suffer from a common cultural-philosophical disposition: They would like to explain intelligence in the image of what was successful in physics—by minimizing the amount and variety of its assumptions. But this seems to be a wrong ideal. We should take our cue from biology rather than physics because what we call thinking does not directly emerge from a few fundamental principles of wave-function symmetry and exclusion rules. Mental activities are not the sort of unitary or elementary phenomenon that can be described by a few mathematical operations on logical axioms. Instead, the functions performed by the brain are the products of the work of thousands of different, specialized subsystems, the intricate product of hundreds of millions of years of biological evolution. We cannot hope to understand such an organization by emulating the techniques of those particle physicists who search for the simplest possible unifying conceptions. Constructing a mind is simply a different kind of problem—how to synthesize organizational systems that can support a large enough diversity of different schemes yet enable them to work together to exploit one another’s abilities.
https://en.wikipedia.org/wiki/Neats_and_scruffies
>In the history of artificial intelligence, neat and scruffy are two contrasting approaches to artificial intelligence (AI) research. The distinction was made in the 70s and was a subject of discussion until the middle 80s.[1][2][3]
>"Neats" use algorithms based on a single formal paradigms, such as logic, mathematical optimization or neural networks. Neats verify their programs are correct with theorems and mathematical rigor. Neat researchers and analysts tend to express the hope that this single formal paradigm can be extended and improved to achieve general intelligence and superintelligence.
>"Scruffies" use any number of different algorithms and methods to achieve intelligent behavior. Scruffies rely on incremental testing to verify their programs and scruffy programming requires large amounts of hand coding or knowledge engineering. Scruffies have argued that general intelligence can only be implemented by solving a large number of essentially unrelated problems, and that there is no magic bullet that will allow programs to develop general intelligence autonomously.
>John Brockman compares the neat approach to physics, in that it uses simple mathematical models as its foundation. The scruffy approach is more like biology, where much of the work involves studying and categorizing diverse phenomena.[a]
[...]
>Modern AI as both neat and scruffy
>New statistical and mathematical approaches to AI were developed in the 1990s, using highly developed formalisms such as mathematical optimization and neural networks. Pamela McCorduck wrote that "As I write, AI enjoys a Neat hegemony, people who believe that machine intelligence, at least, is best expressed in logical, even mathematical terms."[6] This general trend towards more formal methods in AI was described as "the victory of the neats" by Peter Norvig and Stuart Russell in 2003.[18]
>However, by 2021, Russell and Norvig had changed their minds.[19] Deep learning networks and machine learning in general require extensive fine tuning -- they must be iteratively tested until they begin to show the desired behavior. This is a scruffy methodology.
Cyc is sort of like that, but for everything. Not just a small limited world. I believe it didn’t work out because it’s really hard.
So you'd use the NN to recognize that the thing in front of the camera is a cat, and that would be fed into the symbolic knowledge base for further reasoning.
The knowledge base will contain facts like the cat is likely to "meow" at some point, especially if it wants attention. Based on the relevant context, the knowledge base would also know that the cat is unlikely to be able to talk, unless it is a cat in a work of fiction, for example.
Leela AI was founded by Henry Minsky and Cyrus Shaoul, and is inspired by ideas about child development by Jean Piaget, Seymour Papert, Marvin Minsky, and Gary Drescher (described in his book “Made-Up Minds”).
https://mitpress.mit.edu/9780262517089/made-up-minds/
>Leela Platform is powered by Leela Core, an innovative AI engine based on research at the MIT Artificial Intelligence Lab. With its dynamic combination of traditional neural networks for pattern recognition and causal-symbolic networks for self-discovery, Leela Core goes beyond accurately recognizing objects to comprehend processes, concepts, and causal connections.
>Leela Core is much faster to train than conventional NNs, using 100x less data and enabling 10x less time-to-value. This highly resilient AI can quickly adjust to changes and explain what it is sensing and doing via the Leela Viewer dashboard. [...]
The key to regulating AI is explainability. The key to explainability may be causal AI.
https://leela.ai/post/the-key-to-regulating-ai-is-explainabi...
>[...] For example, the Leela Core engine that drives the Leela Platform for visual intelligence in manufacturing adds a symbolic causal agent that can reason about the world in a way that is more familiar to the human mind than neural networks. The causal layer can cross-check Leela Core's traditional NN components in a hybrid causal/neural architecture. Leela Core is already better at explaining its decisions than NN-only platforms, making it easier to troubleshoot and customize. Much greater transparency is expected in future versions. [...]
The goal was that in a decade it would become self-sustaining. It would have enough knowledge that it could start reading natural language. And it just... didn't.
Contrast it with LLMs and diffusion and such. They make stupid, asinine mistakes -- real howlers, because they don't understand anything at all about the world. If it could draw, Cyc would never draw a human with 7 fingers on each hand, because it knows that most humans have 5. (It had a decent-ish ontology of human anatomy which could handle injuries and birth defects, but would default reason over the normal case.) I often see ChatGPT stumped by simple variations of brain teasers, and Cyc wouldn't make those mistakes -- once you'd translated them into CycL (its language, because it couldn't read natural language in any meaningful way).
But those same models do a scary job of passing the Turing Test. Nobody would ever have thought to try it on Cyc. It was never anywhere close.
Philosophically I can't say why Cyc never developed "magic" and LLMs (seemingly) do. And I'm still not convinced that they're on the right path, though they actually have some legitimate usages right now. I tried to find uses for Cyc in exactly the opposite direction, guaranteeing data quality, but it turned out nobody really wanted that.
Cyc, on the other hand, lacks flesh and skin. It's all skeleton and can generate facts but not embellish them into narratives.
The best human writing has both, much as artists (traditional painters, sculptors, and more recently computer animators) has a skeleton (outline, index cards, Zettlekasten, wireframe) to which flesh, skin, and fur are attached. LLM generative AIs are too plastic, Cyc is insufficiently plastic.
I suspect there's some sort of a middle path between the two. Though that path and its destination also increasingly terrify me.
LLMs understand plenty, in any way that can be tested. It's really funny when i see making mistakes as the evidence of lack of understanding. Guess people don't understand anything at all too.
> I often see ChatGPT stumped by simple variations of brain teasers
Only if everything else is exactly as the basic teaser and guess what ? humans fall for this too. They see something they memorized and go full speed ahead. Simply changing names is enough to get it to solve it.
May essentialism just does not work.
I do suspect that well-curated and hand-tuned corpora, including possibly Cyc's, are of significant use to LLM AI. And will likely be more so as the feedback / autophagy problem exacerbates.
Natural-language content-based classification as by Google and Web text-based search relies effectively on documents self-descriptions (that is, their content itself) to classify and search works, though a ranking scheme (e.g., PageRank) is typically layered on top of that. What distinguished early Google from prior full-text search was that the latter had no ranking criteria, leading to keyword stuffing. An alternative approach was Yahoo, originally Yet Another Hierarchical Officious Oracle, which was a curated and ontological classification of websites. This was already proving infeasible by 1997/98 as a whole, though as training data for machine classification might prove useful.
I suppose you'd architect it as a layer. It wants to say something, and the ontology layer says, "No, that's stupid, say something else". The ontology layer can recognize ontology-like statements and use them to build and evolve the ontology.
It would be even more interesting built into the visual/image models.
I have no idea if that's any kind of real progress, or if it's merely filtering out the dumb stuff. A good service, to be sure, but still not "AGI", whatever the hell that turns out to be.
Unless it turns out to be the missing element that puts it over the top. If I had any idea I wouldn't have been working with Cyc in the first place.
is precisely Doug Lenat & Gary Marcus' thoughts on how to combine them (July 31st 2023, Lenat's last paper)
His question is: Is it preferable for scruffies to become neater, or for neats to become scruffier? His answer explains why he aspires to be a neater scruffy.
"But I use the example as symptomatic of one kind of approach to the cognitive science fusion problem: you start from a neat, right-wing point of view, but acknowledge some limited role for scruffy, left-wing orientations. The other type of approach is the obvious mirror: you start from the disorderly leftwing side and struggle to be neater about what you are doing. I prefer the latter approach to the former. I will tell you why, and then lay out the beginnings of such an approach."
https://cse.buffalo.edu/~rapaport/676/F01/neat.scruffy.txt
Article: 35781 of comp.ai
From: fass@cs.sfu.ca (Dan Fass)
Newsgroups: comp.ai
Subject: Re: who first used "scruffy" and "neat"?
Date: 26 Jan 1996 10:03:35 -0800
Organization: Simon Fraser University, Burnaby, B.C.
Abelson (1981) credits the neat/scruffy distinction to Roger Schank.
Abelson says, ``an unnamed but easily guessable colleague of mine
... claims that the major clashes in human affairs are between the
"neats" and the "scruffies". The primary concern of the neat is
that things should be orderly and predictable while the scruffy
seeks the rough-and-tumble of life as it comes'' (p. 1).
Abelson (1981) argues that these two prototypic identities --- neat
and scruffy --- ``cause a very serious clash'' in cognitive science
and explores ``some areas in which a fusion of identities seems
possible'' (p. 1).
- Dan Fass
REF
Abelson, Robert P. (1981).
Constraint, Construal, and Cognitive Science.
Proceedings of the 3rd Annual Conference of the Cognitive Science
Society, Berkeley, CA, pp. 1-9.
[I'll quote the most relevant first part of the article, which is still worth reading in its entirety if you have time, since scanned two column pdf files are so hard to read on mobile, and it's so interesting and relevant to Douglas Lenat's work on Cyc.]
CONSTRAINT, CONSTRUAL, AND COGNITIVE SCIENCE
Robert P. Abelson, Yale University
Cognitive science has barely emerged as a discipline -- or an interdiscipline, or whatever it is -- and already it is having an identity crisis.
Within us and among us we have many competing identities. Two particular prototypic identities cause a very serious clash, and I would like to explicate this conflict and then explore some areas in which a fusion of identities seems possible. Consider the two-word name "cognitive science". It represents a hybridization of two different impulses. On the one hand, we want to study human and artificial cognition, the structure of mental representatives, the nature of mind. On the other hand, we want to be scientific, be principled, be exact. These two impulses are not necessarily incompatible, but given free rein they can develop what seems to be a diametric opposition.
The study of the knowledge in a mental system tends toward both naturalism and phenomenology. The mind needs to represent what is out there in the real world, and it needs to manipulate it for particular purposes. But the world is messy, and purposes are manifold. Models of mind, therefore, can become garrulous and intractable as they become more and more realistic. If one's emphasis is on science more than on cognition, however, the canons of hard science dictate a strategy of the isolation of idealized subsystems which can be modeled with elegant productive formalisms. Clarity and precision are highly prized, even at the expense of common sense realism. To caricature this tendency with a phrase from John Tukey (1959), the motto of the narrow hard scientist is, "Be exactly wrong, rather than approximately right".
The one tendency points inside the mind, to see what might be there. The other points outside the mind, to some formal system which can be logically manipulated (Kintsch et al., 1981). Neither camp grants the other a legitimate claim on cognitive science. One side says, "What you're doing may seem to be science, but it's got nothing to do with cognition." The other side says, "What you're doing may seem to be about cognition, but it's got nothing to do with science."
Superficially, it may seem that the trouble arises primarily because of the two-headed name cognitive science. I well remember the discussions of possible names, even though I never liked "cognitive science", the alternatives were worse; abominations like "epistology" or "representonomy".
But in any case, the conflict goes far deeper than the name itself. Indeed, the stylistic division is the same polarization than arises in all fields of science, as well as in art, in politics, in religion, in child rearing -- and in all spheres of human endeavor. Psychologist Silvan Tomkins (1965) characterizes this overriding conflict as that between characterologically left-wing and right-wing world views. The left-wing personality finds the sources of value and truth to lie within individuals, whose reactions to the world define what is important. The right-wing personality asserts that all human behavior is to be understood and judged according to rules or norms which exist independent of human reaction. A similar distinction has been made by an unnamed but easily guessable colleague of mine, who claims that the major clashes in human affairs are between the "neats" and the "scruffies". The primary concern of the neat is that things should be orderly and predictable while the scruffy seeks the rough-and-tumble of life as it comes.
I am exaggerating slightly, but only slightly, in saying that the major disagreements within cognitive science are instantiations of a ubiquitous division between neat right-wing analysis and scruffy left-wing ideation. In truth there are some signs of an attempt to fuse or to compromise these two tendencies. Indeed, one could view the success of cognitive science as primarily dependent not upon the cooperation of linguistics, AI, psychology, etc., but rather, upon the union of clashing world views about the fundamental nature of mentation. Hopefully, we can be open minded and realistic about the important contents of thought at the same time we are principled, even elegant, in our characterizations of the forms of thought.
The fusion task is not easy. It is hard to neaten up a scruffy or scruffy up a neat. It is difficult to formalize aspects of human thought which are variable, disorderly, and seemingly irrational, or to build tightly principled models of realistic language processing in messy natural domains. Writings about cognitive science are beginning to show a recognition of the need for world-view unification, but the signs of strain are clear. Consider the following passage from a recent article by Frank Keil (1981) in Pscyhological Review, giving background for a discussion of his formalistic analysis of the concept of constraint:
"Constraints will be defined...as formal restrictions that limit the class of logically possible knowledge structures that can normally be used in a given cognitive domain." (p. 198).
Now, what is the word "normally" doing in a statement about logical possibility? Does it mean that something which is logically impossible can be used if conditions are not normal? This seems to require a cognitive hyperspace where the impossible is possible.
It is not my intention to disparage an author on the basis of a single statement infelicitously put. I think he was genuinely trying to come to grips with the reality that there is some boundary somewhere to the penetration of his formal constraint analysis into the viscissitudes of human affairs. But I use the example as symptomatic of one kind of approach to the cognitive science fusion problem: you start from a neat, right-wing point of view, but acknowledge some limited role for scruffy, left-wing orientations. The other type of approach is the obvious mirror: you start from the disorderly leftwing side and struggle to be neater about what you are doing. I prefer the latter approach to the former. I will tell you why, and then lay out the beginnings of such an approach.
[...]
To read why and how:
https://cognitivesciencesociety.org/wp-content/uploads/2019/...
See my sibling post citing Roger Schank who coined the terms, and quoting Marvin Minsky's paper, "Logical Versus Analogical or Symbolic Versus Connectionist or Neat Versus Scruffy" and the "Neats and Scruffies" wikipedia page.
But I guess I also don't know enough about the CYC approach to say. Maybe neither of them fit what I think of as "neat".
https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...
https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...
"We should take our cue from biology rather than physics..." -Marvin Minsky
https://grandtextauto.soe.ucsc.edu/2008/02/14/ep-44-ai-neat-...
EP 4.4: AI, Neat and Scruffy
by Noah Wardrip-Fruin, 6:11 am
A name that does appear in Weizenbaum’s book, however, is that of Roger Schank, Abelson’s most famous collaborator. When Schank arrived from Stanford to join Abelson at Yale, together they represented the most identifiable center for a particular approach to artificial intelligence: what would later (in the early 1980s) come to be known as the “scruffy” approach. [7] Meanwhile, perhaps the most identifiable proponent of what would later be called the “neat” approach, John McCarthy, remained at Stanford.
McCarthy had coined the term “artificial intelligence” in the application for the field-defining workshop he organized at Dartmouth in 1956. Howard Gardner, in his influential reflection on the field, The Mind’s New Science (1985), characterized McCarthy’s neat approach this way: “McCarthy believes that the route to making machines intelligent is through a rigorous formal approach in which the acts that make up intelligence are reduced to a set of logical relationships or axioms that can be expressed precisely in mathematical terms” (154).
This sort of approach lent itself well to problems easily cast in formal and mathematical terms. But the scruffy branch of AI, growing out of fields such as linguistics and psychology, wanted to tackle problems of a different nature. Scruffy AI built systems for tasks as diverse as rephrasing newspaper reports, generating fictions, translating between languages, and (as we have seen) modeling ideological reasoning. In order to accomplish this, Abelson, Schank, and their collaborators developed an approach quite unlike formal reasoning from first principles. One foundation for their work was Schank’s “conceptual dependency” structure for language-independent semantic representation. Another foundation was the notion of “scripts” (later “cases”) an embryonic form of which could be seen in the calling sequence of the ideology machine’s executive. Both of these will be considered in more detail in the next chapter.
Scruffy AI got attention because it achieved results in areas that seemed much more “real world” than those of other approaches. For comparison’s sake, consider that the MIT AI lab, at the time of Schank’s move to Yale, was celebrating success at building systems that could understand the relationships in stacks of children’s wooden blocks. But scruffy AI was also critiqued — both within and outside the AI field — for its “unscientific” ad-hoc approach. Weizenbaum was unimpressed, in particular, with the conceptual dependency structures underlying many of the projects, writing, “Schank provides no demonstration that his scheme is more than a collection of heuristics that happen to work on specific classes of examples” (199). Whichever side one took in the debate, there can be no doubt that scruffy projects depending on coding large amounts of human knowledge into AI systems — often more than the authors acknowledged, and perhaps much more than they realized.
[...]
[7] After the terms “neat” and “scruffy” were introduced into the AI and cognitive science discourse by Abelson’s 1981 essay, in which he attributes the coinage to “an unnamed but easily guessable colleague” — Schank.
https://cse.buffalo.edu/~rapaport/676/F01/neat.scruffy.txt
Article: 35704 of comp.ai
From: engelson@bimacs.cs.biu.ac.il (Dr. Shlomo (Sean) Engelson)
Newsgroups: comp.ai
Subject: Re: who first used "scruffy" and "neat"?
Date: 25 Jan 1996 08:17:13 GMT
Organization: Bar-Ilan University Computer Science
In article <4e2th9$lkm@cantaloupe.srv.cs.cmu.edu> Lonnie Chrisman <ldc+@cs.cmu.edu> writes:
so@brownie.cs.wisc.edu (Bryan So) wrote:
>A question of curiosity. Who first used the terms "scruffy" and "neat"?
>And in what document? How about "strong" and "weak"?
Since I don't see a response yet, I'll take a stab. The earliest use of
"scruffy" and "neat" that comes to my mind was in David Chapman's "Planning
for Conjunctive Goals", Artificial Intelligence 32:333-377, 1987. "Weak"
evidence for this being the earliest use is that he does not cite any earlier
use of the terms, but perhaps someone else will correct me and give an
earlier citation.
One earlier citation is Eugene Charniak's paper in AAAI 1986, "A Neat
Theory of Marker Passing". I think, though, that the terms go way
back in common parlance, almost certainly to the 70s at least. Any of
the "old-timers" out there like to comment?
Article: 35781 of comp.ai
From: fass@cs.sfu.ca (Dan Fass)
Newsgroups: comp.ai
Subject: Re: who first used "scruffy" and "neat"?
Date: 26 Jan 1996 10:03:35 -0800
Organization: Simon Fraser University, Burnaby, B.C.
Abelson (1981) credits the neat/scruffy distinction to Roger Schank.
Abelson says, ``an unnamed but easily guessable colleague of mine
... claims that the major clashes in human affairs are between the
"neats" and the "scruffies". The primary concern of the neat is
that things should be orderly and predictable while the scruffy
seeks the rough-and-tumble of life as it comes'' (p. 1).
Abelson (1981) argues that these two prototypic identities --- neat
and scruffy --- ``cause a very serious clash'' in cognitive science
and explores ``some areas in which a fusion of identities seems
possible'' (p. 1).
- Dan Fass
REF
Abelson, Robert P. (1981).
Constraint, Construal, and Cognitive Science.
Proceedings of the 3rd Annual Conference of the Cognitive Science
Society, Berkeley, CA, pp. 1-9.
Aaron Sloman, 1989: "Introduction: Neats vs Scruffies"
https://www.cs.bham.ac.uk//research/projects/cogaff/misc/scr...
>There has been a long-standing opposition within AI between "neats" and "scruffies" (I think the terms were first invented in the late 70s by Roger Schank and/or Bob Abelson at Yale University).
>The neats regard it as a disgrace that many AI programs are complex, ill-structured, and so hard to understand that it is not possible to explain or predict their behaviour, let alone prove that they do what they are intended to do. John McCarthy in a televised debate in 1972 once complained about the "Look Ma no hands!" approach. Similarly, Carl Hewitt, complained around the same time, in seminars, about the "Hairy kludge (pronounced klooge) a month" approach to software development. (His "actor" system was going to be a partial solution to this.)
>The scruffies regard messy complexity as inevitable in intelligent systems and point to the failure so far of all attempts to find workable clear and general mechanisms, or mathematical solutions to any important AI problems. There are nice ideas in the General Problem Solver, logical theorem provers, and suchlike but when confronted with non-toy problems they normally get bogged down in combinatorial explosions. Messy complexity, according to scruffies, lies in the nature of problem domains (e.g. our physical environment) and only by using large numbers of ad-hoc special-purpose rules or heuristics, and specially tailored representational devices can problems be solved in a reasonable time.
Roger Schank
https://en.wikipedia.org/wiki/Roger_Schank
Robert Abelson
https://en.wikipedia.org/wiki/Robert_Abelson
Marvin Minsky
https://en.wikipedia.org/wiki/Marvin_Minsky
Neats and scruffies
https://en.wikipedia.org/wiki/Neats_and_scruffies
>Scruffy projects in the 1980s
>The scruffy approach was applied to robotics by Rodney Brooks in the mid-1980s. He advocated building robots that were, as he put it, Fast, Cheap and Out of Control, the title of a 1989 paper co-authored with Anita Flynn. Unlike earlier robots such as Shakey or the Stanford cart, they did not build up representations of the world by analyzing visual information with algorithms drawn from mathematical machine learning techniques, and they did not plan their actions using formalizations based on logic, such as the 'Planner' language. They simply reacted to their sensors in a way that tended to help them survive and move.[13]
>Douglas Lenat's Cyc project was initiated in 1984 one of earliest and most ambitious projects to capture all of human knowledge in machine readable form, is "a determinedly scruffy enterprise".[14] The Cyc database contains millions of facts about all the complexities of the world, each of which must be entered one at a time, by knowledge engineers. Each of these entries is an ad hoc addition to the intelligence of the system. While there may be a "neat" solution to the problem of commonsense knowledge (such as machine learning algorithms with natural language processing that could study the text available over the internet), no such project has yet been successful.
[...]
>John Brockman writes "Chomsky has always adopted the physicist's philosophy of science, which is that you have hypotheses you check out, and that you could be wrong. This is absolutely antithetical to the AI philosophy of science, which is much more like the way a biologist looks at the world. The biologist's philosophy of science says that human beings are what they are, you find what you find, you try to understand it, categorize it, name it, and organize it. If you build a model and it doesn't work quite right, you have to fix it. It's much more of a "discovery" view of the world."[4]
The wikipedia page about Neats and Scruffies that I linked you to is in my opinion well written, clearly defines the meanings each term, and presents plenty of evidence and citations and background. I'll give you the benefit of doubt that of course you've already read and understand it, so if you disagree with the history and citations on the wikipedia page and all the original papers and books and people cited and quoted, and can present better evidence and arguments to prove that you're right and they're all wrong, then you are free to go try to rewrite history by sharing your own definitions and citations, and correcting the errors on wikipedia. Good luck! I suggest you start by writing suggestions and presenting your evidence on the talk page first, instead of just directly editing the wikipedia page itself, to see what other experts in the field think and achieve consensus, or else it will likely be considered vandalism and be reverted.
You seem to be missing the point that the world is not strictly black and white, and ever since the terms were originally coined, the people who defined them and many other people have strongly recommended fusing both the "neat" and "scruffy" approaches, and LLMs actually do incorporate some ad-hoc "scruffy" aspects into their mathematical "neat" approach, and that's why they work so much better than simple perceptrons or neural nets. But they are still much more "neat" than "scruffy", and combining the two approaches does not flip the meaning of the two terms. I just discussed the fusion of scruffy and neat here, and quoted the original 41-year-old essay from 1982 by Robert Abelson that defined the terms and recommended fusing the two different approaches:
https://news.ycombinator.com/item?id=37359235
And also:
https://news.ycombinator.com/item?id=37354564
But before you go off and edit the Neats and Scruffies wikipedia page with your own definitions, please take the time to read the original essay by Robert Abelson that defines the terms first, like I did. In the link above, I cited it, tracked down the pdf, and quoted the relevant part of it for you, but you should probably do your homework first and read the whole thing before editing the wikipedia page about it. But be aware that it uses a lot of other technical terms and jargon that have well known definitions to practitioners in the field, so the common layman definitions of words you learned in grammar school may not apply.
Cyc is clearly the paradigm of "scruffy" and like biology, and perceptrons and neural nets are clearly the paradigm of "neat" and like physics, and that's how those terms have been widely used for more than four decades.
I think what's interesting in the jargon vs. plain definition tension here is related to what you noted in this most recent comment. It's that the words "neat" and "scruffy" - that is, just the english words, not the AI jargon terms - are not really symmetrical. A scruffy thing can easily become more neat while remaining scruffy, but introducing scruffiness into a neat thing tends to just make it scruffy. Neat is more totalizing.
So you say LLMs still fall into the "neat" camp - AI jargon this time - because of their mathematical core and lineage, and that's fair enough. But you also say that they incorporate "scruffy" techniques - jargon again - and I think that makes them - switching to the english words here - seem pretty scruffy, because the scruffy techniques are themselves scruffy, and incorporating all these different techniques is itself a scruffy thing to do.
Same. I wonder if training an LLM on the database would make it more "grounded"? We'll probably never know as Cycorp will keep the data locked away in their vaults forever. For what purpose? Probably even they don't know.
>AM (no relation to any need for screaming)
heh.
The post originally pointed to Lenat's Wikipedia page; now it's an obituary by Gary Marcus which seems more appropriate.
Doug was amazing - bold, brilliant, visionary, charismatic.
RIP, dear leader.
https://chat.openai.com/share/dbd59c92-696b-45d3-8097-c09a23...
The article discusses generative AI models like ChatGPT and contrasts them with knowledge-based AI systems like Cyc.
Generative models can produce very fluent text, but they lack true reasoning abilities and can make up plausible-sounding but false information. This makes them untrustworthy.
In contrast, Cyc represents knowledge explicitly and can logically reason over it. This makes it more reliable, though it struggles with natural language and speed.
The article proposes 16 capabilities an ideal AI system should have, including explanation, reasoning, knowledge, ethics, and language skills. Cyc and generative models each have strengths and weaknesses on these dimensions.
The authors suggest combining symbolic systems like Cyc with generative models to get the best of both approaches. Ways to synergize them include:
Using Cyc to filter out false information from generative models.
Using Cyc's knowledge to train generative models to be more correct.
Using generative models to suggest knowledge to add to Cyc's knowledge base.
Using Cyc's reasoning to expand what generative models can say.
Using Cyc to explain the reasoning behind generative model outputs.
Overall, the article argues combining reasoning-focused systems like Cyc with data-driven generative models could produce more robust and trustworthy AI. Each approach can shore up weaknesses of the other.
May he rest in peace.
Cyc - https://news.ycombinator.com/item?id=33011596 - Sept 2022 (2 comments)
Why AM and Eurisko Appear to Work (1983) [pdf] - https://news.ycombinator.com/item?id=28343118 - Aug 2021 (17 comments)
Early AI: “Eurisko, the Computer with a Mind of Its Own” (1984) - https://news.ycombinator.com/item?id=27298167 - May 2021 (2 comments)
Cyc - https://news.ycombinator.com/item?id=21781597 - Dec 2019 (173 comments)
Some documents on AM and EURISKO - https://news.ycombinator.com/item?id=18443607 - Nov 2018 (10 comments)
One genius's lonely crusade to teach a computer common sense (2016) - https://news.ycombinator.com/item?id=16510766 - March 2018 (1 comment)
Douglas Lenat's Cyc is now being commercialized - https://news.ycombinator.com/item?id=11300567 - March 2016 (49 comments)
Why AM and Eurisko Appear to Work (1983) [pdf] - https://news.ycombinator.com/item?id=9750349 - June 2015 (5 comments)
Ask HN: Cyc – Whatever happened to its connection to AI? - https://news.ycombinator.com/item?id=9566015 - May 2015 (3 comments)
Eurisko, The Computer With A Mind Of Its Own - https://news.ycombinator.com/item?id=2111826 - Jan 2011 (9 comments)
Open Cyc (open source common sense) - https://news.ycombinator.com/item?id=1913994 - Nov 2010 (22 comments)
Lenat (of Cyc) reviews Wolfram Alpha - https://news.ycombinator.com/item?id=510579 - March 2009 (16 comments)
Eurisko, The Computer With A Mind Of Its Own - https://news.ycombinator.com/item?id=396796 - Dec 2008 (13 comments)
Cycorp, Inc. (Attempt at Common Sense AI) - https://news.ycombinator.com/item?id=20725 - May 2007 (1 comment)
HN location for discussion of Lenat's last paper (with Gary Marcus) about ways to reconcile Cyc's strengths with LLMs.
[1] https://medium.com/@jcbaillie/beyond-the-symbolic-vs-non-sym...
[2] https://past.date-conference.com/proceedings-archive/2016/pd...
"Lenat found out about computers in a a manner typical of his entrepreneurial spirit. As a high school student in Philadelphia, working for $1.00 an hour to clean the cages of experimental animals, he discovered that another student was earning $1.50 to program the institution's minicomputer. Finding this occupation more to his liking, he taught himself programming over a weekend and squeezed his competitor out of the job by offering to work for fifty cents an hour less.31 A few years later, Lenat was programming Automated Mathematician (AM, for short) as a doctoral thesis project at the Stanford AI Laboratory." p. 178
And here's an count of an early victory for AI in gaming against humans by Lenat's EURISKO system (https://en.wikipedia.org/wiki/Eurisko):
"Ever the achiever, Lenat was looking for a more dramatic way to prove teh capabilities of his creation. The identified the occasion space-war game called Traveler TCS, then quite popular with the public Lenat wanted to reach. The idea was for each player to design a fleet of space battleships according to a thick, hundred-page set of rules. Within a budget limit of one trillion galactic credits, one could adjust such parameters as the size, speed, armor thickness, autonomy and armament of each ship: about fifty adjustments per ship were needed. Since the fleet size could reach a hundred ships, the game thus offered ample room for ingenuity in spite of the anticlimactic character of the battles. These were fought by throwing dice following complex tables based on probability of survival of each ship according to its design. The winner of the yearly national championship was commissioned inter galactic admiral and received title to a planet of his or her choice ouside the solar system.
Several months before the 1981 competition, Lenat fed into EURISKO 146 Traveler concepts, ranging from the nature of games in general to the technicalities of meson guns. He then instructed the program to develop heuristics for making winning war-fleet designs. The now familiar routine of nightly computer runs turned into a merciless Darwinian contest: Lenat and EURISKO together designed fleets that battled each other. Designs were evaluated by how well they won battles, and heuristics by how well they designed fleets. This rating method required several battles per design, and several designs per heuristic, which amounted to a lot of battles: ten thousand in all, fought over two thousand hours of computer time.
To participants in the national championship of San Mateo,California, the resulting fleet of ninety-six small, heavily armored ships looked ludicrous. Accepted wisdom dictated fleets of about twenty behemoth ships, and many couldn't help laughing. When engagements started, they found out that the weird armada held more than met the eye. One interesting ace up Lenat's sleeve was a small ship so fast as to be almost unstoppable, which guaranteed at least a draw. EURISKO had conceived of it through the "look for extreme cases" heuristic (which had mutated, incidentally, into mutated, incidentally, into "look for almost extreme cases")." p. 182
If you're a young person working in AI, by which I mean you're less than 30, and if you have not already done so, you should read about AI history in three decade 60s - 90s.
Edit: Fixed wrong name for the competition
In which case, yes, something like that did happen. Per the Wikipedia page:
Lenat and Eurisko gained notoriety by submitting the winning fleet (a large number of stationary, lightly-armored ships with many small weapons)[3] to the United States Traveller TCS national championship in 1981, forcing extensive changes to the game's rules. However, Eurisko won again in 1982 when the program discovered that the rules permitted the program to destroy its own ships, permitting it to continue to use much the same strategy.[3] Tournament officials announced that if Eurisko won another championship the competition would be abolished; Lenat retired Eurisko from the game.[4] The Traveller TCS wins brought Lenat to the attention of DARPA,[5] which has funded much of his subsequent work.
Lenat is on a short list of people I expected/hoped to meet at some point when context provided the practical reason.
He has been a hero to me for his creativity and fearlessness regarding his symbolic vision.
So sad I will never meet him, but my appreciation for him will never die.
But I think that is one of the things he should be admired for. How can anyone know how an ambitious approach will pan out without the great risk of going all in?
Anyone willing to risk a Don Quixote aspect to their career, in pursuit of a breakthrough, is someone who cares deeply about something beyond themselves.
And recognizing the limits of Lenat’s impact today doesn’t preclude both the direct and indirect impact on future progress.
I found him inspiring on multiple levels.