A Retrospective on Paradigms of AI Programming (2002) | Hacker News Reader

A Retrospective on Paradigms of AI Programming (2002)

norvig.com

45 points0xpgm33 comments

Peter wrote this about the same time I first met him at the Lisp Users Venders conference in San Diego. I had just had my Common Lisp book published by Springer Verlag. Fast forward a few years and he helped me a lot my reading through my Java AI book manuscript, offering a lot of good advice. A year or two after that when I had lunch with him at Google I was really surprised that he had moved on to Python. It is ironic to me, that I followed him to mostly using Python, but 20 years later!

In the end, it is all about getting valuable code written and deployed, not using a particular language. That said, I still enjoy using a variety of languages, even though Python is the most practical and productive language for what I work on now.

If anyone is interested PAIP is downloadable at https://github.com/norvig/paip-lisp

Funny to see this now, when AI is done by using Python to orchestrate numeric kernels written in C++ or Fortran. GOFAI lost, big time, and the key to success in AI is the ability to perform matrix multiplications as fast as machinely possible -- which means the central data structure you should be using is the array or vector, to ensure data locality so you don't blow up your cache.

Lisp is, almost, the opposite of a language you would choose to do AI in today.

It may be a more powerful language for other applications, but that will hardly matter because the productivity gains from using AI will dwarf those from programming language choice.

Common Lisp has certainly better array and vector support than Python so this can't be it.

on the minimal chance you're not being sarcastic, could you say more about this?

Common Lisp has has native support for vectors and arrays. They can be adjustable or fixed size, displaced to other arrays (kind of a "view" with a different geometry), specialized on elements of particular type or generic. For adjustable vectors you can specify their allocation growth increments for push operations. Or pre-allocate adjustable arrays but specify a lower initial size. Arrays can be accessed in row-major order mapping them into vectors etc etc.

Foreign function interface of different implementations typically provide a way to access C allocated memory areas as Lisp arrays.

numpy changed the ecosystem dramatically.. the combination of engineering-efficient matrix math, plus the plug-and-play, low barriers to entry broad ecosystem, tipped the scales and it never went back. Millions of coders literally are playing and implementing using these python stack tools.

The efficient part in numpy is certainly not the Python code.

Yes but what difference does it make? The interface is python.

Besides, libraries are much less rigid than language features so maintainers can and do iterate on these much faster.

Saying this as a big Lisp/scheme/emacs fan, btw.

What I was getting at is the reason why Python is popular and CL isn't is not technical but social.

But this is fundamentally a native code library and we can bind it to any language that supports this. Nothing unique to Python here.

except, evolution (and market share) are built in things that really exist now. So .. millions of users, I would emphasize that part today.

Why would you think it’s sarcasm? Modern Common Lisp implementations not only generate well optimized code, they make it simple to check what’s really happening with DISASSEMBLE.

i am genuinely curious, because I think CL could do some amazing things in the symbolic algebra/compiler space here. Tensorflow's AutoGrad and AutoGraph (and I think JAX maybe) had to do terrible things to the python bytecode interpreter to compile methods, and Theano only worked by overhauling the entire number tower and adding its own control flow.

If CL could become a frontend for XLA perhaps, then maybe we can go full circle and turn Torch back into the lisp project it originally started out as!

That's why the numeric kernels, where all the speed-sensitive array and vector processing is done, are written in C++ or Fortran.

Not really, it just lacks the ecosystem. Rapid feedback cycles, prototyping, introspection, performance, etc. would actually make it the ideal language. If CL 2.0 came out today it would be ideal for ML/DL.

I have vague hopes that a Clojure implementation liberated from the JVM can be this language.

Practitioners make distinctions between machine learning (statistical analysis + optimization) and artificial intelligence (symbolic reasoning) for important reasons. For starters, machine learning systems based on matrix multiplies and accumulation are fundamentally unexplainable in human terms; for others, they're applied to different problems (path planning and game-playing started as AI problems, and haven't seen statistical optimization applied until very recently)

Out of curiosity have you ever done any nontrivial development in a production quality Lisp using appropriate tooling like Emacs/SLIME or one of the proprietary IDEs? It’s exactly the programming environment that would be optimal for an interactive coding dialog with a LLM.

Nothing stopping a lisp language from manipulating native data structures just like we do in Python.

I think that Python is actually kinda bad as a language for writing logic and building abstractions. We’re still using it for AI because that’s a small part of building these systems and inertia trumps everything.

Python at scale is really just build a graph, doesn't really matter what language you do that in other than history

The most helpful content here is the "list of the 52 most important lessons in PAIP" near the end.

Despite the title of the book, many of these lessons are not just about AI, or even particularly about symbolic programming or Lisp. They are generally useful advice about programming.

I think a good question might be, what language is better for gpt to program. If you want to write applications that accomplish intelligent things via symbolic manipulation, is lisp a good language to ask an AIto write it in? Is PAIP in the learning set for GPT4?

I’d be willing to bet Java, Python, or C are the best because of how much training data was in those languages. They aren’t the best at a theoretical level but at a practical level, almost definitely.

Has anyone tried to update Norvig's Lisp popularity metrics? (Though counting Usenet articles ain't what it used to be.) I'd love to see how it's aged after another 20 years.

I hate to say this but symbolic reasoning and planning has aged like milk. The use case is so limited and the only people using these are the people full of nostalgia. One paradigm shift into deep learning in 2012 and 10 years later we have something as general as GPT-4. Gatekeeping in academia is a real thing. Sad but true.

Symbolic reasoning is for things we truly know how to do. It can be used as supplemental implementation for known special cases as well.

There was no gate-keeping. Yann LeCun was doing neural networks in the late 80s.

Late 80s was the second AI winter, it wasn't all peaches. Why do you think just a handful of people were working on neural nets?

Because before backpropagation (mid-80's) nobody could train multi-layer neural nets.

Although to be fair, there wasn't "just a handful of people". Just look at Geoff Hinton's citations count throughout the '80s.

Machine learning has been around for decades. The real breakthrough has been the massive increase in compute power, the ubiquity of distributed systems, and the amount of data available.

Symbolic reasoning and related fields are just areas to were important to be researched back in the day. There are some very interesting results that came of that family of research, especially on the intersection of traditional math and CS, i.e. logic stuff, proof assistants, various solvers, etc.

Machine learning and its flavours were always there. What is new in deep learning is the scale of everything: amount of compute made possible by clusters of GPU-centric machines and amount of data made available by the Internet.

The shocking thing here how almost boring the core of recent advances is: do more of the same, much, much more, and quantity will turn into new qualities yet again.

Its amazing that everything is some conspiracy of 'they' just wouldn't let us advance. Most times these sentiments are from those that are just learning a field, and haven't realized that the 'establishment' wasn't stopping advancement, it was also advancing. Just because there is a big break through, doesn't mean that everything that came before wasn't needed to reach it.

Reply on news.ycombinator.com