How to Define New Intrinsics in SBCL | Hacker News Reader

How to Define New Intrinsics in SBCL

pvk.ca

92 pointschiachun20 comments

TIL: a solid CL implementation can provide safe maps from high-level intrinsic types to low-level implementation types at run-time. Also, SBCL has at least two IRs with each able to provide optimization hints to the code generator and compiler. So cool.

I love reading Paul's posts. His bit on Tobasco sort[0] was amazing.

[0] http://www.pvk.ca/Blog/2012/08/27/tabasco-sort-super-optimal...

update: link

That is indeed a good post; thanks for linking it. Beyond the sort itself, there's a nice illustration at the end of dynamically scoped ("special") variables being useful on occasion for something other than the usual example of temporarily shadowing a configuration variable (e.g. the destination of stdout, or logging level).

In certain ways, the difference between (Common) Lisp and other languages is the difference between Linux and Windows. When you have a fundamental problem in Windows, your only solution is to wait for a fix then reformat/reinstall. Similarly, when there's a problem in, say, a C compiler, all you can do is code around it, open an issue and wait for a fix in the next release.

In Common Lisp? Well, the compiler's right there under these floorboards. Have a long-running web application that's affected by this bug, and don't want to restart it? Just run this in the REPL.

Not sure exactly what you're trying to say here but there are plenty of open source C compilers [1], including GCC, which allow anyone to fix bugs as they see fit.

To your second point about not restarting a long running web applications — running code in a REPL won't fix your web app since it's a separate process. You might need a remote debugging mechanism to connect a console to the app, but at that point it may be much easier to restart the app.

[1] http://en.wikipedia.org/wiki/List_of_compilers#C_compilers

>running code in a REPL won't fix your web app since it's a separate process.

In the server you have a script that fires up a Lisp process and starts a Swank[0] server on port whatever. Then the developers, from their own machines, fire up Emacs and SLIME[1] into that port and run something along the lines of

  > (myapp.server:start :port 8000)
  <#server process>

  > (... some work ...)

[0] http://common-lisp.net/project/slime/doc/html/Lisp_002dside....

[1] http://common-lisp.net/project/slime/

This is a great feature of Lisp, Smalltalk, and similar languages, in development. In production however it can be extremely bad since the source code you believe is needed to reconstruct an image may not be what is actually in there, or some bootstrapping may have been done by hand and incorrectly captured later on. If the image has been saved with this anomaly a couple of decades ago then it's really fun trying to sort it out.

I've spent more time than I'd like to think doing software archeology to find out just why two systems behaved differently, so I would seriously recommend that if you do perform hot fixes in this manner then you reproduce the problem on a system built from source, apply and test the hot fix, make the change in the source and build your test system again to test, apply the hot fix to production, and schedule a redeploy.

Fixing a live system can also be hazardous if you are replacing a function that may be on the stack at the moment, or if you are in any sort of multithreaded/process environment. Bugs become much harder to reason around and fix in those cases because there can be race conditions about which functions actually get called while you're applying your fix.

TL;DR You can, but probably shouldn't.

> If the image has been saved with this anomaly a couple of decades ago then it's really fun trying to sort it out.

Nobody uses CL images that way, mostly because there aren't any CL implementations that produce images that are portable, and most implementations kill themselves to make the image (that's the easiest way to deal with memory allocation, multithreading, and open file descriptors). I've read about a couple of people trying but they don't get very far.

> Fixing a live system can also be hazardous if you are replacing a function that may be on the stack at the moment, or if you are in any sort of multithreaded/process environment. Bugs become much harder to reason around and fix in those cases because there can be race conditions about which functions actually get called while you're applying your fix.

This is a well known problem in Lisp-land, not because you're always hot-patching things in production, but because that's the way you develop code. There are a lot of things you can do to make atomic code loading possible, and different CL implementations have various levels of race condition proofing in the compiler and runtime systems.

> TL;DR You can, but probably shouldn't.

Most people who make this argument make it disingenuously or out of ignorance. I've heard it from people who don't seem to have any problems using ipython or pry or a SQL console to debug in production.

I don't have the exact link, but I recall an article on HN where the OP complained about NASA stopping to use LISP. The killer feature LISP had was what is just described here: the possibility to interact with live systems through the REPL - in this particular case, it wasn't a web app but a space probe. With it, you could simply fix some bugs while the probe was far away - orbiting around Mars IIRC, while you wouldn't be able to reprogram the whole probe system.

So, there are cases where you want to use the REPL to modify live systems in production.

Providing patches as compiled code loadable into applications is fairly common (!) in Lisp.

But usually those patches are already validated. Providing validated patches is one thing; developing them on the live system is another.

You would develop it in some live system anyway. That's the usual way to develop in Lisp. It does not mean that the 'live system' is also a 'production system'.

> running code in a REPL won't fix your web app since it's a separate process

Not in a Common Lisp web app.

Apparently what OP was talking about was a remote console into the webapp. You still can't open a random REPL and change the features of another process.

It's true that you can't open a random REPL and change the features of another process.

It is the norm for a webapp in Common Lisp to have an accessible REPL or "remote console".

Although you can fix a bug in GCC and submit it to the project (been there done that) you cannot deploy a gcc change as part of a C program.

Then again, you can't really deploy a Linux bug fix as part of a Linux program either, so the whole linux-windows-lisp-others analogy is tenuous.

The analogy holds in the sense that in Lisp we have less of a barrier between the programmer and the language, and in free software we have less of a barrier between users and the software. But the barriers are different.

And, ironically, the barrier which is out of your way in Lisp is still out of your way if the Lisp is a closed-source, proprietary Lisp. That is to say, doing many things in the Lispy way does not require you to rebuild the implementation from source, or to have the source.

Proprietary Lisps don't need to be totally closed source. For example Allegro CL provides much of the source code. mocl provides a source code license. LispWorks otoh is mostly closed source - minus parts of the editor.

> when there's a problem in, say, a C compiler, all you can do is code around it, open an issue and wait for a fix in the next release.

Fix at as you'd (presumably) do in lisp. I don't see your point. (Yes, you have to recompile it, C is compiled language.)

> long-running web application that's affected by this bug, and don't want to restart it? Just run this in the REPL.

Yeah, if you it don't have long-running code, it isn't heavy threaded, that change doesn't alter any data structures, you don't care about possible downtime etc

Does the SBCL compiler use a separate third-party assembler? Otherwise, how does it know the POPCNT opcode/encoding? I didn't see the author specify it in the post.

SBCL comes with its own assembler: https://github.com/sbcl/sbcl/blob/master/src/compiler/assem....

And the instructions themselves are defined here (e.g. for x86-64): https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64...

I think the question might've been how you can just write things like (instr popcnt...) and expect SBCL to know how to emit the right machine code for popcnt. It's mentioned briefly at the top, but the answer is that this post isn't adding the instruction (it would indeed not work if it were a new instruction SBCL doesn't yet know about). The example is just walking through modifying how it's generated by the VOPs, the code-generation level that sits a level above the assembler. The popcnt instruction itself had already been added to SBCL some time ago, with the define-instruction seen in the link above.

Reply on news.ycombinator.com