Hacker News Reader

Comment by mathisfun123 | Hacker News Reader

>Common Lisp programs run by default in a way that calls to undefined functions are detected.

Cool so what you're telling me is that by default every single function call incurs the unavoidable overhead of indirecting through some lookup for a function bound to a symbol. And you're proud of this?

View on HN

I thought you know Lisp? Now you are surprised that Lisp often looks up functions via symbols -> aka "late binding"? How can that be? That's one of the basic Lisp features.

Next you can find out what optimizing compilers do to avoid it, where possible or where wanted.

At no point in time did I claim to know lisp well. I stated my familiarity at the outset. But what you all did was claim to know a lot about every other interpreted runtime without a grain of salt.

>Next you can find out what optimizing compilers do to avoid it, where possible or where wanted.

But compilers I am an expert in and what you're implying is impossible - either you have dynamic linkage, which means symbol resolution is deferred until call (and possibly guarded) or you have the equivalent of RTLD_NOW ie early/eager binding. There is no "optimization" possible here because the symbol is not Schrodinger's cat - it is either resolved statically or at runtime - prefetching symbols with some lookahead or cabinet is the same thing as resolving at calltime/runtime because you still need a guard.

> But compilers I am an expert in and what you're implying is impossible

> it is either resolved statically or at runtime

Just tell Lisp which calls to statically resolve, inline, optimize. Overwrite the global default.

  (defun foo (a)
    (declare (inline +)
             (optimize (speed 3))
             (type (integer 0 100) a))
    (* 10 (+ 3 a)))

Above tells Lisp to inline the + function, optimize for speed and declares the type of a to be an integer in the range 0 to 100.

  * (disassemble #'foo)
  ; disassembly for FOO
  ; Size: 32 bytes. Origin: #x7006DC8544  ; FOO
  ; 44:       40190091         ADD NL0, R0, #6
  ; 48:       5C0180D2         MOVZ TMP, #10
  ; 4C:       0A7C1C9B         MUL R0, NL0, TMP
  ; 50:       FB031AAA         MOV CSP, CFP
  ; 54:       5A7B40A9         LDP CFP, LR, [CFP]
  ; 58:       BF0300F1         CMP NULL, #0
  ; 5C:       C0035FD6         RET
  ; 60:       E00120D4         BRK #15    ; Invalid argument count trap

As you can see in the machine code, Lisp then uses the native machine code ADD and MUL instructions.

What you're missing is that, unlike any other commonly used language runtime, compilation in CL is not all-or-nothing, nor is it left solely to the runtime to decide which to use. A CL program can very well have a mix of interpreted functions and compiled functions, and use late or eager binding based on that. This is mostly up to the programmer to decide, by using declarations to control how, when, and if compilation should happen.

To be fair, any dynamic language with a JIT will mix interpreted and compiled functions, and will probably claim as a strength not leaving to the programmer the problem of which to compile.

Opinions may vary on that point.

It should also be noted that by spec symbols in the system package (like + and such) should not be redefined. This offers “unspecified” behavior and lets the system make optimizations out of the box.

Outside of that you can selectively optimize definitions to empower the system to make better decisions at the cost of runtime protection or dynamism. However these are all compiler specific.

You are incorrect; optimizations are possible in dynamic linking by making first references go through a slow path, which then patches some code thunk to make a direct call. This is limited only by the undesirability of making either the callling object or the called object a private, writable mapping. Because we want to keep both objects immutable, the call has to go into some privately mapped jump table. That table contains a thunk that can be rewritten to do a direct call to an absolute address. If we didn't care about sharing executables between address spaces we could patch the actual code in one object to jump directly to a resolved address in the other object. (mmap can do this: MAP_FILE plus MAP_PRIVATE: you map a file in a way that you can change the memory, but the changes appear only in your address space and not the file.)

Compared to Python, Common Lisp hardly has any performance issues.

Okay well when pytorch, tensorflow, pandas, Django, flask, numpy, networks, script, xgboost, matplotlib, spacy, scrapy, selenium get ported to lisp, I'll consider switching (only consider though since the are probably at least another 20 python python packages that I couldn't do my job without).

That is a typical consumerism: "Give me everything ready to use and then I'll use it."

How about bringing some value to the community?

For the sake of anyone reading this thread who isn't in the know: many of these libraries are really written in C/C++ and have Python bindings.

i said ported not implemented; the likelihood that any of those libraries sprout lisp bindings is about as likely as them being rewritten in lisp. so it's the same thing and the point is clear: i don't care about some zany runtime feature, i care about the ecosystem.

Stop moving the goalposts: your answer to a commenter who stated that Common Lisp was faster than Python (a fact) was a list of packages, many of which are (1) not even written in Python and (2) some of them actually do have Common Lisp bindings.

This is not necessarily the case.

Firstly, functions that are in the same compilation unit that refer to each other can use a faster mechanism, not going through a symbol. The same applies to lexical functions. Lisp compilers support inlining, and the spec allows automatic inlining between functions in the same compilation unit, and it allows calls to be less dynamic and m more optimized. If f and g are in the same file, where g calls f, then implementations are not required to allow f and go to be separately redefinable. So that is to say, if f is redefined only, the existing g may keep calling the old f. The intent is that redefinition has the granularity of compiled files: if a new version of the entire compiled file is loaded, then f and g get redefined together and all is cool.

Lisp symbol lookup takes place at read time. If we are calling some function foo and have to go through the symbol (it's in another compilation unit), there is no hashing of the string "foo" going on at call time. The calling code hangs on to the foo symbol, which is an object. The hashing is done when the caller is loaded. The caller's compiled file contains literal objects, some of which are symbols. A compiled file on disk records externalized images of symbols which have the textual names; when those are internalized again, they become objects.

The "classic" Lisp approach for implementing a global function binding of a symbol is be to have dedicated "function cell" field in the symbol itself. So, the compiled module from which the call is emanating is hanging on to the foo symbol as static data, and that symbol has a field in it (at a fixed offset) from which it can pull the current function object in order to call it (or use it indirectly).

Cross-module Lisp calls have overhead due to the dynamism; that's a fact of life. You don't get safety for nothing.

(Yes, yes, you can name ten "Lisp" implementations which do a hashed lookup on a string every time a function is called, I know.)

> If f and g are in the same file, where g calls f, then implementations are not required to allow f and go to be separately redefinable. So that is to say, if f is redefined only, the existing g may keep calling the old f. The intent is that redefinition has the granularity of compiled files: if a new version of the entire compiled file is loaded, then f and g get redefined together and all is cool.

This isn't the default behaviour though, right?

That depends. The Common Lisp standard says nothing on the subject. CMUCL[1] and its descendent SBCL[2] do something clever called local call. It's not terribly difficult to optimize hot spots in your code to use local call. Outside of the bottlenecks, the full call overhead isn't significant for the overwhelming majority of cases. It's not like full call is any more expensive than a vtable lookup anyhow.

[1] https://cmucl.org/downloads/doc/cmu-user-2010-05-03/compiler...

[2] https://www.sbcl.org/manual/#Miscellaneous-Efficiency-Issues

Do you think Python or Ruby or PHP are any different? And yet, not one of them actually chose to use this in a sane way, where a simple lookup error doesn't have to crash the whole program.

Reply on news.ycombinator.com