How to Make Common Lisp go Faster Than C | Hacker News Reader

How to Make Common Lisp go Faster Than C

lrde.epita.fr

18 pointssmanek42 comments

My co-founder tried out some of the techniques in the paper on our system: "I was skeptical, probably because I've never done much performance tuning like this. But I changed (triples) to create a 2-d array of fixnums instead of the default list of lists from the select. I added the declarations. It's about 100x faster. 100!" Very nice :)

When I mentioned that it may be possible for a web startup in beta to improve the speed of their code by a factor of 1024 over no particular timescale, I was downmodded by five points ( http://news.ycombinator.com/item?id=187692 ).

I can only advise that you shouldn't quit while you're ahead. Now that you've eliminated the most obvious performance bottlenecks, you may find that less obvious bottlenecks have emerged. Removing these could give you an additional factor of 10 improvement.

We're using the SBCL profiler on a regular basis to speed up the system whenever we find an area that feels a bit sluggish. We've made very large speed gains with only minor coding modifications.

I'm very impressed with your quantative approach. I'm also very curious about the type of modifications you're making. What proportion of changes are local changes which affect common cases, what proportion of changes are local changes which affect corner cases and what proportion of changes are architectural? Is it 70:20:10?

Thats a good question. Our workflow is to look at the profiler results, and work on the functions that are taking the most time. Sometimes we'll find a function being called far too many times for the page being viewed, so we'll look into the program logic to solve that. Our code is highly integrated, so most pages are using functions or methods common to many others, so a fix (or a bug) effects a large part of the system. The example 100x triples improvement sped up the entire system as triples are used at the core of our security and relations. This is a long winded way of saying I'm not sure what the proportions would be, a guess would be 60:10:30

Probably not. Remember Amdah's Law: http://en.wikipedia.org/wiki/Ahmdal%27s_Law

I'm aware of Amdahl's law. I also know of a case where adding an index to a production database gave a factor of 1000 improvement. If your design started as a proof-of-concept, you made a rash assumption, you had a tight deadline or you're just plain incompetent then there's almost no lower bound to initial system efficiency.

My comment was aimed at your down-modded comment. You can make 10% performance improvements in dozens of modules throughout your system, but if they only account for, say, 5% of the total execution time, it won't make much difference.

Premature optimization may be the root of all evil, but failure to optimize isn't too hot either.

Glad to hear your results were so dramatic.

We only optimize when things start feeling a bit slow - never prematurely.

How did this comment end up in the rss feed?

Interestingly, the "comments" link for it points to http://news.ycombinator.com/item?id=196342 -- the original post without any comments shown!

It was submitted as a standalone post. There are no comments on the new submission.

http://www.lrde.epita.fr/~didier/research/verna.06.imecs.pdf

There is an easy work around for this. Submit the pdf link. Then comment with the pdf link. That adds one level of indirection, until pg gets around to fixing it.

This is hacker news, So I'm assuming people can use their browser and copy and paste.

  1) copy link location.
  2) paste link location in address bar.
  2) remove scribd.com crap, leaving the original pdf link.
  3) load original pdf link.

"While Lisp is mainly known for its basement on list processing, the Common Lisp standard features very efficient data types such as specialized arrays, structs or hash tables, making lists almost completely obsolete."

Uh?!?

Authors of computer science papers aren't exactly known for their command of the finer points of the English language. basement?

In languages that can mutate data structures using side-effects, linked lists are rarely the best choice of data structure. Note, for example, that Python and Ruby have built-in array-based lists, but no linked lists. Similar structures are the default for lists in C++, Java, and C#.

That said, I'm pretty sure people use linked lists a lot in Common Lisp.

Memory usage and random access times are far better in arrays, but for dynamic resizing, deletion, insertion, appending, and splitting, that is, operations that deal with data structure rather than the content, lists are far superior to arrays.

I used to hand-code linked lists back when I used C/C++ heavily. They have some very significant advantages for modeling certain kinds of data.

I didn't mean to imply that linked lists aren't a useful data structure, just that arrays and array-based lists are well suited to an apparently wider range of problems.

No one uses lists in a LISt Processing language. :P

Many of the interesting macros I've seen use list operations heavily to slice and dice their arguments and body code. Most experienced Lispers use macros quite a bit.

Given that macros take lists as input and output lists, it would be surprising if they didn't use list operations heavily...

Yes, I didn't intend for my comment to be news to anybody who uses Lisp. I just wanted to point out why list processing is still important.

Oh, and sorry. I actually submitted a link to the PDF directly, but YCNews scribd it and now it's unreadable. Here is the original:

http://www.lrde.epita.fr/~didier/research/verna.06.imecs.pdf

This is ridiculous. They should change the name back to YC News since they've decided to shill for their own investments.

This scribd thing is getting ridiculous. Why not make it an opt-out arrangement for submitters?

The Scribd file was degraded in quality to absolute illegibility, I had to grab the original link out of the URL just so I could read it.

Opt-out? It's annoying. They should just stop doing it for everyone perhaps, or just use it as a backup in case the site is knocked off line by the influx of social news traffic.

And while they're at it, they could also check for duplicate submissions: http://news.ycombinator.com/item?id=172432

They do check. The sequence of events here seems to be:

- Person submits PDF article.

- HN rewrites article to scribd link.

- People complain about scribd link.

- Admin manually corrects link to point directly to PDF.

- A month later, another person submits same PDF article.

- HN rewrites article to scribd link, but because the original article was manually corrected, the scribd link doesn't match the original article, so it doesn't register as a dupe.

I sense that there's a simple solution to this. Perhaps we could drop the automatic scribdifynig of PDF links?

I'm not sure why HN auto-posts to scridb.

If they're trying to send more traffic to that site, then we're S.O.L.

OTOH, if they're just trying to increase the number of documents in scribd's repository, then a parallel submission process would be the solution.

I.e., keep HN posts pointing to the original pdf, but also auto-submit to scribe, behind the scenes in a separate thread.

To be fair, the purpose of HN is to serve YC -- that's why we don't see ads on the site. People have suggested not scribd-ing everything before; I can only assume that the reason this has not yet happened is because the feedback we are providing is more valuable to YC than a few upset readers.

And this may well be the case: When most people have problems accessing a web page one way, they won't tell you directly; instead, they'll try to find another way to access it or give up. By providing only one way to read PDFs on a discussion board, you get access to this otherwise invisible negative feedback.

Yes, but not every idea turns out to be a good one.

Based on the unanimous jeers every time a PDF submission gets autoconverted, scribd appears not to be one of the good ones. Maybe it's time to try something else.

It isn't unanimous.

> the purpose of HN is to serve YC -- that's why we don't see ads on the site.

We don't see ads because the site is run by millionaires and hardly takes any computing resources or bandwidth.

I assume you remember how those millionaires got to be that way and what they're doing these days, right?

But wouldn't either the original pdf link or the resulting scribd link show up as a duplicate in HN's dupe checker, since both were in the submitted list at one time or another?

Look at the link in http://news.ycombinator.com/item?id=172432 ... It's been changed from the scribd link to the original pdf file. I'm guessing they only check for dupes against a list of the current article links, and that they don't maintain a separate list of submitted links (which, except for cases like this one, would be the same list).

It's been changed from the scribd link to the original pdf file.

Ah, I didn't see that (I thought once it's been trans-scribd, it stays that way).

But that raises another question: if http://news.ycombinator.com/item?id=172432 points to the original pdf (i.e., http://www.lrde.epita.fr/~didier/research/verna.06.imecs.pdf) and the guy who submitted this also posted the same original pdf link, why wasn't it detected as a dupe right then?

The scribdfication must take place before duplicate-checking.

We can hack around the scribd censorship. Everybody who aubmits a .pdf knows it will be converted into the useless scribd format. So, the original submitter needs to immediately post a comment that is the original .pdf link.

If HN scribd damage occurs on that link, then we will find other ways to route around that damage, for example, by using tinyurl.

Down with scribd!!!

.pdf != HD-DVD key

Hacker News != Digg

tinyurl is a great idea too.

Reply on news.ycombinator.com