New Lisp-Stat Release | Hacker News Reader

New Lisp-Stat Release

lisp-stat.dev

105 pointsLisp-Stat55 comments

Open articleView on HN

Third major refactoring. Plotting significantly upgraded.

Fascinating project! Thank you for sharing, I plan to dig deeper.

A minor detail about the site: on the Privacy page, the first paragraph is covered up by the site header.

Reminds me that there was an attempt to realize the "Back to the Future" vision in Clojure.

https://github.com/incanter/incanter

It never took off but looks like there was modifications made up until three years ago.

There's a lively data science Clojure community and many libs, see eg: https://scicloj.github.io/docs/resources/libs/

I've been using lisp-stat in production as part of an algorithmic trading application I wrote. It's been very solid, and though the plotting is (perhaps was, in light of this new release) kinda unwieldy, I really enjoyed using it. Excited to check out the newest release.

Can you (or anyone who has run Lisp software in production) share how the experience has been? Do the management and other colleagues like it? Do they express any concerns? Want to understand the social aspect of running Lisp in production.

I’ve run lisp in production (SBCL). Fast to develop and run, and solid. Unfortunately the engineering team tasked with taking the product forward decided to rewrite it entirely in some version of react on (the quite unfortunately named, from my point of view) AWS lambdas because the frameworks these days are so choc full of stuff, most importantly security stuff!, that nothing can really compete with them anymore, even though the engineering overhead to use them is massive. (Edit: To be fair, the engineering overhead for me to write all that crap into Lisp would have been way more massive.) As a non-user-facing back-end, however, lisp is fantastic.

We're using CL in prod for an embedded system (https://evacsound.com/) for some years now, fairly smooth sailing. It started out as an MVP/prototype so implementation was of no concern, then gained enough velocity and market interest that a rewrite was infeasible. We re-train talent on the job instead.

Incidentally I also did an algo trading system in CL at some point (who didn't?). The sibling comment is spot on that outside of deep backend it's a self-reinforcing ecosystem with too much focus on auxiliary stuff if you have to re-implement it.

Did you write that trading program from scratch? Or by chance did you start with:

https://github.com/wzrdsappr/trading-core

I have been considering giving this code base a spin.

(got your email and sent a reply)

I wrote it from scratch.

institutional or retail? i wanted to do this on a retail scale but i don't know where to start as far as APIs for orders and market feeds. do you have any pointers?

Retail.

I do low-freq stat arb and it's modestly profitable. I just use IBKR's API, their snapshot data, and postgres. Nothing fancy.

Feel free to shoot me an email, it's in my profile.

There's a lot of examples of plots made with lisp-stat at https://lisp-stat.dev/docs/examples/plotting/.

Looks really nice, I'll be using this next time I need charts! I had been using a Rust crate which was really hard to use and plots were uglier than this library's.

Impressive release. I love the pragmatic approach of targeting Vega-Lite to get powerful plotting out of the gates.

Can this compete with the momentum/performance of Julia?

I'd be stoked if this became widely adopted but community size seems to be a huge determinant of success with these types of languages.

I migrated from Lisp to Julia for the ecosystem. It hasn't been worth it from my point of view. I'll migrate back to Lisp eventually.

So I can easily imagine packages like this becoming widely adopted /within/ the Lisp community.

Could you elaborate on this a bit ? Lisp has next to no numerical libraries IIRC ?

I think this depends on what part of the statistics universe you're working in.

For example, within Lisp-Stat the statistics routines [1] were written by an econometrician working for the Austrian government (Julia folks might know him - Tamas Papp). It would not be exaggerating to say his job depending on it. These are state of the art, high performance algorithms, equal to anything available in R or Python. So, if you're doing econometrics, or something related, everything you need is already there in the tin.

For machine learning, there's CLML [2], developed by NTT. This is the largest telco in Japan, equivalent to ATT in the USA. As well, there is MGL [3], used to win the Higgs Boson challenge a few years back. Both actively maintained.

For linear algebra, MagicCL was mention elsewhere in the thread. My favourite is MGL-MAT [4], also by the author of MGL. This supports both BLAS and CUBLAS (CUDA for GPUs) for solutions.

Finally, there's the XLISP-STAT archive [5]. Prior to Luke Tierney, the author of XLISP-Stat joining the core R team, XLISP-STAT was the dominate statistical computing platform. There's heaps of stuff in the archive, most at least as good as what's in base R, that could be ported to Lisp-Stat.

Common Lisp is a viable platform for statistics and machine learning. It isn't (yet) quite as well organised as R or Python, but it's all there.

[1] https://github.com/Lisp-Stat/numerical-utilities/blob/master...

[2] https://github.com/mmaul/clml

[3] https://github.com/melisgl/mgl

[4] https://github.com/melisgl/mgl-mat

[5] https://lisp-stat.dev/docs/resources/xlisp/

> Lisp has next to no numerical libraries IIRC ?

https://github.com/CodyReichert/awesome-cl#machine-learning

https://github.com/CodyReichert/awesome-cl#numerical-and-sci...

You can hardly compare any of these one-off projects to something like numpy (let alone something like pytorch etc. which Julia is going after).

what do you mean a one-off project? some of these have corporate/institutional backing but thats not the point. if you know how to implement machine learning models then there should be nothing (except maybe availability of pretrained models) preventing you from using common lisp. if you dont want to dig deep and are happy using just the front end (perfectly acceptable) then python is a much better bet

all the needed ingredients are there to build a custom production class ml solution in common lisp: vectorization, cuda, blas, lapack

Do any of those libraries linked above provide an example of training a resnet or bert on a GPU? I’m not looking for anything fancy, or production quality, just a basic example.

although not necessarily bert or resnet the following probably has all the ingredients for what you are looking for. the author of this library is a research scientist at deepmind since 2015

https://github.com/melisgl/mgl#x-28MGL-BP-3A-40MGL-BP-20MGL-...

There's not even a convolutional layer provided as a building block. This project looks pretty dead to me: most of the code is 8 years old, and the last code-changing commit is over 2 years old. I guess it's okay as a proof of concept for ML with Lisp, but I don't think anyone interested in ML would actually consider using it.

In defense of common lisp: The language is ansi standard and hasn't seen core changes in decades. This means cl pagckages can be "done". It is very common in cl to have projects that haven't been touched in years, because it is highly unlikely to break, hence, "it just works". As for this specific ML conversation, I am not in this field and have nothing valuble to say about who uses what and how viable/appropriate any of the approaches(languages) are for ML.

> his means cl pagckages can be "done".

this is true if there is nothing functional that can be added to a package. however its very much not true for ml frameworks right now. new things are being added all the time in the field. however even in the package i linked you have the necessary ingredients for any deep learning model: cuda and back propagation. the other person mentioned convolution which i think is pretty trivial to implement but still, if you expect everything for you to be ready made then you should probably stick to tf and pytorch. if you want to explore the cutting edge and push the boundaries then i think common lisp is a good tool. as an aside it might also be interesting to note that a common lisp package (Petalisp) is being used for high performance computing by a german university and it has a convolutional layer implemented

https://github.com/marcoheisig/Petalisp

https://github.com/marcoheisig/Petalisp/blob/master/examples...

Thank you for clarifying. I'll have to check out petalisp, I love discovreing new(to me) cl projects.

i work in ml and you asked for an example so i thought a proof of concept answer is sufficient. as i said somewhere else if you are looking for a ready made framework then this is not for you and should probably stick with pytorch, tf, and maybe even pyro. but if you are willing to engineer your own system for optimal performance while avoiding c++ then you would be served good by common lisp. common lisp for ml is suited for software engineers doing ml not so much statisticians

won't answer but a good starting point is https://github.com/CodyReichert/awesome-cl#numerical-and-sci..., did you see it? It has some libraries, for example magicl is developed by and for Quantum companies.

the rally call of lisps is to get a lot done with less, so it is definitely possible that it can compete in some sense. also common lisp packages such as these can cater to the programmer-first type of data scientist by allowing greater (interactive) tinkering with the underlying code

>> cater to the programmer-first type of data scientist

Exactly. Especially those who want to dabble with lisp by playing with familiar problems and applications. I find it much more instructive to play with linear regression code in a new language than with, say, a game development engine, because I have a pretty good idea about the strengths and quirks of `lm` in R.

Am I understanding it right that those plots are only generated in HTML form? I.e. I can't use it to generate an EPS file to be included in a LaTeX document?

I believe the engine vega-lite can create an svg, which is pretty easily converted to eps/pdf etc.

Yes, vega can do that. If you click on the round button with three ellipses (...) you'll see a menu and one of the options is a PNG or SVG download of the plot.

Anyone know if there's a Scheme equivalent to lisp-stat. It looks really good. But I'm much more familiar with Scheme than Lisp.

I also would have thought Scheme to be the natural choice, but I get that "CL" is considered industrial-grade and as many people in the CL community have its advocates.

R is heavily inspired by Scheme, though I'm not sure if it's close enough for your liking.

This is really exciting! I'm a big Lisp-Stat fan, although like everybody else I've been using R for the past twenty years.

No relation. The name was just used because we thought it would be cool to try and revive the spirit of the old company, and the name is long dead.

You guys registered this name in SG ? Uber cool.

(ql:quickload :plot/vega) isn't working for me with the latest quicklisp dist, what am I doing wrong

Quicklisp ships releases once a month (think Debian releases instead of npm), so it is very possible it didn't pick the latest release yet.

Your solution is to clone the repository into ~/quicklisp/local-projects/.

Another one would be to use the Ultralisp distribution, that ships every five minutes. https://ultralisp.org/

(ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)

and now you could quickload plot/vega, except we must ask the author to add it, it takes a couple mouse clicks.

I gave this a try but still ran into issues. First it was a missing file, description.text, so I created a temporary one, but have not been able to figure it out after going through all of the different possibilities on the web site and github.

I am going to keep an eye on this project because I would really like to be able to work on this. Hopefully updates in a month or so will get things in sync.

FIXED.

That's a new file to help with the auto generated API documentation and it's possible one was missed at checkin.

Vindarel has correctly identified the main problem though, Quicklisp hasn't yet picked up the new files.

Hi, sorry for the delay, I wanted to do a clean install of sbcl and quicklisp just to double check. I was having trouble getting the plotting code to work, so I cloned Lisp-Stat/plot and tried to follow the instructions:

* (asdf:clear-source-registry)

* (asdf:load-system :plot/vglt)

The error message was:

debugger invoked on a LOAD-SYSTEM-DEFINITION-ERROR in thread #<THREAD "main thread" RUNNING {1004BF80A3}>: Error while trying to load definition for system plot from pathname /Users/bobochan/common-lisp/plot/plot.asd: READ error during LOAD: The file #P"/Users/bobochan/common-lisp/plot/description.text" does not exist: No such file or directory(in form starting at line: 4, column: 0, position: 147)

The folder contents are:

 ~/common-lisp/plot/ [master] ls -1

CONTRIBUTING.md

docs/

LICENSE

plot.asd

README.md

src/

tests/

'plot/vglt' is from version 1 plotting. Version 2 uses 'plot/vega'. If you're seeing VGLT in the ASDF file it means that it's the old version, and you'll need to pull in the source for a manual install from github until Quicklisp gets updated. The systems you'll need are:

- alexandria+

- data-frame

- dfio

- lisp-stat

- plot

- numerical-utilities

- select

- sqldf

It may be easier to get help from one of the community resources [1] (StackOverflow, github issue/discussion, mailing list) than HN, as those are always monitored.

[1] https://lisp-stat.dev/community/

Great. Thanks so much for following up. This is a great project and I am definitely looking forward to trying it and learning more.

Glad it is not just me. I did a clean install and follow the examples on the web site step by step:

debugger invoked on a QUICKLISP-CLIENT:SYSTEM-NOT-FOUND in thread #<THREAD "main thread" RUNNING {1004BF80A3}>: System "plot/vega" not found

Requiring Chrome is a dealbreaker for me. I prefer statistics without spyware.

That requirement has been relaxed. I use Firefox. I'll update the docs.

Those charts look so clean and crisp!

Please stop using multiple accounts and please don't use HN primarily for promotion—it's against the site guidelines.

We've banned this account and some others. If you stick to one account and submit your own stuff as part of a mix of unrelated things, and don't overdo it, that would be ok.

https://news.ycombinator.com/newsguidelines.html

In case it is not apparent to others, Lisp-Stat (OP of this story) and singaporecode (parent commenter here) are different people.

The account singaporecode and a few other related accounts really look dubious. They are promoting a few websites created by the same author: https://hn.algolia.com/?query=ashok%20khanna&type=all

The comments by singaporecode where they say "I should sponsor the guide’s author" (when the author is himself/herself/themselves) and "I found online" (where they published it themselves) are clearly deceptive self-promotions of https://ashok-khanna.medium.com/ and https://github.com/ashok-khanna but do notice that they are unrelated to Lisp-Stat.

Reply on news.ycombinator.com