(next Rich)
clojure.org
clojure.org
I will also say I used clojure + aleph to write the only significant application I ever released which never had a reported bug or outage. It ran for seven years and that application was under constant heavy load (more than 10k TPS).
Its was one of those things that was always on my mind, but couldn’t express it well to my peers, before encountering this talk. Plus its about both python and java, so increases the chances to be heeded by non-fp people.
A lot of programmers I’ve worked with either don’t want to learn well the environment they are working in, thus repeating stuff that doesn’t fit well, or over-engineer abstractions to force their environment to behave the way they like, and both approaches can leave the codebase in shambles.
1 (the last result) 2 (the result two expressions ago) *3 (the result three expressions ago)
oh yeah. I cant see this giving problems in the future....
It was 2011 and I'd had about 3 years of lisp experience. I got a bit of side eye from people when I told them I was using a relatively new programming language but the fact that it was based on the JVM, which alot of HFT firms were using helped make the case.
We didn't use if for more than a few years before it was retired and rewritten, though that was due to new requirements that included C++ interop.
In the end tracking memory usage and allocations got too hard and if you've ever written something that is time sensitive you'll know just how slow memory allocation is, so you could argue I made a poor choice but for the rewrite was easy to reuse the java libraries when we moved to java.
The harder part was porting the algorithm implementation code. The reason was that I chose a common lisp trick of first writing a DSL in Clojure that was then used to write the algos.
That was the one part that seemed worse in the rewrite as it became far more verbose and clunky.
But man was it fun, I learned more in a month than I often do in a year now.
Thanks Rich!! It's not often you get to experiment with cool tech and make lots of money doing it. I appreciate your work.
Doesn't it mean that Lisp macros should really be avoided at all costs?
The upsides of DSL(Domain specific languages) and macros in general is that they cut down on code duplication and give everyone using them a common language and set of tools on which to build an app. That part is awesome and I think really helped to speed up development of the algos that were built on top of the DSL.
The downside is, that you need to learn the DSL but that's a very small impact in my experience. THe larger downside as you and I point out is that those tools aren't really there for other languages so if you do plan on porting the code base to another language then maybe steer clear of macros.
But again, I wouldn't say I have anywhere near enough experience to claim to be an expert in that.
Clojure yes, java no. Many HFT firms build their systems in java and run on the jvm(specialized hardware aside), its plenty fast if you avoid memory allocations.
It is also possible Clojure is more than up to the task now and its also reasonable to believe that I wasn't a strong enough clojure developer to fix those issues at the time and a more competent developer could have made it work.
But like almost all things if you chose a very out of the mainstream language and start to have issues, most developers will point fingers at the language first.
Side note, its a miracle that Jane Street kept using OCaml after the first few years and points to how strong their core tech team was.
C++ was an external dependency brought on by other systems that needed to be integrated and is a far safer language to use. I'd imagine that if you say HFT to most developers, their first thought would be C++ even if they haven't worked in the domain before.
It's like choosing Julia for your machine learning platform and then switching to python.
A few years ago another commenter noted they had developed a trading system in Lisp and C https://news.ycombinator.com/item?id=25222297 (further comments elaborate a bit) before being asked to rewrite it in Java. What keeps less popular languages alive at companies, when there's no forcing reason to get rid of it like really unacceptable performance, is a conviction (borne out as far as I can tell) that it's ok to hire people who don't know the language -- they'll get up to speed more than fast enough.
Technically it can likely be done but in practice pushing Clojure code for performance (eg, [0]) doesn't follow the usual language idioms and relies on a sophisticated understanding of the interactions between program design, Clojure, the Clojure compiler and the JVM. I'd suggest pointing more at the language than the programmer. In a situation where direct memory management is required Clojure is a weaker choice.
Although in Clojure's defence it leverages immutability to squeeze out some surprising performance benefits without much effort. I would expect Clojure code to be fast by default but challenging to hand-optimise further if that extra level of control is required. Clojure makes it easy to write code that performs well, it is a stronger language for situations where ordinary business logic changes are the productivity bottleneck.
[0] https://blog.redplanetlabs.com/2020/09/02/clojure-faster/
Languages aren't "performant". You can write good or bad code in most languages, or code that isn't a good fit for a language.
That said, there are languages which will make you incur certain penalties, as there is a price to be paid for automatic memory management with garbage collection, for example. These penalties generally do not matter except for edge cases (and HFT trading might very well be one!).
There are also languages which help you write much faster code, but on a higher level. Clojure transducers are a good example: pipelines built of composable transformations, where data sequences are not fully realized between the transformations. These can provide significant performance (and memory allocation) improvements, while still letting you write high-level code.
Just a small nitpick, but allocation is definitely not slow in case of the JVM, it is often faster than manually managed languages. It is a pointer bump only.
All the necessary mechanisms of a GC does have an overhead, so your point stands, but not for the mentioned reason.
Typically in HFT We just want to avoid any full sweep garbage collection between 8am and 6pm
Lisp in general was probably the most fulfilling programming rabbit hole I ever followed down, and I don't regret a moment of it. A true gold mine of skill, ability, and thought training that changed the way I program, even in imperative/procedural languages.
Having only had the good fortune to meet you in person a few times:
Congratulations, Rich. You have my deepest gratitude. It fills me with great joy to know you are taking this step. I look forward to what your next stage brings.
Side question: Rich seems to use etymology as a tool for original thinking and clear explanations. Are there other people who do this as well as he does?
e.g. https://www.etymonline.com/word/system - see how it's about composition, not so much about ways of doing things.
But for the sake of argument, I'm saying "complect" is neither simple *nor* easy. It's obviously not easy since it's unfamiliar. But nor is it simple, or to be really precise, no simpler than alternatives, since it doesn't conceptually improve on plain words and phrases like "complicate", "intertwine", "mix up", "separate", "tease apart", etc. It adds nothing, really.
Your comment kind of highlights some of the errors many take away from "Simple Made Easy": that ease is in opposition to simplicity and/or that they are completely orthogonal.
One of the funniest Conj talks ever.
Perhaps you've never listened to a Rich Hickey talk before but he's notorious for riffing on the meaning of words.
Anyways thanks for the explanation. I wasn't in on the meme
While it has gone through the revolutions from sincere, to sarcastic in intent, to back again, I believe the inherently joyful attitude underpinning the original video carries the buoyishly optimism of sincere joy.
If it hadn't been for Rich Hickey, or Clojure, I'm not sure I'd have the slightest inkling of how joyful and creative it can be to write software. It was an A+ experience, even if I'd be swiftly fired for trying to push logic like that into production.
https://cognitect.com/blog/2020/07/23/Cognitect-Joins-Nubank
https://news.ycombinator.com/item?id=23926407
https://building.nubank.com.br/welcoming-cognitect-nubank/
Very interesting story!
edit: not one year ago at all
(edit: typos, formatting)
https://www.youtube.com/watch?v=dGVqrGmwOAw
At that time I had been working with Java for years, both the conciseness of the code and the interactivity were flat-out amazing to me. It's saved me time and made me 100 times more productive.
Congratulations to Rich, I am also extremely grateful for his work!
I reached out to him to come to our meeting because I met him when I took his Advanced C++ class at NYU continuing education in the 90’s. The class culminated in some advanced usage of his C++ functor library.
Rich, I'll always be grateful to you for creating a language and community that got to the root of so many problems our industry still faces today. I'm confident it will continue to change the world as it has my own life.
I didn't end up sticking with Clojure because I don't enjoy the JVM tools/ecosystem, but Rich's talks gave me a lot to think about and I've been spending time in more FP and FP adjacent languages like Common Lisp and F# because of his thoughts and ideas.
Thanks for everything and enjoy your retirement from professional development!
TFA: I look forward to continuing to lead ongoing work maintaining and enhancing Clojure with Alex, Stu, Fogus, and many others, as an independent developer once again... Retirement returns me to the freedom and independence I had when originally developing Clojure. The journey continues!
Some days later I started learning Emacs, Clojure, et al. and I was/am definitely the better for it.
Congratulations, Rich.
[0] https://download.clojure.org/papers/clojure-hopl-iv-final.pd...
I attended your presentation on Clojure at the International Lisp Conference at MIT in 2009, where we were celebrating the fiftieth anniversary of Lisp. After your talk, you were surrounded by admiring Lispers, including several of the elder statesmen of the community. These were people who, to put it mildly, were not known for being quick with praise. But one after another, they were saying things like "This is the future of Lisp!"
By the way, I love the title. It reminds me of what I wrote when I started my last job:
(call-with-current-continuation employer)While, as is to be expected, I don’t agree with every decision made for Clojure, I find it to be an entirely pleasant Lisp to work with.
So, kudos to Mr. Hickey, and thanks for sneaking Lisp back into corporate America!
[1] - https://changelog.com/posts/rich-hickeys-greatest-hits
I’ve been enjoying the fruits of your labors for more than a decade, and I have since introduced my daughter (a budding computer science student) to “simple made easy” and a bunch of your other talks.
Thanks for everything you’ve contributed to the software world. It’s a happier place to be as a result of your efforts.
Enjoy your retirement!
The impact of Clojure cannot be overstated. It's amazing to see how far it has come since its inception. You've inspired so many developers, including myself, to explore functional programming and think differently about software design.
As a schemer there are so many small things I would like to change that put me off using it for my personal projects, but regardless I have really enjoyed contributing to other people's Clojure projects. It is not far away from what I would consider a good starting position for my own perfect language.
The company I founded uses full stack Clojure for our product development (https://kpow.io).
We're a small team pushing through a big product roadmap at pace, love programming every day, no chance we'd have made it in Java (and I quite like Java).
Thanks Rich (and Stu, and Alex, and Fogus, and David, and..)!
although he’s no longer working on datomic, i really hope cognitect values growing it. it feels like such an improvement over sql to work with.
1. I stopped thinking about OOP, classes, “patterns”, frameworks, and mostly think about what data flows through the program, how it’s (they’re?) transformed and stored. It’s a much clearer view of a system, I think.
2. I started being very conservative about mutable state. I keep mutable parts few and mostly on the top level, and compose program logic from practically pure functions. It helps with debugging issues immensely, and fewer issues arise because pieces can’t interact unpredictably.
3. I stopped caring about types/shape of data inside modules, and spec the hell out of it on module boundaries. IMO static typing is too strict for the former use case, and too loose for the latter. It allows more flexibility in implementation without actual breaking changes.
4. I now enjoy developing in REPL, updating a running program, adding features in real time. It feels so natural.
It's a bit distasteful to virtually follow him around and speculate about his mental health.
It was contentious, but I'm glad we dumped clojure.
No matter how good a project is, it needs loads of support behind it
But, I've always been turned off to Clojure by the lack of static typing (I know spec exists, but it's not really the same thing). I did a little project once to try it out, and even there I found myself spending time debugging basic issues like passing the wrong number or wrong kinds of arguments to functions
Can somebody sell me on it? I feel like I'm the target audience
Also, in case of Clojure you usually handle FP code with no side effects, written in a REPL-driven, interactive style. This combination is not as prone to typing errors in my opinion than "traditional" dynamic languages like Python and JS.
Another aspect might be the one being touched in the 'Maybe Not' talk: Static types operate on a closed-world basis, where the sum of all relevant code is available at the same time. While this is common for certain kinds of programs, it is not really true of systems which have to interoperate with services owned by different companies alltogether, having to support many different versions of other interfaces at the same time. In that case you want to be liberal in what you accept, and arguably, dynamic types are better fit for this job. I am not 100% sold on this part, mostly just reiterating what I've seen others claim, notably Oilshell's author is a big proponent of this thought.
I'm not sure how being liberal in what you accept is at odds with static typing. Static typing is about encoding the rules and shapes of the data that your program *knows* how to work on. It's not preventing some other data that you don't care from existing alongside.
Even if I'm missing something, that doesn't change the fact that in most static typed languages you can fallback to some sort of "any" types. Seems to me it's better to have option to use static or dynamic types in the same program, instead of being forced to do *everything* in dynamically typed way.
Some advice I would give myself when I started learning Clojure:
Think not in types, but in data. It's a different model that requires an 'aha' moment, but it's revelatory when you do get it.
If you come from an OO language, think not in objects (Elephant), but in information about objects (elephant's record, a map).
Clojure is best served REPL driven.
You can technically compile and execute, but this is not how the language shows its power. Here's how I use it:
I evaluate code in the REPL as I write it, reload definition on every change. Inside a `comment` block, I call the function I'm writing with the required arguments. I can quickly test how my function behaves in corner-cases. You then construct another function, which calls the first. You evaluate/execute the function as you write it.
So your program keeps growing, while you keep running/changing individual parts of it.
Once you get the hang of immutability and working with maps/collections, this process becomes very engaging because of the short feedback loop between thought and result.
It’s like having a conversation with your program while you are developing it.
You'll get over that pretty fast. The bigger issue is looking at some code and wondering "just what in God's name is in that 10-level map and why?"
“Simple made easy” was a classic even if everyone proceeded to ignore the practical advice contained therein. And “Maybe not” is my personal favorite, a great discussion of requirements/provisions and the downsides of option types.
For those who haven't seen it, in the presentation Rich talks about figuring out the problem before you work on a solution. It's definitely something that could be improved in myself and in a lot of people and organizations involved in software development.
For example his note about "Categorical descriptions" and then the explanation about maps/and spec at 25:00[1] seems to indicate he's unfamiliar with the difference between extensional and intensional type theory.
The core difference between the two is extensional type systems decide equality on the observable behavior of the output. For example if two functions take the same input, and produce the same output extensional type systems/theories say they are the same. The problem with this is type equality is then undecidable.
Extensional type systems also struggle because there are more than one way things might be equal! For example two string might be equal if you ignore case, but unequal if you don't. So if we want our type system to check if two functions are equal under case insensitivity an extensional type system will struggle with this.[2][3]
Now in fairness to Rich, these concepts come more from the math community, especially the notion of types being equal under different paths, and in the case of Homotopy type theory are burred in unapproachable language and concepts... even by mathematicians standards.
He also talks about select from spec, and it not preventing you from passing types broader types, and the necessity of specifying deeper type.[4] But, that's just an eliminator defined on the type[5].
Now most strongly typed programming languages suffer from the issue he identifies when talking about not making brittle systems [6] and the issues you get with coupling around taking a map/struct/product type C = A X B (e.g a map C = {A: something, B: something}), and you require A so you're function should be good but you fail to compile because the function says A, and now you're passing C.
It's frankly pretty annoying in places like Java that if you have a Point = Int x Int y you can't use it in functions which take a Tuple = Int x Int y. I'd actually say that's the common observation that both Rich, the Go Inventors, and the TypeScript folks have all made. You need to care about the extensional type of your input (and output) fairly often. And carring around the intensional type causes both dependency issues and just some general PITA's.
[1] https://www.youtube.com/watch?v=YR5WdGrpoug&t=1504s [2] https://math.stackexchange.com/questions/4486995/what-is-an-... [3] https://en.wikipedia.org/wiki/Intuitionistic_type_theory#Ext... [4] https://youtu.be/YR5WdGrpoug?t=2668 [5] https://www.quora.com/In-type-theory-what-is-an-eliminator-a... [6] https://youtu.be/YR5WdGrpoug?t=2823
A true computing thought leader. All the best Rich. My only complaint about you was not open-sourcing datomic and making it free for developers.
- https://blog.datomic.com/2023/04/datomic-is-free.html - https://blog.datomic.com/2023/06/datomic-cloud-is-free.html