Or is there many things to consider at once in the system instead of many individual things?
I do think that a static analysis tool would be of some help. Some sort of tooling to better handle tree structures would be very handy. I often find myself being off-by-one level with get-in calls on tree data (E.g. (get-in m [:a :b :d]) where m is {:a {:b {:c {:d 1}}}}), which is annoying because the NPE gets thrown 3 function calls up the stack.
Did you have prior experience with dynamic languages? If so, how much?
Did you have prior experience with statically typed languages? If so, how much?
I'm trying to see if Clojure requires more fundamental programming intuition and experience to sustain in large projects. I work with tens of thousands of Clojure LOC, I don't feel these issues and can't relate, and love Clojure, so I'm curious to understand what context it best applies too. I wouldn't want to force it on a team that wouldn't benefit from it, so I'm interested about learning these aspects, so I'm able to recognize in what context it would make sense for me to influence a team to adopt it or not.
Thanks.
And yet... I agree with GP. I love clojure, but over time I’m becoming less and less sold as dynamic typing. Spec helps, a little. Property-based generative tests (especially when used with spec) helps a little too. Neither are a replacement for proper static types, though, especially an ML-esque type system with type inference. Bonus points if spec validation could pass type data to the compiler/type inference (eg the code path after validation can assume that the data is of the the types described in the spec).
I dream of a statically typed Clojure with type inference, spec-inferred types and optional dynamic typing for REPL experimentation and glue code.
Could I ask what kind of project it is? I wonder if I'm just lucky that Clojure fits perfectly my use case, which is mostly a set of distributed systems of all kinds. So while as a whole there's tens of thousands of LOC. The components have very strong boundaries being as it's a set of services assembled together through RPC, PubSubs and DBs. Maybe that alleviate the lack of a static type checker.
We've adopted Clojure about 3 years ago, team of 10. We've had a few people leave and join throughout. Only one person knew Clojure beforehand. Our stack is about 50% Clojure, 40% Java and 10% Scala. Of all three, Clojure has given us the least issues, has been pretty easy to maintain and generally has fewer defects. Java tend to have the most bugs, almost always related to some shared state. Scala I find the hardest to extend and maintain, but the code base we have for it I think does Scala wrong, it's like the worst mix of OOP and FP.
> I dream of a statically typed Clojure with type inference, spec-inferred types and optional dynamic typing for REPL experimentation and glue code
That's pretty much exactly core.typed: https://blog.ambrosebs.com/2018/09/20/towards-typed-clj.html
That said, the project never managed to get more contributors.
If you're looking for a typed Lisp, I've been keeping my eyes out on Carp: https://github.com/carp-lang/Carp
Don’t get me wrong, if I restarted the project again from scratch, I’d choose the same setup (or a very similar one). I love Clojure and am very productive in it, but that doesn’t mean I don’t think it could be better still.
> That's pretty much exactly core.typed
I haven’t looked at it in a couple of years, maybe its changed, but when I did, it didn’t really do it for me. It’s still a separate tool that lives separate from Clojure itself and it felt very “heavy”. In my personal opinion and experience, having certain things as separate entities (decomplected as Rich would say) isn’t always a good thing and leads to an inferior thing. I’ve played around with many programming languages in my time (I’m a bit of an enthusiast, I guess. I like trying out languages that are very different from what I already know – that’s how I originally got into Clojure) and it seems like a common theme. The closer a feature is to the compiler/runtime, the better it works and seamless it is over all. Another Clojure example is the limitations core.async has, because its an external library: things like <! cannot be placed inside functions or the go macro can’t see it to transform it as macros cannot look inside function calls. I’ve also encountered an exception recently where the stacktrace only showed core.async and clojure.core code, not a single stack frame referenced MY source files. These problems are hard to solve as an external library.
> If you’re looking for a typed Lisp
I’m not. I like Clojure’s particular mix of sensible syntax, immutability, sequence abstraction and general way of doing things. Other Lisps I’ve looked at don’t have the same emphasis on these things as Clojure does, so I don’t want another Lisp, I want a language that makes the exact same decisions and tradeoffs as Clojure, except on dynamic vs static types (and actually useful error messages). Maybe one day I’ll give it a try, I certainly don’t expect Cognitect to change their language because of what my preferences are.
The issue is how would type definitions be introduced, and what kind of types would be most appropriate? As you said yourself, the way core.typed did it felt too "heavy". Yet it isn't clear how to make a more lightweight variant for a Lisp language such as Clojure without rendering the types worthless.
The first, and one of the biggest issue in my mind, is that what everyone loves about Clojure is the data-oriented style. In that style, you represents entities and their relationships using heterogeneous collections. That's where in Clojure you model your domain with Maps, Lists, Vectors, Sets, etc. It is awesome, but no one has figured out a non "heavy" way to statically type it. All methods I know of have bad programmer ergonomics. In effect, adding types back to it almost kills the data-oriented style, and it ends up feeling a lot like modeling with Classes instead. See Haskell's wiki section on this problem: https://wiki.haskell.org/Heterogenous_collections they haven't solved it, and have multiple ways to possibly handle the scenario, and non are ideal.
> I want a language that makes the exact same decisions and tradeoffs as Clojure, except on dynamic vs static types
I would too, but with the caveat that the development experience would be the same, and the programming ergonomics and styles would be retained. And this, I'm afraid, is an open problem that no one has solved yet. It's not just a case of personal preference. Having a language which has the pros of Clojure and the pros of static types, without the cons of static types is hard. That's why for now, you need to choose one or the other.
In Erlang/Elixir, you have patterns, and interfaces which are fairly low level, but provide meaningful abstractions over common goals... I.e. we have OTP. OTP is fairly simple, and that doesn't mean it's easy, it's really not, but it can be simple once you're familiar with it.
Rust, on the other hand, is easy to get started with, the core of the language is fairly small (and relatively simple) but that's where the complexity starts to creep in.
It's so small, I have to constantly reach for a crate for common goals... There are tons, and they're fucking awesome, BUT they're easy, not simple, and my application's complexity grows exponentially relative to my familiarity each time I grab another crate.
I think it's easy ;) to say the Rust ecosystem has to contain the complexity it does, and making it easy is the best we can do, but with great challenges comes great reward... Batteries not included, yes, but at least you could know the size (spec) required for the ones you need? That's probably a shitty metaphor but I tried...
...
One more thought on this tirade, when Rich Hickey casually says that engineers should be working to remove complexity from the business... That hit me right in the feels. Too often do we get complex business decisions/goals and just accept them. We need to push back and help them simplify their goals... too often I've gotten some crazy request, where instead of implementing it, I pushed back to get to the root of the problem and ended up with something wildly simpler and better for both parties. When we push back, it shouldn't be to make something easy, it should be to make something simple. You can scale simplicity almost infinitely, but complexity will eventually come crashing down.
More on topic... Thanks for posting this talk, it and the article have inspired me, I'm gonna go try to write a bit of Clojure today. You should write some Elixir, or maybe LFE (Lisp Flavored Erlang), I think you'd quite like BEAM :)
At my last job the client reps were notorious for saying “yes” to every client request and other devs were notorious for implementing those requests without question. I really tried hard to push back and find out what the client was trying to achieve. So many times after a brief discussion I would be able to inform them that what they wanted was already possible or doable with way less effort and expense. Sure, we didn’t get to bill them as much, but I’d much rather solve their pain points without adding code (especially when the code has already been written) than needlessly charging them.
Would love to hear why? What is that make Clojure such a good experience for you?
What's made clojure so great, imo, is its unicorn status as a principled-yet-practical language. That "principled" part is not worthless---it means that a lot of great minds are drawn to it. Before react took over the world, clojure folks were already taking steps in that direction.
A lot of other things. The "sequence" as a core abstraction is very powerful. Immutable data structures by default make functional programming perfomant and efficient.
Speaking of data structures, data structure literals have spoiled me for other languages. After Java, especially.
And none of this mentions clojure's lispiness. The ease of metaprogramming has allowed the community to build some of the best tooling out there, between CIDER for emacs, and Figwheel for the browser---oh, did I forget to mention clojurescript? Being able to reuse code on the front and backend is great for web applications.
Clojure isn't "everything." I think I'd still benefit from learning Haskell, APL, and Forth, and I wouldn't mind knowing Ruby and js a bit better. And I'll probably be dragged back to Python and R if I keep doing maths.
But if someone asked me which single language would probably do the most for them professionally, I'd say clojure. It's the language people start startups so they can use it.
There are downsides, however. The UIs that it creates definitely look like Swing apps. I.e. ugly at least on Linux. Also, the up to date documentation is difficult to find, I'll like to it here when I get home. The GitHub-linked docs work fine, but they do miss a few features.
Check out the examples/mini-tutorial: https://github.com/cljfx/cljfx
It's all React-like. You have a state atom and a GUI map data structure (which I think is equivalent to your DOM in React..) and then you hook up events that update the state and blah blah. It's all very clean and easy to read/use. The underlying JavaFX is also great for an OO GUI library so it's not gross to dive into if you need it
- Immutable data-structures with concise literals for lists, vectors, maps, and sets. Having pure functions and immutable data-structures makes code easier to reason about, easier to test, and thread-safe.
(But clojure doesn't "force" you to be pure. The idea is that you write as much of your code in pure functions as you can, and push the IO and impure parts to the extremities. It's very pragmatic in this way, and you can get real work done.)
- Simple syntax:
While the syntax may be intimidating at first and arithmetic looks weird to untrained eye, once you realize that everything is a function (or a special form that also looks just like a function), the very light syntax and consistency feels amazing/refreshing.
- Macro system:
This is tied to the previous point. Since all your code is technically a "list", and you have something akin to a pre-processor where you have the full library of clojure functions to manipulate data. But in this pre-processing phase, your data is the code. (the code is a list). Now you can dynamically re-arrange or re-write code. Lookup homoiconicty and learn about the kinds of things you can do in macros that can't be done in other languages.
- Capabilitis for general programming, abstractions, code re-use etc.
- Performance is quite good. It's often nearly as fast as java, yet your codebase might be 10x smaller because the language is so expressive.
- Concise/expressiveness. Clojure comes with a nice built-in library with generic functions that you re-use again and again. Give a programmer a few dozen of these functions and it's amazing what can be composed to solve many problems succinctly and elegantly.
- There are surely other benefits, but the last one I'll leave with is hard to explain unless you have felt it before. It's REPL driven development.
Clojure comes with some seriously awesome REPLs (i.e. a shell for interactive tinkering with the language). Other languages may have some form of REPL, but no other language in my experience has come close to the feel you get in a clojure REPL. I can best explain it as a freedom of very light-weight experimentation that you use to write your code. It's a playground for writing functions with very quick feedback to see if your code will work or not. One factor that makes the clojure repl experience so nice ties back again to the succinctness and expressiveness of the language. Typing commands in the repl is painless because it's concise, not a lot to type, and then the feedback is so instant.
Somehow when I write code in python, java, javascript, or other languages, I just don't use their REPL or shell as often. It's just not quite the same as the clojure repl experience.
Take the following clojure:
(+ 1 2)
=> 3
It means if we do this
(1 + 2)
However, we can tell Clojure to treat this code as data by quoting the list using the '
'(1 + 2)
=> (1 + 2)
WHat if we write a function to swap the first two items in a list?
(defn swap [x]
(list (second x) (first x) (last x)))
(swap '(1 + 2))
=> (+ 1 2)
So you can see
(+ 1 2)
We can eval it:
(eval (swap '(1 + 2)))
=> 3
Up steps macros, macros are evaluated before compile time and don't evaluate their arguments. So we can rewrite swap as a macro
(defmacro swap [x]
(list (second x) (first x) (last x)))
(swap (1 + 2))
=> 3
At that point you're emulating lisp without the elegance, and the first approach is only possible because functions are first class objects. If you were trying to rearrange an expression that had control flow or keywords in it (eg. Modify the behaviour of an if) then you would have to reify the "if" (change the original code so it had a class to represent the if not use the keyword). So you're kind of reinventing lisp by wrapping every part of your language as an object
(defun swap (x y) (y x))
Then you can call it like:
(swap (2 3)) => (3 2)
You could do `(defn swap [the-list] ((second the-list) (first the-list)))` (which would invoke the second item as a function on the first item). It comes down to is the list a literal list as a parameter to something or is the first item a function to be invoked.
(defn swap [x]
(list (second x) (first x) (last x)))
(swap (1 + 2))
=>Exception! 1 isn't a function.
So we quoted it in the function call by putting ' in front of the list '(1 + 2):
(swap '(1 + 2))
=> (+ 1 2)
So if I was to
(+ (swap '(1 + 2)) (swap '(3 + 4)))
=> Crashes! Can't convert alist to a number.
(+ '(+ 1 2) '(+ 3 4))
(+ (swap (1 + 2)) (swap (3 + 4)))
=> 10
(swap (1 + 2))
(+ 1 2)
The main point of macros is when you use a regular function in Clojure you have applicative order evaluation. Macros do not, as macros are designed to transform and generate code.
A better example would have been the (when) macro.
In Clojure you have (cond) and (if) for conditional evaluations. If has the form
(if (cond)
(when-true)
(when-false))
(if (= 1 1)
(println "1 = 1")
(println "uh-oh the unviverse is broken!"))
=> "1 = 1"
(if (= 1 1)
(println "1 = 1")
(println "also hello"))
(if (= 1 1)
(do
(println "1=1")
(println "Also hello")))
1=1
Also hello
(when (= 1 1)
(println "1=1")
(println "Also hello"))
"1=1"
"Also hello"
Essentially, we want to write
(when (= 1 1)
(println "1=1")
(println "Also hello"))
(if (= 1 1)
(do
(println "1=1")
(println "Also hello")))
This is incredibly easy in Clojure. We just need to create a list where the first item in the list is `if`, the second item is the conditional we pass to the macro, the 3rd item is a sublist, of which its first item is `do` followed by the list / functions to execute when our test evaluates to true!
(defmacro when [test & body]
(list 'if test (cons 'do body)))
If we run the macro expansion on when we can see the code that it generates:
(macroexpand '(when (= 1 1)
(println "1=1")
(println "Also hello")))
=> (if (= 1 1) (do (println "1=1") (println "Also hello")))
Say we have the nested function calls:
(reduce + (filter even? (range 1 11)))
=> 30
(->> (range 1 11)
(filter even?)
(reduce +))
(defmacro ->>
[x & forms]
(loop [x x, forms forms]
(if forms
(let [form (first forms)
threaded (if (seq? form)
(with-meta `(~(first form) ~@(next form) ~x) (meta form))
(list form x))]
(recur threaded (next forms)))
x)))
Basically, using macros you do things like control symbolic resolution time, extend the compiler to create a DSL spcific to your domain and reduce boiler plate code.
That's before you start getting into properly weird stuff like anaphoric macros.
I've read a few Lisp books and dozens of internet blog posts, so I know about macros and why people use them without getting the full understanding which comes with actually writing code.
Macro arguments are not evaluated, and so this works.
You could make it a function, and pass the arguments quoted, but it'd be more cumbersome.
cljs.user=> (defmacro swap [x] (list (second x) (first x) (last x)))
#'cljs.user/swap
cljs.user=> (swap (1 + 2))
Execution error (Error) at (<cljs repl>:1).
1.call is not a function swap = (x,y,...rest) => [y,x,...rest]Macros are actually my least favourite part of Clojure. Sure, its great to have them and there are some libraries that use them to excellent effect (instaparse, Hugsql, etc), but most of the time, I prefer tools that don't use macros. You can see by comparing libraries that were made in the earlier days of Clojure versus more recent ones: the earlier ones love to use macros while the newer ones prefer functions and data. The data-first libraries are, in my opinion, easier to test and easier to build complex things on top of. If I look at a readme and see that the expected way to interact with a library is a macro, I usually look for a data-first library instead and only use the macro one if I can't find an alternative.
Macros are technically cool, you can do some interesting things with them, but I find that in most cases where they're used, they are inferior to alternatives. Its technically very cool that core.async could be implemented as macros, but I feel that core.async greatly suffers from it versus being built into the runtime: you cannot use many operations in functions called by core.async/go because the macro can't see inside function calls. Also, I've had exceptions thrown by core.async where the stack trace did not mention any of my code. That was not fun to debug.
Don't get me wrong, macros do have their place: hugsql uses macros to parse the SQL file and generate functions for you, which is awesome! But for every great macro-based library, there are many more that I wish didn't use macros.
(This is user-facing macros in a libraries' API. I have no problem with using macros internally)
I do love Clojure overall, though.
When I meet a girl, I ask her about macros and functions first.
I except to hear something like: macro < function < data.
But having that option is better than not having the option at all IMHO.
Then, when I learned about Elixir's macro system, I thought, "wait a minute, this is LISP! This is just a LISP in Ruby's clothing in Erlang's pasture!"
That was a delight. :) My heart still belongs to the LISP family, but right now I'm working as an Elixir developer.