Clojure, the Good Parts
rasterize.io
rasterize.io
So I also get a bit twitchy when I hear for wholesale use of Timbre, Component, Schema. Every one of these libraries has tradeoffs. And those tradeoffs should be understood before adopting them. As an example, if you have a need for Schema, perhaps your data model needs some refining. Maybe you need Schema at the edges of your API, but you should define why you need a library before you just adopt it wholesale.
Likewise with Timbre, often we need tight integration with Java tools, so perhaps my project really does need a "native" java logging library.
Perhaps "understand all the trade-offs" should be the mantra of programming, in any language.
But the OP implicitly assumed everyone was writing "production" software, assumably as part of a commercial service offering.
To make my biases explicit, I mostly write webapps and data analysis on servers in the cloud. If you use Clojure in significantly different applications, these recommendations might not apply to you.
Yes, yes, a thousand times yes! (Perhaps too close to James Joyce, but that's not the point.) Understanding cost/benefit and making good decisions is the basis of winning a lot of games, the game of programming included! A programming shop needs to be good at the task of optimizing its programming, obviously, and as in any optimization task, cost/benefit comes into play.
Global atoms provide a good way to confine state into a single place. In my applications they serve as a in-memory database while everything persistent is written to disk in the form of a log (kafka, file append, you name it).
Agents provide a good way to serialise events akin to a transactor. Not every problem requires the parallelism or the concurrency gained by smearing it out into a nondeterministic pile of communicating processes that is a nightmare to test. In those cases having determinism and centralised state is a godsend. I see them as the atoms bigger brother.
In cases where you need some concurrency or parallelism futures are fine. They provide a simpler interface and can even be canceled as first class citizens, unlike CSProcesses which potentially linger around indefinitely without having a first class handle to kill them or see their status.
Core.async is heavily overused. If your application doesn't contain a multitude of moving parts that all need to execute concurrently I wouldn't touch it with a ten feet pole. The go macro is a clojure to continuation passing style compiler, and don't get me wrong it's a marvel of engineering, but it's also a sausage machine that makes code impossible to debug. The fact that processes aren't first class citizens makes it impossible to build erlang style resilient systems, and creates tons of zombie processes when using figwheel. People immediately grab it when building cljs applications even though a log/queue would give them much more reliable performance guarantees, much better debug-ability and better tool compatibility. (Re-Frame for example ditched core.async for their own queue implementation.) (If somebody is sharing that sentiment and looking for a kafka style log for cljs, I wrote one and it's been used in production for a while :) https://gitlab.com/j-pb/franz)
Last but not least, transducers make it onto my list of things to avoid. I'm not sure if they or core.async are clojures million dollar mistake. They are incredibly complex, even brought the spawn of the devil, volatile, into the language. Just for a bit of speedup and core.async integration.
If you have low contention, use an atom. If you have high contention, use atoms and/or pieces of java.util.concurrent.
Btw I think your libraries did the job core.async tries to do a lot better :) and to this date I can't figure out why they didn't received the same amount of attention, so thanks for those!
I think it's simpler to pass state as an argument to your functions instead. That's less complex and easier to test.
> Last but not least, transducers make it onto my list of things to avoid... They are incredibly complex, even brought the spawn of the devil, volatile, into the language. Just for a bit of speedup and core.async integration.
Transducers are not just for performance and core.async. They're a hugely useful tool that allows the various collection functions to be applied to sources that are not easily seq-able.
For instance, say I have some source of events, and I can register a callback to receive events. With transducers I can still use all the standard collection functions, even though there isn't an obvious collection to use them on.
James,
I was wondering what is the best course of action ini one particular situation. I am using your excellent libraries to create an HTTP API that needs to have state. I usually have an init function and using the ring init feature (see below). How would you not have an atom that hold lets say the DB connection info for a function that renders a html page displaying the version of the database, accessing the dbinfo with @dbinfo (that was updated at the init). Would it be possible to pass in to this function the state? Is it any different if I am calling @dbinfo in the actual function that renders the page or the calling function (router, or app definition)?
:ring {:init myproject.core/init :destroy myproject.core/destroy :handler myproject.core/handler}
I completely agree with that sentiment. But I prefer to execute those functions with swap! on my application state.
That way I can always inspect it in the repl during development and production. Which is not so easy when it's just bound in the local variable of some thread somewhere.
As for transducers, tbh I haven't encountered the use case you describe yet.
[0] https://github.com/divs1210/difference-engine/blob/master/sr...
I've recently moved from Component to Mount. It is less powerful but just a vast syntactic overhead reduction.
Absolutely avoid metadata. The worst thing ever is using it like how Reagent does in ClojureScript where it actually provides important program semantics. This is a Terrifically Bad Idea.
Schema is wonderful. Schema is terrible. It is nothing resembling a replacement for static types. The more you use it the more obviously it is deficient for such things.
Absolutely just use Clojure.Test. I don't even understand why this is a question. What is up with cutesy testing APIs?
In Clojure, I don't recommend it because it's Just Another Place To Stick State. It's cool, but provides little semantic value over compare-and-set! on an atom.
It is pretty much as like this
1. Dereference a variable and keep the old reference.
2. Create a new modified variable.
3. Use CAS on old reference and new variable
to update the reference to point to the new vairable.
In case of failure (some other thread made the update before) go to 1.
It has the same drawbacks and advantages as atomic references: your code does not depend on lock taking order (as they are managed by the transaction manager) and you avoid dead locks as the locks are taken in the same order before applying the update, but now two different threads can start the transaction to update the same variable, but only one of them can finish as the first one and the other one should be cancelled and repeated.
In principle you can get away with a simple atomic reference (that is much faster) when the set of the updated variables is always the same - you make a pair of them and update them together.
If your compare-and-set returns false, what do you do?
STM resolves these issues by letting you coordinate updates to more granular, finer-scoped refs
It actually does not have to be, but this is how it is (was) implemented. I built a simple STM (in Java) as part of my bachelor thesis and its performance was comparable to the atomic reference.
I'm not convinced a single atomic reference sufficiently covers enough use cases, however.
Can you compare and set on multiple atoms at the same time? If not, how do you make updates atomic without using either STM, a global lock, or re-implementing STM yourself with an incremental locking and rollback system.
http://chimera.labs.oreilly.com/books/1230000000929/ch10.htm... http://book.realworldhaskell.org/read/software-transactional...
Actually, this is not recommendation for a specific use case - a (web based) database application.
For applications that require to manage state internally it might be still way to go (sorry, I am not a Clojure user).
STM in a simple request->response web application probably is overkill and I'm much more aligned with his recommendation now.
I don't think metadata is a bad idea. It's data about data. As long as you don't assume something will have the same metadata as the data it's derived from, you should be fine. I don't necessarily like the way Reagent uses it, but it's not out of character for Clojure to use metadata semantically.
Eh? Reagent does not require any metadata, ever. Much less metadata what provides "important program semantics". So I'm pretty puzzled by your comment. Either you have never used Reagent, or our definitions of metadata is really different, which would be odd, given ClojureScript is pretty clear on what metadata is and isn't.
Optionally, as a convenience, Reagent allows "key" to be provided as metadata but that's really a React thing (and available the same way in OM) and that's because it allows React to supply better performance in some cases, but even then you can supply key via means other than metadata if somehow that offends you. If this is somehow "the worst thing ever" I'd suggest you have lived an overly sheltered life :-)
I am no expert in reagent, but I think it requires it to access Reacts component lifecycle callbacks like `component-did-mount` and such.
(r/create-class
{:reagent-render render-fn
:component-did-mount (fn [] (js/alert "hi"))})
Given that there is a supported and suggested pathway to achieving core functionality of the library which involves the use of metadata to define the semantics of your components I stand by my statement. I'm happy to roll back on "the worst thing ever" if a bit of exaggeration is intolerable, but the fact that this exists in the API is a huge wart even given the fact that it's not required.
Weavejester below suggests that use of metadata like this isn't even such a big deal for Clojure, but I cannot agree. In a language with primarily immutable data structures it's just silly to trust metadata as semantics-bearing given that it's invisible, ubiquitous, and often discarded.
I do really like it on vars, though.
Sorry, but you are just completely wrong. Have you ever used Reagent? Reagent doesn't require the use of metadata AT ALL, EVER. Much less in some fundamental way, as you claim.
* before you decide to use Component, take a look at Mount and see which one suits you better (I settled on mount for both Clojure and ClojureScript and I'm happy so far),
* use Timbre: yes. Definitely. In fact, pretty much anything by Peter Taoussanis will be a good choice (for example, Sente is a wonderful tool for writing client+server apps). If you use Timbre in a mixed Clojure+ClojureScript application, consider sending logging from client-side to server side via Sente. I did and it's fantastic to be able to log something in ClojureScript code and have it appear in your server logs. Just don't overdo it, or you'll be flooded with logs.
* avoid lein plugins: yes, in general. But I still use one, lein-git-version, which names output artifacts using `git describe --tags` and places a version.txt file in resources/. This is difficult to do in any other way.
But we do take a tip or two from stuartsierra's "Reloadable" work flow (which he based Component on), specifically that there should be no global state, and all dependencies should be passed into functions that need them as a parameter.
In our case, we call it `env` since it's the "environment" that a function runs in or can affect, and contains things like the database connection, email service, etc.
Some of these are records which implement a protocol, so that you can have e.g. LiveEmailService in production and MemoryEmailService in development or while running tests.
All in all, I like and have used some of Stuart's ideas, but not necessarily Component itself. There's strong mindshare for it in the Clojure community right now, and it's tempting to look at it as The Solution™ for almost everyone's use-cases because of that. And the ideas it has are good, but that doesn't always mean it's the best solution for each person.
If not, can you elaborate?
obviously this data has ways to manipulate the world, because an app that did nothing would be pointless. but it's no different than any other param.
what would be bad, which is pretty much the opposite of this situation, is if you had global resources that werent passed in as params and caused side affects and you had to access a global variable to get to it (because it wasn't passed in).
https://gist.github.com/emidln/8f5993a37ff300e36897debe9c5bf...
:uberjar-name ~(str "foobar-%s-"
(-> (clojure.java.shell/sh "git" "rev-parse" "--short" "HEAD") :out .trim)
".uber.jar")That said, I feel it's a little too harsh in places. Metadata is fine so long as you don't expect it to persist to derived data structures. If you think of metadata as being about a particular piece of data and nothing else, then you won't run into problems. One place I find metadata particularly useful is annotating event data, which could allow, for example, treating events from different sources with different priorities.
I find that there are a lot of specialised tools in Clojure. Most of the time it's good advice to avoid them, but in the rare cases where they are needed, you're glad to have them. Keyword inheritance, for instance, is something I've been using recently, yet I've rarely had need of it before.
Otherwise, lots of great points. One other thing I will say about Schema (https://github.com/plumatic/schema): while it's a lifesaver in a lot of ways, its mere existence really does expose some of Clojure's deficiencies when it comes to the type system (or lack thereof). I'm probably in the minority in the Clojure world but I really wish that it had better static typing and a more sophisticated type system sometimes. I've mostly made my peace with it but once in a while I look longingly at ML-family languages from afar...
I agree knowledge of how Java logging works is still useful, but I'm not willing to be part of the problem anymore :-)
Agreed on Schema, and how it illustrates deficiencies in the type system. I still want core.typed, or something like it, but it's too immature for production use.
Yeah, the post you folks put out on core.typed last fall (https://circleci.com/blog/why-were-no-longer-using-core-type...) was certainly disappointing but not unexpected...I'm curious to see if Jaunt goes anywhere at this point too: https://www.arrdem.com/2016/02/22/clojarr_-_a_friendly_cloju...
Anyways, thanks for the piece!
* Explain Like I'm A Java Programmer Who Knows Very Little About Clojure
I'd personally add the potentially controvertial "prefer transducers to lazy sequences." Lazy seq laziness is a big source of errors for newbies, and even for old hands since 1.7 Iterable-backed lazy seqs have surprising chunked realization behavior. Transducers take a bit more up-front effort to gain familiarity, but then yield fewer surprises.
What are good examples of situations where static typing is better than schemas?
I wish more people (could afford to) vote with their feet, as it were.
If your goal is "fire and forget" behavior (e.g. sending off a confirmation email), then it's better to use java.util.concurrent.Executors directly. It's actually easier than it sounds: https://gist.github.com/pesterhazy/c41786690ee0f8a3e5a21f88f.... Also see https://stuartsierra.com/2015/05/27/clojure-uncaught-excepti...
I think brevity is far more important than being explicit all the time. When we speak, we use a lot of context to infer what is said - no reason code shouldn't look like that.
(transaction-> db-name
(put! ...)
...)
The nightmare scenario here is not that the sequence will lazily evaluate outside of `with-read-txn`, because that at least will throw an error. Rather, it's that it will be evaluated inside a different transaction, without anyone ever realizing it. By leaving that possibility open, you're doing a huge disservice to the users of your library.
I am curious how to guarantee eager evaluation - dorun, doall and run! are all sequence-oriented right?
Libraries typically can't guarantee eager evaluation, since that's a property of the top-level execution. That's why libraries shouldn't use `binding`.
Back to square one.
The main feature that Clojure provides for concurrency is immutability, not just the possibility of using existing data structures in an immutable manner, but an entire suite of immutable data types and data structures. This makes concurrency, even using Java's primitives, like Thread, (which was the original way Clojure was intended to be used anyway,) with Clojure's data structures is a win over using Java's if you want safe, predictable behaviour in a concurrent program.
Can you elaborate on why using Thread was the intended way to use Clojure for concurrency?
core.async is also great, but not without caveats, and it also has some strong footgun potential. I've written a little bit on the subject http://blog.goose.haus/2016/04/01/the-puts-and-takes-of-core... and http://blog.goose.haus/2016/04/04/producers-and-error-handli..., with more coming, but the tl;dr is that core.async will swallow exceptions by default and there are a few other gotchas.
Also: Seeing what clojure considers bad parts makes me laugh since I spend most of my day writing python and javascript.
More language orthogonality more problems I guess..
Seeing how sophisticated clojure is compared to python makes me weep however.
Clojure has naive implementations of pmap, send, send-off & future. They're useful in the REPL, but you need to understand their shortcomings before using in would-be production code. Promise & delay have limitations, but are not so hamstrung as former functions.
Even if you don't expect any function to preserve meta, metadata is still useful. E.g., I've used it on a compiler to provide compiler tracing and decompilation.
Component is not essential. It helps if you're accustomed to OO languages. It's arguably better than the procedural style of with-redefs & bindings.
As amazing as core.async is, I am not a fan. I'd rather use queues, logs or an actor library.
Is it because of Java interop?
1. immutable by default 2. strong focus on concurrency/parallelism 3. well-thought-out design of core library (one of the most cohesive I've ever seen) 4. largest single Lisp community (just going by GitHub, Clojure has ~18k repos, CL has 10k, Racket has 5.5k, Scheme has 7.5k) 5. unified syntactic sugar for common data structures like maps/arrays
The above are just factors that distinguish it from other Lisp-based languages. If you're coming from Java/C/C++/Ruby/Python/PHP/etc, there's way more reasons than that.
I wonder why Clojure vs. ABCL (Common Lisp on JVM) though.
Clojure scripts take several seconds to start for me, and even jars take almost a full second. Am I doing g it wrong? Do people actually find this tolerable, when they are piping several of these scripts together, with the later stages being started a new for each input?
I must be doing it wrong. Are these clojure build tools supposed to run as daemons, that take requests from... bash scripts?
+1 for the pro tip!
If your doing a new language, you should look into it.
I start projects with the template:
lein new compojure my-app
Ring API docs: http://ring-clojure.github.io/ring/