Clojure Async Flow Guide
clojure.github.io
clojure.github.io
Just like `core.async` itself was a frontrunner of Virtual Threads on the JVM, I view `core.async.flow` as the Clojure version of the upcoming [0]Structured Concurrency JEP. I do wonder if it will use that under the hood once it becomes stable, the same way `core.async` is planning to move away from the `go` macro to just have it dispatch a virtual thread.
For example, class initialization pins a thread so any singleton defined in the standard, recommended Java way (using a static inner instance of an inner class) can hang your program forever if the singleton definition suspends. And because they worked really hard on making async colourless, there is no way to know that a method call will suspend. This is a known issue with a workaround if the suspend is due to a network call,
https://bugs.openjdk.org/browse/JDK-8355036
which is useful for some applications (web servers). Figuring out that this is why my program was hanging required quite a bit of work too. We are still frustratingly far from the ergonomics of Go concurrency (all threads are virtual threads, hangs automatically panic).
The difficult problems are execution problems like pinning. There are plenty of existing concurrency libraries on the JVM (Cats Effect, clojure async, Kotlin coroutines, RxJava, quarkus, etc etc). The promise of VT is that you will no longer need those for scheduling and execution of work (whether that's tasks, fibers, coroutines, actors etc.) This only works if you use VTs throughout, not just on IO pools.
I am no expert on the topic, but this seems like a very edge case scenario, that is not trivial to reproduce on even the linked bug ticket. Do you think it really is that big of an issue?
- vt1 locks lock1
- vt1 suspends on lock2
- n VTs attempt to initialize a singleton that requires lock1, so they all suspend within pinning class init.
- you release lock2.
- all platform threads are pinned, so vt1 can't run and you hang forever.
There is no lock inversion and progress would have been entirely possible with platform threads, even with just one.
It happened on our system because we have parallel streams that all access the same singleton at the same time, which is fairly easy to do (you have lots of parallelism, you have a map operation that needs a value from your singleton and that's it.)
The solution is to never suspend while in a static block, but it's hard to generalize because... any method may suspend, and there is no way to know that it will because of colourlessness. And also the singleton pattern is common, often involve accessing expensive resources or IO (and suspending) and doing so with class init lock is recommended and common,
https://shipilev.net/blog/2014/safe-public-construction/#_sa...
In our case this involves scala's object {} which is a singleton defined using class init. Kotlin probably works the same way.
Hopefully it will be solved, similarly to the other pinning issues then! Though wouldn't reserving a few platform threads solve the issue?
- getting back pressure right. You need to tune the buffers for the input chans correctly, which takes some thinking, and in my case, some blowing up in prd.
- flow monitor (the UI for debugging flows) can't handle large process states, which makes it pretty much useless for my program.
- understanding the flow as a whole (rather than specific processes, which are easy to understand in isolation). For example, answering questions like "why hasn't this process received a message for a while?" was tricky.
We are building apps for our clients in it, we will also have our own product built with Clojure and ClojureScript soon.
A process in Elixir has an id, state, and is sent messages to its address. The message are queued inside the process, and handled one at a time.
One process can spawn another, and so on.
It's more like white collar workers sending emails to each other.
In core.async, a Process is anonymous, it doesn't have an id and doesn't have an address. You cannot send messages to it.
Instead a process is more like a worker on an assembly line with conveyor belts.
What matters are the conveyor belts and what's on them. Where things go from one belt to the next and what happens to the things as they flow through. You can have multiple workers working a belt and if something jams dependent belts stop. The belts are called Channels.
I guess one could in theory swap flows the same way values are swapped, but I wonder if this is the way this library is supposed to be used. I also wonder what happens to non-empty channel buffers in this case.
I am curious to hear other opinions.
Even though the flow topology is fixed, it's perfectly acceptable for a flow component to use other variable resources and act merely as a coordinator. So you could for example have a process that send data out to an external dynamic thread pool and gets callbacks via a channel.
I do find that for about 5 years things seemed to be slowing down. Though I keep seeing it pop up, and new exciting projects seem to pop up from time to time.
Just today I saw an article about Dyna3, a relational programming language for AI and ML that was implemented on top of Clojure.
I miss the Strange Loop conference. I think a lot of Clojure buzz was generated there. Clojure West and a few others so a decent job, but the quality of the talks at Strange Loop were second to none. Not that it was a Clojure specific conference, but it had that focus on elegance that I don't see very often, and the organizer was a something like the Prince of Clojure, if I recall correctly.
I'm still enjoying the language, and all my projects still build and run just fine.
The major frustration I have with the platform is 3D graphics. That's a JVM issue overall though.
Link to demo @ timestamp: https://youtu.be/UVsevEdYSwI?t=653
My experience with 3D graphics is minimal, but I'm curious to know if these newer developments are significant in any way for 3D work.
Cursive, Calva and CIDER are already quite good.
After that, it is all about the ecosystem, what libraries people care to build.
I'm eagerly awaiting Jank to stabilize.
The small-medium sized community is actually fantastic for learning. The big names in the community are only a slack away, and everybody is so enthusiastic.
Things just mature and hype isn't as cool when you heard it 5 years ago.
which is why now is exactly the right time to start using clojure - after the hype died, but have active community and users.
2. https://dotnet.microsoft.com/en-us/apps/aspnet/web-apps/blaz...
Not every project uses @FunctionalInterface, but I've been trying to add it to places [1] [2] [3], and now I'm able to use Clojure in a lot more places.
[1] https://github.com/LMAX-Exchange/disruptor/pull/492
FWIW, Google Trends shows the hype peaking in 2016, but I doubt that reflects usage as much as buzz.
Instead, if you look at the annual State of Clojure survey results, which solicits opinions directly from the community, the number of responders peaked in 2020 at ~2500, and is down to ~1500 for the most recent 2024 survey.
- 2020 State of Clojure - https://www.surveymonkey.com/results/SM-CDBF7CYT7/
- 2024 State of Clojure - https://www.surveymonkey.com/results/SM-hht04mGydwZ6Nqr7N8vj...
- 2024 Highlights
- Trends Over Time
- 2024 New Users
- Previous Results
> Clojure versions
> Clojure 1.12.0 was released in September 2024 and the survey showed rapid uptake, with 58% already using it, and 65% developing or deploying with the prior versions 1.11, and a steep drop-off after that. Clojure’s focus on stability and avoiding breaking changes makes upgrades safe and easy.
> Trends (use at work, hobby, and study have all up-trended)
> https://clojure.org/news/2024/12/02/state-of-clojure-2024#tr...
> Because this survey has been running since 2010 (thanks to Chas Emerick originally!), we have lots of great longitudinal data and it’s interesting to compare some of the answers over time.
> Looking at the question of how Clojure developers use Clojure, we can see this has generally trended more towards using it at work. However, this year we saw an uptick of people using it for hobbies or in their studies:
Look for the number of responses, and you can see a decline each year after 2020.
---
It's possible that the survey may not have been advertised as well, but afaik, it's still posted the same way it always was: announcements on Clojurians, Clojureverse, reddit, etc. I haven't heard of any reason that survey numbers would have been artificially depressed for several years running.
Or, I picked a random, reasonably popular library to check on Clojars: http-kit. The most recent stable release, 2.8.0, which came out last year, has only been downloaded ~600k times. 2.7.0 from 2023 was downloaded ~1.4m times. 2.6.0 from 2022 was dled ~2m times. Ditto for 2.5.3 from 2021.
I would have used Clojure itself, but I can't find maven dl statistics.
https://clojars.org/http-kit/versions/2.8.0 https://clojars.org/http-kit/versions/2.7.0 https://clojars.org/http-kit/versions/2.6.0 https://clojars.org/http-kit/versions/2.5.3
---
The thing is, I've been seeing little pieces of evidence all over that Clojure is waning, and not much that it's genuinely increasing in popularity. Any individual example doesn't weigh that much, true, but everything seems in the same direction.
If people want Clojure to grow, whether because they need job opportunities, a big employee pool, whatever, it starts with a clear assessment of where it's at.
Maven stats are available to artifact deployers, but they are useless for estimating users or community size as downloads are largely from CI servers constantly downloading artifacts for testing. Download numbers are large and seesaw erratically. Unique IP counts are a little more stable but also inflated beyond relevance by CI.
When I was a scientist, we were trained to look at the overall body of evidence when trying to assess a claim. Any individual study has flaws, but when we survey the literature as a whole, do we see many studies confirming a finding? Independent studies presumably don't share the same set of flaws (usually), and the more studies demonstrating something, the more likely it is to be true.
And I just haven't heard any counter-evidence. Are job postings increasing or decreasing? What's the change in MAU for Clojurians? Etc.
I looked for job/social media numbers, but it's hard to get those numbers. If you have better data, I'd love to see it.
I need a tool that helps with problem-solving and product development, and works reliably and effectively across a wide range of use cases, from basic mobile apps to high-performance computing.
Clojure delivers that better than any other language or ecosystem that I know of in a uniform, well-designed package, all the way from the core internals of the language to deps.edn.
Yes, doing Clojure is more demanding initially than many other languages where you can basically template/ LLM your way to anything. With Clojure you might find it easier to do bigger and longer term projects that are rather easy to maintain even for a single person or a small team. That of course doesn't cause much buzz, because it's boring in the good sense.
It's bad for marketing, but seems promising for the project longevity.
I thought Rich was being a bit pretentious when he compared to violin, but few people ask "are violins still a thing?"
Between Clojure and Racket, those have always been my two favorite Lisp / Scheme languages. I don't do a lot of Lisp but when I do its either in Clojure (thanks to Lein) or in Racket.
Meanwhile: the core team has been extended the last couple of years. Also this summer NuBank (the company behind Clojure) announced the first 'Clojure Developer Advocate'. Their role will be to "focus on ways to support the existing Clojure community and grow the community through outreach and development."^1
[1]: https://building.nubank.com/clojure-developer-advocate-nuban...
EDIT: wording.
Debugging and profiling is still somewhat Java based, and yeah that's can be irritating, but you get used to it.
Personally I do think that it's worth it; Clojure is a very pleasant language that has no business being as fast as it is, and core.async is an absolutely lovely concurrency framework if you can convert your logic into messaging, and you have Haskell-style transactional memory for stuff that can't be. So many problems become less irritating in Clojure.
It or or was a build tool like Leiningen.
Debug specifically is state-of-the-art. Look at YourKit, or any debugger included with common IDEs.
None of those tools has a shiny Visual 2025 aesthetic, but again, they work reliably and are going to work the same way a year from now.
[0] - Technincally Objective-C and Swift could also be considered, but they lack the industry wide adoption, as many cool tools only exist in Apple land.
Swift is quite good. I assume the tooling is catching up to Objective-C. It has been a couple of years since I had to solve something with Swift.
Any reason, apart from outside requirements, to pick the .NET CLR over the JVM as a runtime in your experience? Outside of library/framework support, which I've heard is good with the CLR.
How's the garbage collector with the CLR?
Switched to SBCL for the faster star-up times. Now lisp-2 also feels more comfortable.
The transition of core.async specifically to VirtualThreads is still WIP as far as I understand, but with minimal tweaks, 90% of the benefits are already there, even with the current latest version.
STM, atoms and agents were there from the beginning. I think futures and promises were added in 1.1. core.async is from 2013. Even popular third-party libraries like promesa and manifold are around 10 years old at this point.
I think flow promises to make it easier to orchestrate core.async, specifically, in complex applications, but the essential primitives are far from new and I don’t consider them any harder to use than JavaScript.
"The description arity takes the current state ..." should be "The transition arity takes the current state ..."
"excepiton" should be "exception"
"Exceptions thrown from the transition or transform arities, the exception ..." should be "If exceptions are thrown from the transition or transform arities, they ..."
"provded" should be "provided"
EDIT:
gpt says,
Choose core.async.flow when you want a declarative, monitorable process graph over channels (pause/resume, error channel, flow monitor).
Choose Manifold when you need deferreds/streams as general primitives, precise per-op backpressure, error-aware chaining, or you’re in the Aleph/Netty stack.
very abstract but based on this, the answer is no.