Hacker News Reader

I use core.async a lot too and really like it, however, I do feel that it suffers from being a macro instead of part of the language, for example, you cannot nest core.async functions inside other functions as they must be directly visible by the go macro, which cannot see jnside function calls. I’ve also had errors where the stack trace did not reference my code files at all, because it happened inside some core.async setup (IIRC it was a variable that was meant to be a channel but was nil, inside a sub or mix or something, ie I connected two core.async things together, it created go blocks internally, the exception happened inside that and therefore never referenced any if my source files, since the exceotion happened in an asynchronous context after my code ran).This was extremely painful to figure out.

Neither of these issues can be solved as long as core.async is implemented as a macro. However, it is extremely cool how far it was able to be taken without changing the language!

I don't think it makes logical sense for there to be a "core.async function". Instead, you have each function return a channel with it's own go macro, then have the parent function consume data from the channels.

Sure, that’s a solution, but its necessary only because of a limitation that only exists because its a macro that needs to look into the code to rewrite it. It also adds extra (cognitive) overhead that such functions are always themselves asynchronous and the return value are channels - they can never just return a raw value - which limits what you can do with them or how you can call them or pass them elsewhere.

I mean, yes, usually it isn’t a problem at all, but it IS a limitation of core.async that wouldn’t need to exist if it had tighter integration into the language.

> I use core.async a lot too and really like it, however, I do feel that it suffers from being a macro instead of part of the language,

Agreed. The core abstraction of a go block is essentially a user-space thread. Because macros are limited to locally analyzing/rewriting code (and limitations of the JVM), go blocks are super limited in scope. As you point out, one call to a function in a go block, and you can't switch out of that go block during the function's execution. And functions calls are very common in Clojure code and can easily be hidden behind macros that obscure the details of the control flow.

There are 'core.async'y ways around all this, but the net effect is that 'core.async' imposes a very distinct style of writing (and one that tends to be contagious). Python's 'async/await' is contagious also, but because it's more tightly integrated into the runtime/compiler, it doesn't feel nearly as restrictive. (at least to me).

> and think it is better than golang's implementation.

How is core.async better than the golang implementation. In golang, all network I/O you do is automatically asynchronous, so your goroutines won't block a thread on network I/O. That is simply something core.async can't do.

I don't know if it does, but why couldn't it? The Java has non blocking IO.

It's not just a matter of non-blocking IO. core.async uses heavy-weight threads. Go-lang uses light-weight go-routines.

The JVM also would need to incorporate a lightweight process scheduler that can wait on I/O and schedule j-routines when I/O is blocked.

I don't know anything about the JVM and my experience with JVM-based languages is incredibly limited, but isn't that the essence of the Loom Project?

https://developers.redhat.com/blog/2019/06/19/project-loom-l...

https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht...

Yes, however, the Loom project is still some years away and the Java standard library will need to be extensively revamped to support this. I see this easily a decade in the future.

They have been rewamping the Java standard library slowly and silently over the last couple of years in anticipation for Loom. With Java 14 they re-wrote the tcp socket class. In Java 15 they're re-implementing the udp socket class. These re-implementations are being done to support loom seamlessly in a future update.

They also collaborated with IntelliJ to find bugs related to debugging Java apps, to make sure Loom doesn't break anything there.

Loom is well underway. I wouldn't be surprised if we saw Loom in Java 19 or 20. So in 2-3 years.

The next LTS version of Java is planneed to to be Java 21 (fall 2021, numbers a coincidence) and my presumption is they are trying to get it into that.

Hell, I just used loom last night on some pet projects. You can use it today if you dare.

`core.async` does M:N scheduling where many go-blocks are scheduled across a smaller number of heavy-weight threads. `core.async` also has channels and select. It pretty much works the same as in go.

However, since the default in JVM is heavy threads and blocking behaviour, extra care needs to be taken to avoid blocking a thread that a go-block is executing on (as that would also block many other go-blocks).

So it's definitely easier in Go, since you don't need to juggle these two contexts.

JVM has NIO, which provides non-blocking IO. You can use this with core.aysnc's channels to resume a go-block. You have all the pieces you need to write a go-scale clojure service.

Sorry, I completely disagree. I don't consider mere M:N thread work distribution as anywhere close to what the go scheduler offers.

This article mentions some of the fundamental problems with core.async: http://danboykis.com/posts/things-i-wish-i-knew-about-core-a...

Having to pay ridiculously careful attention to what code blocks across libraries and function boundaries is something that golang completely takes away the need for - because it has a true cooperating scheduler built into the runtime at the boundary of every syscall.

Decision making is in hands of the Go runtime. You can write in standard dumb sync fashion at 3am in the night within a go-routine making all the Go library calls you wish to make without needing to scan code with a microscope to see what blocks and what doesn't.

https://www.ardanlabs.com/blog/2018/08/scheduling-in-go-part...

core.async is a half-baked error prone solution. People will make mistakes with it. Until the JVM has native support for fibers, continuations and a compile mechanism to classify all legacy blocking calls as unsafe, core.async will continue to be hobbled.

M:N thread distribution is exactly what the go _scheduler_ offers. It's a work stealing scheduler like Java's ForkJoinPool.

The benefit lies entirely in that there's no other way to do concurrency. Which is great, as it's less error prone. But if you were to introduce native threads in Go you'd have the exact same problems as you'd have with core.async. So it's not like core.async (or async/await in other languages for that matter) is half-baked or badly implemented, but it won't automatically remove a part of the Java runtime that has existed since the beginning.

In a similar vain, Rust's actix and Java's Akka are not half-baked or bad implementations of the actor system, but it is more error-prone to use compared to languages built entirely around the actor paradigm, like Erlang and Pony.

So no, core.async doesn't magically give you non-blocking concurrency, but it does enable it. And that is a much better option than re-writing your entire codebase in Go.

> core.async is a half-baked error prone solution. People will make mistakes with it. Until the JVM has native support for fibers, continuations and a compile mechanism to classify all legacy blocking calls as unsafe, core.async will continue to be hobbled.

With Loom it won't be necessary to add a compile mechanism to classify legacy blocking calls, as there won't be any. This is why Loom is taking so long, the entire runtime is being re-written to not block if IO is performed in a virtual thread, but work as before if a full thread is being used.

And when you can simply create virtual threads instead of native ones, there's really not much value in core.async anymore (well, ClojureScript will still benefit) as Clojure already has great primitives for working with threads.

> So the guys who built golang did it because they had this cool idea for creating coroutines using channels.

Communicating sequential processes is an "old" idea. See: https://en.wikipedia.org/wiki/Communicating_sequential_proce...

That is true, but my impression is that the original syntax was cumbersome and that golang made major advancements on that front (though I'm no expert and haven't studied the original papers)

I'm no expert myself and wasn't trying to call you out or anything. (I've only read that paper once or twice before a Papers We Love event I attended.) I think it's a common misconception, though, and thought people might find the history interesting.

This stack overflow answer has a very fascinating explanation of the history of the evolution of CSP from Hoare's original paper, to the work done on it with Occam up onto Go, I recommend reading it if you are interested in this stuff: https://stackoverflow.com/a/32696464/172272

2. requires static typing in my personal opinion

cf. http://lambda-the-ultimate.org/node/4136#comment-62959

tl;dr on the above: The Expression Problem was originally expressly formulated with maintaining a Haskellian level of static type checking as one of its core requirements.

Saying you've solved it, when your solution involves on dynamic typing, or even casts in an otherwise static language, is arguably akin to kicking the ball across the center field line and then claiming you've scored a goal.

(See: http://homepages.inf.ed.ac.uk/wadler/papers/expression/expre...)

Actually, clojure compiles down to java classes, which are statically typed. Existing java types can be extended with clojure protocols. The link below talks about this, specifically about the expression problem.

See https://www.ibm.com/developerworks/library/j-clojure-protoco...

However, I learnt something. I didn't know that the definition of the expression problem required static typing. I prefer to think of it as adding static types to multiple dispatch.

You missed a detail: or even casts in an otherwise static language.

Clojure does compile down to Java classes, but Java is only a partially statically typed language. It also permits run-time casting, which, in a language that even tries to be strongly typed, means run-time (to wit: dynamic) type checking. And Clojure relies heavily on that.

That's why I invoked Haskell. It's an example of a language that is truly statically typed, in that it doesn't (generally) permit any run-time type conversions. ALL type checks must be done statically.

Then, the conclusion can be that dynamic languages don't suffer from the expression problem.

Pretty much. Not entirely unlike how one can't implement the Y combinator in a statically typed language, but it's trivial to do so in a dynamic language.

“I know of no widely-used language that solves The Expression Problem while satisfying the constraints of independent compilation and static typing” - kind of implies that the problem is different from whether or not the language is statically typed, doesn’t it?

I don’t think you should be writing off Clojure’s multimethods too quickly - they are quite nice, and I haven’t seen anything like them in other dynamic languages like Python.

You've cherry-picked that sentence, though. Given the entire context of the article, I would argue that one should interpret that as a case of unclear writing, and not something that is intended to directly contradict the definition given in the second sentence: "The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts)." (emphasis mine)

I’m not much of a clojure fan myself, but I think the observation is that by relaxing that last constraint of static typing (and given the other appropriate tools), that while you can’t solve The Expression Problem itself, it’s a bit easier to solve the real world problem that happened to manifest itself as the expression problem in your code.

By the way, I was also enticed by multi-methods in this area, but I ultimately don’t think multiple dispatch is necessary.

Yes, absolutely, but doing so would be moving the goalpost right out of the stadium. The expression problem isn't supposed to describe a challenge for people implementing line-of-business applications. It's a problem for programming language researchers.

> So, the expression problem is the problem of solving another, unnamed problem, while satisfying the constraints of static typing? It seems a not very useful term then, and implies that every problem will need two names - the name for the actual problem and the name for the problem of solving that problem while satisfying type system constraints. Bleh. -- Rich Hickey

Solving the problem doesn't require static typing.

Spec is pretty close to being a library for static typing; Clojure core being dynamically typed isn't really a blocker.

If you really want static typing Clojure can probably do it. The community support isn't going to be as solid as for something like Haskell. But I'm pretty sure spec offers stronger control of the useful parts of a type system than something like, eg, C. Except the control over RAM; that isn't a Clojure competence.

>core.async: So the guys who built golang did it because they had this cool idea for creating coroutines using channels. However, since it required very low-level functionality to be part of the core of the programming language, they thought they'd have to design a brand new language to implement their idea. However, shortly after golang was released, Rich and some other clojure folks implemented the same feature in clojure as a totally ordinary external library, proving that the core of clojure was general enough to support this. And it wasn't just a gimmick: I use core.async every day and think it is better than golang's implementation.

It is very limited -- it was done by a single university student as his MS thesis -- but luaproc had added this functionality to Lua before Go even existed:

http://www.inf.puc-rio.br/~roberto/docs/ry08-05.pdf

Other M.S. theses from PUC have extended the library (in particular, adding support for sending tables) but it is otherwise unmaintained:

https://www.maxwell.vrac.puc-rio.br/30267/30267.PDF https://github.com/lmillanfdez/luaproc-master https://www.maxwell.vrac.puc-rio.br/35424/35424.PDF https://github.com/fernando-ala/luaproc---messaging-tables-a...

Go was inspired by several systems that have come before it. You can write coroutine code with channels in C.

http://man.cat-v.org/plan_9/2/thread

It’s available for unix systems as well in the Plan9Port, and that predates Go by a bit.

I’m not sure what any of this has to do with Clojure though other than to say “look we have Go-like concurrency without starting over”. It’s been shown that you don’t have to start over to add features to other languages as well.

See also libmill http://libmill.org/

Go was not solely created to have goroutines and channels, but that was definitely one of the features desired.

Yes! I must point out a little factual error: JVM Clojure is not transpiled to Java, it is compiled on-demand to JVM bytecode.

core.async in Clojure is not equivalent to Go's asynchronous support. Even worse, core.async is the simplest & least-performant way to implement asynchronicity: It's just using an internal thread pool, meaning that go blocks in Clojure are not at all like goroutines in Go since they are state-machines produced by a macro that can block the entire thread when doing blocking calls. So, one has to manually "park" them by using core.async channels.

Summarizing: core.async is macro sugar on top of java.util.concurrent.

Go has actual lightweight processes that are scheduled on top of OS threads. Big difference in both design and runtime performance. Erlang too has m:n threading and offers a simpler and safer programming model (each Erlang process has its own heap).

Re "core.async is macro sugar on top of java.util.concurrent" - yes, but also no. Java does not (yet) have a way to spin light-weight processes that can park without consuming a thread OR a way to select/alt over multiple parked ops, so being able to do that in Clojure is actually novel (and enabled by having macros that can rewrite code as a state machine).

There have been a few projects that did this in the past on Java using runtime bytecode modification but none of them are widely used afaik.

This will change once Project Loom lands in Java and provides m:n fibers.

Yes, yes in the future at some undefined point java will be better.

Meanwhile go works very well today. And you don't need to be saddled with the JVM and the very bloated java ecosystem.

>very bloated java ecosystem

Why waste time reinventing what java has already solved?

Life is too short to waste on Java? Yes, some of us really find Java that bad.

Plenty of other languages have solved the same problems in much better ways (as demonstrated by this very discussion re: asynchronicity). I'll pick Go over Java, Erlang over Go and wouldn't pick Java (incl languages running on top of) for anything today.

Why not?

The java ecosystem has solved so many problems already.

Don’t you want to just focus on the business problems?

I don't solve business problems, I solve my own problems and try to have fun along the way.

The business problem still can include things like "concurrency", which is just -painful- in Java, especially if I/O bound.

I much prefer languages that make easy stuff harder, but hard stuff easier, than languages like Java that make easy stuff easier, and hard stuff harder.

Java doesnt solve business problems.

It solves enterprise ones.

Or at least claims to.

Enterprises do business last time I checked.

The Java ecosystem has solved many business problems, for me and plenty of other people I personally know, both in small startups and in large companies. Just because you are personally prejudiced against Java for some reason, doesn't mean that its not a good ecosystem, because it is. I don't like Java the language (and certainly not the crazy frameworks), although it has improved, but the ecosystem, especially from Clojure, is fantastic.

I usually find immutability to be more important than this particular detail. Because of this my preference is usually Elixir/Erlang, followed closely by Clojure (for the potential interop with a huge ecosystem), with languages like Go and Java fairly close to eachother but pretty far down list.

I would love it if Clojure on BEAM were more mainstream.

https://github.com/clojerl/clojerl ?

You are confusing Java with the JVM.

No I'm not, there is no escaping Java when you work with Clojure, since you'll be using Java libraries and working with Java APIs all the time.

Depends on what you're doing.

Sure, sometimes a simple, crude runtime will do, but if you need really good performance and low-overhead deep observability, you might want to be saddled with something that's more state-of-the-art.

Regarding Java, sure. But in the context of Clojure, it's here right now as well, no need to wait for anything.

I also don't really understand the "bloat" comment. What do you refer too? Are you referring to the fact the JVM supports too many features?

I am confused. If I remember correctly, go's runtime uses a thread pool as well. It's not like go somehow invented a runtime that does not rely on OS threads. So in the end of the day, what difference does it make?

If you ask me, java.util.concurrent is a much better and battle tested concurrent infrastructure than the simplistic implementation in go, because java.util.concurrent is actually written by the world's foremost concurrency expert, Doug Lee. I will take that over someone else's hand rolling concurrent runtime any day. God knows how many bugs there are in your home grown concurrency code.

It makes a huge difference. You can't use blocking calls in core.async, since they'll block the entire thread. There is no switching when you do a blocking call, the thread your code runs on will be out of commission until the blocking call returns. So this means that blocking APIs are out of the question and this translates to you not being able to write linear, straightforward code. You have to use non-blocking APIs and callbacks, and rely on core.async channels to force switching between blocks (core.async doesn't have lightweight processes). This is a total nightmare to debug and forces you to bend your programming logic to support this non-optimal programming model.

Constrast with Go goroutines or Erlang processes where you write linear, synchronous code that executes as part of lightweight processes. The built-in scheduler will automatically schedule these processes on real OS threads, and switch between them when they block (either through channels or blocking calls). OS thread utilization is far more effective, debugging is a piece of cake and the code that you write is easy to read and understand.

I'm bypassing your "world's foremost concurrency expert" comments. I suggest you use the concurrency offered by Go and Erlang by spending a few hours to do tasks you've done with core.async. You will learn something and you will see how Clojure falls far short of the optimal.

Hum, what you're saying seems to contradict a lot of what I've read. We're talking about IO here. So something has to block unless we're using non-blocking IO. I think what you mean is that Go will detect the block, and will yield your execution, and manage the IO in a non-blocking way automatically for you?

Under the hood, go still relies on non-blocking IO, epoll and all. And for disk and other serial IO I think it actually allocates real blocking OS threads as well.

I agree, the automatic management is convenient, but I fail to see the difference from core.async and the direct use of async IO or blocking OS threads when that's not available like for disk IO.

Go scheduler is sophisticated and it indeed intercepts all blocking system calls.

Essentially Go runtime will run a different goroutine, if current goroutine is blocked on: blocking syscall (for example opening a file), network input, channel operations, primitives in the sync package.

http://www.cs.columbia.edu/~aho/cs6998/reports/12-12-11_Desh...

I would always prefer the runtime to do this job instead of having to scan code under a microscope across functions, libraries and module boundaries to see what blocks and what doesn't.

http://danboykis.com/posts/things-i-wish-i-knew-about-core-a...

If I wanted complete control, I would use C++

I don't disagree about the convenience of having the Go runtime handle this automatically for you. I do wish core.async could as well.

That said, all your prior comments seem misleading, at least they confused me because it sounded like you were implying something that didn't feel accurate. Core.async has lightweight processes. The core.async Go macro can yield to other Go blocks and it will be really cheap to do so. You can have hundreds of thousands of them concurrently parked and cooperatively switching between each other, etc.

In effect, it means that you can achieve the same scale of concurrency in Clojure as you can in Go.

Yes, it will be trickier in Clojure, because you have to manage the async IO on your own, and you have to delegate an OS thread on your own when blocking IO is needed. In ClojureScript it won't be as much an issue since all IO is async, but still not as nice as auto-yielding by the runtime under the hood as Go does.

core.async doesn't have lightweight processes, for any sane definition of these terms. A code transformation spat out by a macro, subject to all the serious drawbacks I've already mentioned, is not a lightweight process.

But looking at the number of downvotes I've received, stating the obvious no less, I think I'll stop here. I will however take the completely nonsensical, misleading and downright erroneous responses illustrated in this thread as signaling me to steer well clear of Clojure and its community. Let's not forget that this thread started with someone claiming core.async is better than Go's goroutines.

> But given the number of downvotes I've received here

If I were to guess, I'd say it's because your comments feel like they come from a place of bad faith.

I'll give you the benefit of the doubt here.

> core.async doesn't have lightweight processes, for any sane definition of these terms

We can argue semantics if you want, but that leads to nothing constructive.

I understand very well that the underlying implementation is different. Core.async is a very clever code rewriting machinery. Go actually has a user level thread implementation, with each Go block its own stack, and the Go runtime schedules those over real OS threads. I know all that.

The implementation doesn't matter for what I'm saying though. At the end of the day, it means you can pause and resume multiple logical code blocks and cooperatively alternate between which one is given another chunk of CPU time. All while keeping memory footprint low, and with very fast context switches.

Thus you can achieve a similar scale of concurrency, and even though they achieve this with a different implementation, Go and Clojure both achieve the same end result.

While I totally acknowledge as well that the implementation Go uses allows for a more ergonomic programmer experience.

Go's concurrency primitives are a less expressive version of concurrentML, which imo has been the nicest way of writing concurrent (but for a long time not parallel) applications the last 30 years.

Guile scheme has a parallel version of it that is implemented as a library: https://github.com/wingo/fibers/wiki/Manual

Comparing concurrent systems is hard. I would suggest implementing selecting with negative acknowledgement (nack). If I can do that nicely I know I will probably like the system.