Jeaye

While we're here, has anyone seen any resources related to data-oriented design when GCs are involved? So much of data-oriented design is arena-focused, but that's not always possible, when the lifetime model of the code requires a GC (for whatever reason).

I feel like the DoD movement is a slow-moving, but big, change through how systems programming is done, but that there's still insufficient material for how to do this in different scenarios. I would really like to apply this more to my areas of work, which are also in C++, but there seems to be a gap between what they're presenting and how it can be applied.

More specifically, I'm using C++ to build a dynamic programming language runtime for a Clojure dialect. That runtime is required to be garbage collected, type-erased, and highly polymorphic. So I surely can't just SoA or AoS everything. Yes, I can pack my data, and I can avoid the GC whenever possible, both in compiler/runtime code and in generated code via escape analysis. But what about everything else, which is the 80% or more of the system? It could be that this runtime is too far at odds with DoD, but I generally see things as a gradient rather than black and white.

Do you have any recommended essential reading for this?

Ah, thanks for chiming in with your inspiring anecdote. I realize that perhaps I interpreted "working on" as pertaining to career work, rather than just investment of effort.

Your example of clowning shows something that requires real work and, as you laid out, hopefully does wonders to improving confidence, humility, humor, and charm. I wish you the best with that! And happy early birthday. :)

Would you mind sharing an example of "Working on something meaningful that pushes at the edge of my self actualization"? In my experience, work, even when meaningful, is quite separate from self actualization. I'm interested to hear how you're combining them.

I also rarely use multi-methods in Clojure. I just use maps and vectors and lists and keywords and strings and functions. Instead of multi-methods, I generally use a map of fns. When not writing open, extendable libraries, this is an even simpler choice which I find works well. Most of my Clojure work has been writing applications, so being able to extend in an open way is not a benefit. I'd prefer to keep everything centralized in one map.

Also, as I described above in the blog post, protocols/interfaces aren't needed to implement Clojure's object model either. They're just one approach. They're the nominal typing approach, in contrast with something like the structural typing approach, documented in the blog post. For that, we use C++20 concepts for structural typing predicates.

Protocols, records, and the like will be implemented in jank. They haven't been prioritized for two reasons.

Firstly, historically, jank had a closed object model for performance, which I have blogged about here: https://jank-lang.org/blog/2023-07-08-object-model/

Over the past few years, I've been spending hammock time to find a solution to this to re-open the object model efficiently. I came up with a design this year which addresses the concerns raised in that blog post and opens up the object model, but jank today is still in a half-way state between the two systems.

Secondly, in the several years I've written Clojure professionally, I don't recall ever writing my own protocol, record, or struct. When I was first learning Clojure, I leaned on them more heavily, since I was coming from OOP land. But once I leaned into the pure Clojure data designs, the OOP side of things just stopped mattering. So, if I were to write just about anything in jank, I wouldn't intend on using these features. That alone has made them low priority for me.

But they'll be implemented. :) I'd estimate Q1 next year for me to tackle those.

I am the creator of jank. I am also not interested in a flame war.

The key differentiator for jank is the seamless C++ interop, which is a problem not many people are willing to tackle, due to its scope and complexity. jank is a novel approach to this, providing incredible JIT and AOT support of arbitrary C++ libraries alongside your Clojure code, including a Cargo-inspired native build system for building your native deps along with your program (or finding them in the installed system).

For some people, there is another key differentiator, which is the amount of AI-generated code involved. My understanding is that Jolt and other newer dialects like Glojure have a lot of spunk and are exploring new ideas, given the velocity that AI-driven coding can provide. I don't see jank ever being categorized in that way.

Yes! This is exciting to see. Erin Catto is such a cool hacker. Thank you, Erin, for sharing your code with the open source community.

There wasn't anything about determinism in the announcement, but I'd really love to see some more about that, too. Trying to use Unity's built-in physics to make a networked billiards game is quite troubling, when none of the clients can happily agree on what happened.

I am researching for a talk on the philosophy of code, the similarities of engineering and art, and why we enjoy reading old code. This amazing work you folks have done may be an interesting tangent.

The biggest question I have for you is why you imagine we are so interested in reading these old scrolls. Surely some of it is to see whether or not, technically, we can. Surely some of it is to get a glimpse into the human expression inscribed on them. Are we looking to learn anything, or just to connect with our ancestors? I'd like to hear your take on it, both for why you think it's important and, if you know, why your colleagues feel similarly.

There are three aspects of this which concern me the most: personal, open source, and business.

On the personal side, code is art, not a means to an end. Many of us love coding. For me, it's my favorite thing in the world to do and it has been that way for 20 years. That's why we enjoy going back over old game code, to see the clever tricks that people came up with. That's why we preserve code in archives, to last thousands of years. Sure, we can generate classic art, too, and it can be fun. But generating Rambrandt paintings is nothing like actually seeing the originals. And it's nothing like creating your own.

Code being art is exactly why coders are so opinionated about their tools. Just like artists are. "I only use vim and functional programming and Linux" says the coder. "I only use these paint brushes, and this type of paint, on this particular type of canvas" says the painter. People don't consider this enough, for coding, so they just say "Use the best tool for the job." But any paint brush will do, for most jobs. Any language will do, for most jobs. Why it all matters is because we care. We care because art is a form of personal expression.

On the open source side, maintainers now just get huge PRs full of slop changes. The submitters tend to feel like they're helping, but they're just wasting everyone's time. The trust landscape of contributing has been eroded. The security concerns of large changes are now more pressing than ever. The gross disregard for licensing spits in the face of both the nature of open source and the actual legal ground on which LLMs try to stand. And then so many projects trying to grow end up getting vibe-coded weekend projects as competitors and people actually think they're comparable. For the non-trivial cases, they're not, because of the business side.

On the business side, let's take a programming language for example, like Zig. A business will be hesitant to adopt a particular technology if it's just one guy working on it. But if the technology has a community, a foundation, and a handful of devs working on it full-time, it is a safer investment. A big reason here is that the bus factor will be higher than one. However, LLM-generated code is a black box. It has a bus factor of zero. Nobody understands all of the code. Probably, nobody ever will. If there is a critical bug at a critical time, there is nobody to call. We would have to rely on LLMs to find and fix the bug, but how much faith can we actually put into this? That covers third party technology, but the same applies to first party technology.

If a small business decides to use Claude to create their mobile app, rather than hire an experienced dev, they now suffer the same black box and bus factor problem. Even if they hire a dev, but they expect the dev to finish at vibe-coding speeds, we end up with the same problem.

---

In short, vibe coding removes everything I love about my favorite thing to do. It also creates heaps of blackbox code with no ownership, beholden to proprietary tech which is only getting more expensive. Why would I like that?

Babashka's interop is with Java, since Babashka uses a Graal-compiled version of the JVM. It's still the JVM, just baked down.

This is different from interop with the native world. It's different from the host runtime actually being native, rather than a baked down version of a whole VM.

Graal's native images blur the line between the JVM and native, I would not say Babashka has a native runtime. Perhaps borkdude would disagree. Might be an interesting discussion.

I am developing a test suite for portability of clojure.core across dialects. You can find it here: https://github.com/jank-lang/clojure-test-suite

Currently, we have Clojure, ClojureScript, ClojureCLR, Babashka, Basilisp, Phel, and jank running the test suite.

I have only used Clojure, ClojureScript, and Babashka in production. But I am the creator of jank.

Once you learn Clojure's syntax and semantics, you're no longer bound to the JVM. There's ClojureScript (JS), ClojureCLR, ClojureDart, jank (C++), Basilisp (Python), babashka (SCI), and many others. This means that, if you don't know Java or don't like the JVM, you can likely use Clojure wherever you already feel most comfortable.

For the most part, any Clojure code which doesn't use host interop will work on all dialects. Clojure also has support for conditional code, depending on the current dialect.

This is one of Clojure's superpowers.

jank's custom IR is completely separate and unrelated to LLVM IR, aside from both of them being SSA-based IRs. We go from jank's AST into jank's IR into C++, which we then give to Clang compile into the LLVM JIT runtime. So LLVM IR is used in the pipeline, but we don't touch it directly. More info on that, and a diagram, is here: https://book.jank-lang.org/dev/ir.html (which I linked in the post)

As another said, jank is not replacing LLVM or LLVM IR. We still use LLVM IR! There is a diagram here which shows the pipeline: https://book.jank-lang.org/dev/ir.html

The main thing is that we just use our own IR first, to perform optimizations with contextual data which is gone by the time we get to LLVM IR. That's also why these optimizations are not practical to write in LLVM, since by the time we get to LLVM IR, we're too far separated from jank's AST with the high level semantics of Clojure.

So we just add an intermediate step. Once we have jank's AST, turn it into our own IR, do some optimizations on it for things that LLVM won't be able to see, and then hand it off to LLVM to do the rest.

The first three paragraphs here are on point! jank's IR passes will not worry much about things like load/store optimization, register allocation, inlining C++ functions, etc. These are in LLVM's domain. We just worry about the Clojure side of things. Polymorphic math is intense, but we do our best to avoid the extra work by unboxing whenever possible.

> A future optimisation might be to specialise for unboxed types: far more potential speed improvement over pointer tagging, and IMO quite amenable to analysis with the Jank IR

All of these math functions are templates with four specific categories:

1. Object and object

2. Primitive and primitive

3. Primitive and object

4. Object and primitive

We handle the difference between typed objects (like integer_ref) and type-erased objects (object_ref) as well. This template then gets inlined, which is exactly what the last step of the benchmark optimizations (adding annotations) ensured. The return type of these functions will prefer primitive types, rather than automatically boxing. jank's analyzer tracks all types used, at compile-time, and supports automatic boxing. This means that we're already using the most optimal primitive math whenever we can and that it will indeed inline to just an operator call when working on two primitives, or two typed objects, or a combination thereof.

You can see the code for this here: https://github.com/jank-lang/jank/blob/29c2adb344526d26c8e82...

I spoke with a couple Clang and LLVM devs about MLIR when I was doing the original design for jank's IR. The general consensus was that MLIR added a great deal of complexity on top of designing/implementing an IR and nobody was confident it was actually worth the effort. Since I knew exactly what I wanted, I just built that.

Hey lemming! You're right, which is why it should be used sparingly. Since clojure.core is compiled (on the JVM) with direct linking, reacting to var changes isn't an intended concern, since they're not going to work properly throughout any clojure.core code using that var. This makes it a good candidate ns for inlining things. But users shouldn't just be doing this for their normal application vars without giving it due consideration.

I think they mean the video thumbnail, which may or may not be AI-generated.

I'm working on the jank programming language!

https://github.com/jank-lang/jank

It's a native Clojure dialect which is also a C++ dialect, including a JIT compiler and nREPL server. I'm currently building out a custom IR so I can do optimization passes at the level of Clojure semantics, since LLVM will not be able to do them at the LLVM IR level.

As much as any C++ project would, yes. That includes either through the C ABI or through the various C++/Rust interop mechanisms.

jank is Clojure and will track upstream Clojure development. I'm working closely with the Clojure team and other dialect devs to ensure this remains the case. I am leading a cross-dialect clojure-test-suite to help ensure parity across all dialects: https://github.com/jank-lang/clojure-test-suite We have support or work ongoing for Clojure JVM, ClojureScript, Clojure CLR, babashka, Basilisp, and jank.

With that said, jank will do some trail blazing down other paths (see my other comments here about Carp), but they will be optional modes which people can enable which are specific to jank. Clojure compatibility will remain constant.

JIT compiling C++ is definitely the slowest thing we do. However, we're working on two different codegen modes.

1. C++

2. LLVM IR

The IR is much faster to compile, but its perf isn't nearly as good. This is meant to be a nice trade off, during iteration, so that you can use C++ codegen for your release artifact, but stick with IR when you're REPLing. The IR gen is still unstable right now, for the alpha, but we'll have both solidified this year.

Carp is great and I would love to include a mode of jank which is very much Carp-esque. If you're interested in working together on this, please let me know.

We have a working nREPL server, but it's not yet merged into the jank repo. https://github.com/kylc/try-jank

There's a Clang bug getting in the way of the progress we want, so we'll need to work around. There's a lot I'm juggling, but this is high priority.

Thanks for the broken link report. That should be fixed now.

No full-lang static type system. jank has seamless C++ interop and all interop is statically typed, but as soon as things get back into Clojure land, it's all dynamically typed and hella polymorphic.

I will be exploring optional static typing modes for jank in the future, a la Carp. That will not be this year, though.

Hi! I'm happy to accept grammatical PRs to the book. You could also report issues via Slack or Github issue. I will not be accepting larger PRs to the book, to maintain a consistent voice.

Please link to https://book.jank-lang.org/ for anything related to the jank alpha. I wasn't quite ready to announce the alpha, but I appreciate that folks are excited. :)

There's just a couple more compiler features I wanted to get merged, but everything should be in order for folks to follow the book and try jank out.

It looks like the key missing part which would be needed for a lexer is source information (bare minimum: byte offset and size). I don't think edn.c can be used as a lexer without that, since error reporting requires accurate source information.

As a side note, I'm curious how much AI was used in the creation of edn.c. These days, I like to get a measure of that for every library I use.

This is superb. Thank you for making it and licensing it MIT. I think this is a contender to replace the lexer within jank. I'll do some benchmarking next year and we'll see!