Tour of our 250k line Clojure codebase (2021) | Hacker News Reader

Tour of our 250k line Clojure codebase (2021)

blog.redplanetlabs.com

86 pointstodsacerdoti45 comments

Sounds like a great way to burn money: the company was founded in 2019 and they still haven't released anything, probably because they've spent most of their time building other stuff. Their devs sure are having a lot of fun though.

That was my issue for a long time. I even talked with their founder several times on twitter back a few years ago. Each time I was greeted with buzzwords, that I knew the meaning of but I think they assumed I didn't.

They would claim grand things like having solved the issues with continuations and delimited continuations, distributed process migration and a whole host of other very hard problems that haven't been solved in the past. I would ask their founder: "right, so you know that delimited continuations have problems with accidental captured scope, they run poorly on the JVM, how did you solve this, have any papers I can read?", and all I ever got was that Clojure, immutable data, X and Y would fix these issues and you just had to wait and see what they were cooking up.

That's when I knew they had no clue what they were doing. I'm all for pushing the boundaries of tech, but if you're doing something attempted many times before, you at least need a good elevator pitch as to why it's solved now.

I think a great example of this done right is Rich talking about Clojure. People would ask "isn't immutable data expensive to reclaim and allocate". And his reply was always that the JVM's GC was just that good that the benefits to be gained from immutable data outweighed the marginal performance penalty of the amount of garbage collected. What changed since the old lisp days? Well we now have GCs that are super fast and JITs that can optimize dynamic code well.

That's the sort of laser focused vision I never saw from Red Planet labs. You gotta get that problem statement and the solution out early, refine the elevator pitch and be able to articulate to people who know what they're talking about how you're going to succeed where others have failed for decades.

I see your overall point which is a good one and I know this is a nitpick, but I thought Rich’s core solution to the cost of immutable data structures was to find a way to get the cost down by extending some existing research by Phil Bagwell.

From https://dl.acm.org/doi/pdf/10.1145/3386321

“I then set out to find a treelike implementation for hash maps which would be amenable to the path-copying with structural sharing approach for persistence. I found what I wanted in hash array mapped tries (HAMTs) [Bagwell 2001]. I built (in Java) a persistent implementation of HAMTs with branching factor of 32, using Java’s fast System.arrayCopy during path copying. The node arrays are freshly allocated and imperatively manipulated during node construction, and never mutated afterwards. Thus the implementation is not purely functional but the resulting data structures are immutable after construction. I designed and built persistent vectors on similar 32-way branching trees, with the path copying strategy. Performance was excellent, more akin to O(1) than the theoretical bounds of O(logN). This was the breakthrough moment for Clojure. Only after this did I feel like Clojure could be practical, and I moved forward with enthusiasm to release it later that year (2007).”

Or you tell me what I’m missing. Big fan of your work in core.async if this is the same halgari.

Same halgari, yes :D.

Yeah he took it from Bagwell, and adapted it, but in general there was a whole discussion way back in the day (~2012) questioning how creating this much garbage by boxing and throw away collections could ever be fast. Datomic is another example: making an immutable DB is a dumb idea right? Well what if storage was super cheap, and almost free? Well then maybe it's not such a bad idea.

So a lot of the Clojure community is based on this idea of taking ideas from way back in the 70's and saying "Well everything has changed, what works now that didn't then"

That’s super interesting and makes sense - even with persistence of trunks and branches there will be leaves to throw away / GC. Thanks for explaining!

Scala has delimited continuations since some time now, so it can seemingly be done in a way that's performant enough.

Oh for sure, it's a bit wonky on the JVM due to the lack of tail calls, but that sort of thing can be done via full-stack bytecode transformation.

But these people are doing this in Clojure, which is quite removed from the JVM bytecode, and talking about how it solves so many distributed problems, which I just don't see happening.

If you look at redplanetlabs github repo, there's a ton of low level manipulation of 'JVM bytecode assembly language', e.g. https://github.com/redplanetlabs/defexception/blob/master/sr..., for projects that aren't even compilers, one would assume their "compiler" does this even more so.

Another hypothesis is what we're doing is so hard and so valuable that it really requires that much effort.

I suggest reserving your judgement until you've seen what we've built, which will be soon.

> People would ask "isn't immutable data expensive to reclaim and allocate". And his reply was always that the JVM's GC was just that good that the benefits to be gained from immutable data outweighed the marginal performance penalty of the amount of garbage collected. What changed since the old lisp days? Well we now have GCs that are super fast and JITs that can optimize dynamic code well.

It's a little odd to see this deeply mistaken belief dating back to the early Java days being advocated for today. GCs are very constrained by the tradeoffs they make, there's no free lunch.

Much in the way an F1 car is only fast on a race track specifically made for it, the only reason a massive rate of allocations can have a merely marginal performance penalty is if the GCs in question have been specifically designed to handle it. But in doing so, they must have made sacrifices elsewhere, e.g. to memory usage.

Code that's not written with the underlying machine that will ultimately execute it in mind will never be fast, no matter how much we jiggle tradeoffs around. Therefore, while the gains of immutable structures may still outweigh the performance loss, the loss cannot possibly be characterized as marginal.

The founder has released successful large projects in the past, e.g. Apache Storm used by Alibaba, Yahoo, Twitter, etc. Probably just need to wait a bit longer.

> the company was founded in 2019 and they still haven't released anything

According to this tweet the author has been working on the product/company for ten years! https://twitter.com/nathanmarz/status/1636050500553699330 (And presumably still hasn't released anything in those 10 years?)

The thing i want to know is how is this company funded? What puts food on the table? Or is it just living off personal wealth? Or is it investor funded, in which case, how come the investors are not clamering for a return on investment?

Edit: just found https://medium.com/red-planet-labs/introducing-red-planet-la... , so i guess they're using investor money.

We're funded by top investors including Initialized, Naval Ravikant, and Max Levchin. They've seen firsthand what we're building and understand both how hard it is and how valuable it will be.

Discussed at the time:

Tour of our 250k line Clojure codebase - https://news.ycombinator.com/item?id=27385208 - June 2021 (227 comments)

Gotta be honest, this sounds like a nightmare. Clojure felt like it got out of hand in my own _personal_ projects, I can't imagine going through 250k lines of it without a strong type system backing it up.

And they're building a "programming tool". Hmmmm

> We have about 400 macros defined through our codebase, 70% of which are part of source code and 30% of which are for test code only.

You can write 'type signatures' easily in clojure code, there's dozens of options, they're just only checked at runtime (and usually relegated to developer builds). if you combine that with good code coverage, it stops being a weakness and starts being a strength, because you declare data types with far more specificity & precision than a static type system.

As rich likes to say, the type signature of your stereotypical statically typed language, e.g. reverse :: [a] -> [a], doesn't have a much information content.

I write clojure, and still 'think in types' & category theory.

if I type ctrl+space, will it give me the correct auto-completion? despite what type-system galaxy brains might think, this is 95% of the value

> the type signature of your stereotypical statically typed language, e.g. reverse :: [a] -> [a], doesn't have a much information content.

It's a little bit unfair to imply any alternative to bleeding-edge clojure is going to have a simple 70's Damas-Milner type system. Today, static types can describe a lot more. Refinement types (e.g liquid Haskell) and/or dependant types (e.g. Idris or Agda) can describe and enforce the desired properties for reverse statically.

I think you're downplaying the strength of types! In any relationally parametric type system `reverse :: [a] -> [a]` has a huge amount of content.

I'm not downplaying anything, it was the parent who was dissatisfied with static checking of reverse. Personally, I think any static checking is better than none at all. Statically typed languages are perfectly capable of checking additional contracts/properties at runtime too (as Clojure does).

Firstly, I was being a bit tongue-in-cheek, but secondly you seem to be saying that a "70's Damas-Milner type system" isn't already an extremely powerful type system. I would say that it is!

I was just trying to combat the common narrative that static types are "what Java has". There is so much potential for static typing that has not been explored by industry (e.g structural typing).

(That common narrative is also quite negative, Java’s type system is surprisingly strong and can express many more things than people give it credit for. See jooq, or several FP java frameworks)

Jooq is not a good example as it is essentially dynamically typed. Queries on the relational algebra can be statically typed, with structural typing and a few type-level features like sets. There are lots of papers on this.

It is quite statically typed, look at the examples. You might be thinking of the stringly-typed version, but it can generate classes from DB inspection and use columns and whatnot with its respective type throughout a query, failing to compile when a group by value is incorrect! It can’t verify everything what a SQL engine can of course, but it is quite remarkable in my opinion.

I feel that every codebase built by one person is understand by that person really well and nobody else gets to the same level of understanding, unless the codebase is embedded, it shall only exist as long as the original person is around.

I want to see a whole discussion based on that comment. This is a real and big issue, and the big reason why it’s hard to scale the team.

I don't believe code _is_ the documentation.

I've thought of information systems for code, I once used OpenGrok to parse a large codebase and try work out how it works. But it's undirected learning.

I think question answering would be useful:

"Where should I change the code to insert behaviour that changes X, Y, Z behaviours"

> This is a real and big issue, and the big reason why it’s hard to scale the team.

it's also why java as a language is good, not because the language is good, but that it's easy to have someone not originally involved in adding/maintaining the code base - you can't get too clever with java.

I would have said the same with Golang, Java though… not sure I understand considering all the crazy patterns that exist there and the huge mess codebases are due to how namespacing work and other reasons.

See Naur’s Programming as Theory Building.

“Measuring programming progress by lines of code is like measuring aircraft building progress by weight“—Bill Gates

which is really a good way to do it. you design it to be so many pounds, when you get it all together you should be there (or close to it).

You start a project knowing how many LOC it will take?

I want to assume you’re being sarcastic but it really doesn’t sound like it

It's responding to the analogy.

not knowing exactly, but with an estimate of the software size. sure. how else do you reliably estimate the projects cost and schedule. "engineering" not "programming"

Difficulty of the problem you’re trying to solve.

Just like it could take 5 hours to come up with 5 lines of code, it could also take 5 minutes to produce another 5 lines. All depends on how complicated the task is.

yep. and when you need to write 300K SLOC, the average takes care of the outliers.

Interesting. I can estimate time by gauging complexity but LOC (for me) is usually not exactly correlated to time or complexity

“This airplane design will be done much faster if we use lead and tungsten in key places.”

If I had to name one design pattern that really helped my org’s codebase scale into the million LOC range, it would be replacing our database with in-repo json files.

So, does it do anything? Is it just pretty code that does nothing? I sure dont see it doing anything, and every sentence on the website is incredibly vague. The CEO just claims 100x more features on twitter, and things like that.

Now up to 345k:

https://twitter.com/nathanmarz/status/1616577265139355655?

How many K parentheses might be more interesting.

Reply on news.ycombinator.com