Hacker News Reader

Honestly I’ve seen just as many disastrous codebases written in highly structured, statically typed languages. Any language with intrinsically interesting features tends to attract inexperienced (or just bad) developers who don’t appreciate the tradeoffs of their tools and design decisions, and think the tool is a panacea. This leads to disastrous codebases, and is not at all limited to lisps or dynamically typed languages. I can point just as many disasters written in statically typed languages, and they tend to be the “interesting” ones with fancy type systems in the ML family like Scala, Haskell, and Rust. (I am a huge fan of both Lisps and ML-family languages.)

I think the important point is to be very aware of this risk and consider it when making technical decisions. Sometimes the tradeoffs are worth it to use a powerful tool, depending on the team and the product, and sometimes you’re better off using boring tools that won’t distract smart engineers.

I 100% agree that disastrous codebases are every bit as likely in other languages. But it somehow hurts more in a lisp.

I think that the syntax really is the culprit. In a language like Java, the Byzantine syntax and semantics enforce some minimum level of structure on the code. It's not much, but it's something, just enough to give an experienced programmer a few extra heuristics they can use to make sense of what they're seeing. Even some things that normally drive me nuts about Java, like the way that there's no standard way to call an anonymous function (because the invocation is done through a method whose name is allowed to vary) end up being useful bread crumbs when I'm reading somebody else's code.

In a Lisp, though, everything looks more-or-less homogeneous. You can memorize the names of special forms, but, beyond that, you can't quickly tell whether some thing you're looking at is data, a function, or a macro, or what. It all blends together into a disorienting fog. You need to go trace its provenance to see how some new value is defined or constructed before you can even tell what kind of thing it is. (This, tangentially, might be a great argument in favor of lisp-2, though I haven't spent enough time in a lisp-2 to have an opinion that's worth beans.) It may, in the grand scheme of things, be only a small reduction in what you can assume without careful study. But, in an unfamiliar codebase where you don't know much to begin with, every little scrap of knowledge counts.

It all points to an insight I should have had 20 years ago when I was working in Perl: Features that make a programming language pleasant to write tend to make it unpleasant to read, and vice versa.

I spent a lot of time thinking about this.

What makes Perl or Clojure such fun languages to work with is they don't impose structure on you. You can do whatever the hell you want.

On the other hand result of this is the code represents basically how you think about the problem. Which would be very different for every person.

Languages with frameworks like Java+Spring, Ruby+Rails etc. are less "fun" to work with because you are not given so much freedom and likely have to fill in a lot of stupid, mundane boilerplate, but are easier to maintain in team setting because developers know more or less what to expect from the code. Very likely developers come with that knowledge and if you know one Spring application you know how to move around almost any Spring application.

It is the frameworks that give more rigidity and boilerplate, not so much those languages themselves (although Java does have more boilerplate than Ruby). You have a lot of freedom when you program in plain Java and Ruby vs. using frameworks in those languages. And that is at least partly by design. See the Template Method design pattern, which is the basis of frameworks that use Inversion of Control (IoC).

Clojure frameworks are written by Clojure developers which means they don't give you structure, only tools to realize your freedom.

Frameworks that give a lot of structure are beneficial to novice developer because they need that structure. In absence of supervision, the various articles on the Internet, stack exchange answers on how to write a controller or other piece of Java app, give a novice developer necessary guidance on how to structure their code.

On the other hand as a more mature, knowledgeable and experience developer you may see that this structure is not perfect. It is repetitive and it has a lot of boilerplate and it does not suit well every application.

You may even figure out that every application has their own perfect framework, depending on its size and other characteristics and the problem it is trying to solve.

Clojure lets you design that framework and then write the application in perfect framework for your application.

That framework might be some zero-code decisions (like decisions on where to put which part of the code) but can also be some set of macros up to full blown DSL.

Assuming you are mature developer, you will know how to use these to solve your problem but if you don't then that's where the problem starts.

Yes, frameworks are kind of like a smarter programmer starting you off with a code base already, so if you don't really know what you're doing you can't mess it up as much. I've still seen people mess it up, and generally it's when the developers begin to "play" with things they don't understand, like bringing in Aspect J, writing custom annotations, slowly moving logic to configuration files, starting to dynamically load modules in/out.

It's a bit of a struggle because an experienced dev will hate the framework, as they know better, but then as less experienced dev work on the code base it'll degenerate in a way the framework would have survived longer due to availability of documentation and Stackoverflow Q/A.

There are two more points.

If you are tech lead or manager, these frameworks are really great value because they "give" (or more like let developer find) guidance on a lot of aspects of the application.

If you created your own perfect framework, you would be responsible for providing a huge amount of that kind of guidance, but if you are using Spring your every developer can just google the answer to most (or even almost all) of the problems.

And the second point is, again, if you are tech lead or manager, these frameworks are great help preventing your "highly intelligent" developers from making a mistake and developing their own framework. I put quotes intentionally, because some developers just think themselves to be very good developers but dismiss the risks and costs of developing software that is not core part of the project (aka NIH).

With Spring regulating most aspects of the application any damage from those kinds of actions is also limited.

> mess it up [...] like bringing in Aspect J

And, see? That's from a Lisp guy. QED.

> What makes Perl or Clojure such fun languages to work with

... while you're writing a small program by yourself ...

> is they don't impose structure on you. You can do whatever the hell you want.

You mean, there is nothing _preventing you_ from doing whatever the hell you want. That's not the same thing.

It may take a while to see the visual clues: indentation, formatting, structure patterns. Beyond simple lists, Lisp uses a variety of list structure patterns for language constructs. It's a bit like learning to ride a bike: initially it looks not possible to balance, steer and move forward at the same time.

One thing that's not that usual is that authors can implement new language constructs themselves. Thus one may need to understand that meta-syntax level when reading and writing new constructs. A starter book like SICP thus does not use this prominent feature: it does mostly NOT define & use macros.

Authors need to learn how to design new macro operators.

I think it is important distinguish the necessary from accidental complexity.

Macros are typically more difficult to understand but if done well that would be because they are sinking complexity from a bunch of code.

For example, if a macro implements variations of repeating construct, you are removing those variations from your entire codebase and putting complexity of dealing with that into a single macro.

Now, the issue is when you start creating accidental complexity.

For example, there are various techniques that reduce overall complexity just by being consistent in how you do things. Using only hygienic macros or writing macros consistently in a way that allows the reader predicting what they do reduces a lot of perceived complexity.

If the reader of the code does not have to understand the macro to be able to more or less predict what it does and what are basic guarantees it provides, it reduces a huge amount of complexity when you try to read and understand.

If writing a macro can be compared to writing an operator in a language then all other language design rules apply. It would be a bad language that constantly surprises the user.

Unfortunately, Lisp code tends to be much more abstract with lots of complex, custom operators (macros) as building blocks. If these building blocks are not clear and cannot be relied upon (ie. you don't understand how they work and can't predict what they will do) then this is much more damaging to trying to understand how the codebase works than writing unclear functions.

> sinking complexity from a bunch of code

Sometimes macros provide domain-level constructs. This creates very dense code - which can improve code understanding.

One thing I would recommend to put extra effort into macros, especially with these aspects:

One should write documentation what the macro expects and what it does. This way this has not to be inferred from reading the often quite complex code. Document the implemented syntax.

The macro should check the syntax of the forms. Provide error messages for rejected forms.

With those things in place, macros can absolutely be a net positive for code understanding. And if I saw that being done with any regularity at all, that would be great.

As I get older and more jaded, though, am coming to think that, in an office setting, it is a mistake to choose a language is optimized for maximizing the effectiveness of a thoughtful programmer. It's much more valuable to minimize the damage that can be done by a careless programmer.

One of the most sobering realizations of my career was that the most effective way to become your team's 10X programmer is not to work in a way that lets you be 10X more efficient than your colleagues. It's to work in a way that forces your colleagues to be 10X less efficient than you.

That is exactly what I have realized after decades of work.

There is a cap on how much work you can do individually. There is no cap on how much damage or improvement you can cause to other people you work with.

It does not matter how clever you are with your code if nobody else is going to be able to continue that work or support its results.

For this reasons I have developed following, informal rules I try to follow:

1. It is ok to be clever with the code if I am going to be only one to ever read or use it. This can assumed to be always false for production code. I limit my cleverness to PoCs, adhoc tools and emergencies.

2. Always think how other, and especially junior members of the team are going to develop and support my product. Is it too complex for them to follow? Is there something that can be done so there is less chance they are going to misuse it?

3. I use my cleverness to try to develop simpler and more reliable products. Simplicity is defined by how much work junior member of the team is going to have to expend to understand it. Reliability is defined by how likely it is to fail in face of junior team members operating / developing the piece of code.

4. If team members have trouble understanding or working with my products it is always my fault. Figure out how to make the product simpler or at the very least provide training and ensure they understand how it works.

It feels like Lisp scales vertically and Java scales horizontally.

Java scales for number of developers, its idioms and ecosystem make it easier to have tons of developers on a single project.

Lisp scales on the size of problem that a small close knit team can solve.

Things like the macro system in Lisp encourage defining your own problem specific DSL, which is great for individual developer productivity, but bad for involving lots of developers.

Like the way those defclass and defmethod macros totally destroy programming at large.

> In a Lisp, though, everything looks more-or-less homogeneous. ... you can't quickly tell whether some thing you're looking at is data, a function, or a macro, or what. It all blends together into a disorienting fog.

I wonder if this is a feature of Lisp, instead of a defect.

When you call up a variable or a function or macro, the intent is to return some kind of result. If variables, functions, and macros all look alike, and then maybe it can be weaved more seamlessly into the code to create its own style.

You can also think of a variable, as a function call with zero parameters.

> you can't quickly tell whether some thing you're looking at is data, a function, or a macro, or what

First of all, there are elements that are unmistakably data: "abc", 123.0E-13, #(1 2 3), :keyword :symbol.

The ambiguity is that a compound expression headed by a symbol could be anything. In order to to be fooled, we just have to read the entire top-level form from the outside in.

Well, that's what language is. For instance, if I'm saying "Bob believes that the Earth is flat", but you only catch my speech starting on at word "the", it looks like I'm saying that the Earth is flat. To get the real meaning (the belief is attributed to Bob), you need the whole sentence.

Or, until you hear the "nai" at the end of a long Japanese sentence, you have no idea that it's going to be negated.

The question isn't whether it is possible to write shitty code in a given language. Every language can be used to write shitty code.

The issue is, given a person that knows the language does not know how to structure their code, what is likely going to be the outcome with regards to maintainability.

My production experience is mostly with C, C++, Python and Java.

As an example, consider typical Java backend application. Even novice developers would typically learn Spring and create applications that contain controllers, views, repositories, services, etc.

They (I mean novice developers) would be constantly repeating rules they don't know why they are following, but heard it is important or seen it as popular. So maybe things like you should have your objects accept injected dependencies.

It does not mean they will know how to use these or even that they will use these productively to create more readable code, but what this does is creates code that has at least some structure.

The code might have huge amount of duplication and redundant or unnecessary constructs, but you will be able to move around, understand what you are looking at (okay, this is controller so I know how this most likely works, this is database layer so I know what I can expect, etc.)

On the other hand Clojure imposes exactly ZERO structure on your application. That is powerful but only if you know how to structure the application yourself, know what kind of choices you need to make and know ins and outs of various options.

If you have no experience creating structure, have been Java dev all your life and have always relied on structure that was given to you and never thought about it until today, you are likely to produce absolutely unmaintainable mess that nobody is going to be able to figure out.

> The issue is, given a person that knows the language does not know how to structure their code, what is likely going to be the outcome with regards to maintainability.

> My production experience is mostly with C, C++, Python and Java

> On the other hand Clojure imposes exactly ZERO structure on your application. That is powerful but only if you know how to structure the application yourself, know what kind of choices you need to make and know ins and outs of various options

All I can say is I have Java, C#, C++, JavaScript professional experience and also Clojure.

From my experience, I've noticed that actually I can navigate better even a messy Clojure codebase, because the language is actually quite simple. Even if someone plasters poorly thought out macros everywhere, the rules of macros and macro-expansion are very simple and clear. I can easily figure out what they do and then start to make sense of the mess. And the REPL allows me to very quickly explore everything that is structured confusingly and hard to read.

This is not true with messy C++, C# or Java from my experience. When Java gets messy, you are now dealing with pre-processors, custom annotations, XML/reflection, magic strings, hidden circular dependencies, and ton of coupling, and those can really get out of hand. There's no simple systematic method to unraveling the mess, like there is in Clojure.

And on the data side of things, this is also true. Clojure data-model being immutable also means it can systematically be unraveled. Again, in Java, C# and C++, you have so many hidden data dependencies that can trip you up, realizing the spread of the shared data across a messy code base is quite difficult, in Clojure, the only challenge is figuring out its flow, but not its sharing.

But, to caveat, all the Clojure services I've worked on were developed while I was lead. So it's possible that having me around managed to prevent getting them in a place that cannot be salvaged. And similarly most Java, C++ or C# projects I've led also never ended up in such a rot for as long as I was around. Where as the Java, C++ and C# projects I've found to be unsalvageable mess I've generally inherited from others or people before me. I haven't yet had to inherit a Clojure project where I wasn't involved with from the start. I do wonder what would happen then. My best experience here is open source, till now, most Clojure open source code base I've explored I've found simple to unravel, but maybe open source has a higher quality bias.

Edit: One last thing that keeps me hopeful is Emacs, probably the oldest most distributed development project ever using a Lisp, and I find it pretty easy to understand its code base and add to it.

You can write Perl in any language. Lack of expressive power in the language can also lead to a disaster. Sure, Java maybe doesn't have a fancy typesystem and macros, but then the same kind of clever developers use runtime reflection and bytecode manipulation which tend to end up with just as big disasters if not bigger.

Exactly my experience. Most Java code base end up using all kind of clever tricks like you said, runtime reflection, bytecode manipulation, source code pre-processing/generation, pushing all logic to configuration, abuse of compiler annotations, etc.

These actually create a bigger disaster, because unlike Lisp macros or higher order functions, they have even less structure, and are completely non-standard. I'd rather people use well thought out mechanism for extension and "cleverness" than some poor ad-hoc variant to do the same.

> You have to be really incompetent to write a typical Java service in a way where it is no longer possible to develop it at all

I've seen multiple Java services reach this state and require complete rewrite as the only path forward. I think you've just had one kind of experience and are making conclusions out of it, but in your case, I don't think the programming language is a factor. You just happened to see some failed Clojure projects, as I happen to have seen failed Java projects.

Yeah...or where it's possible to develop, but a feature that -should- be all of a week, is a multi-month effort.

In fact that seems like the norm for any Java project that has been ongoing more than a year.

> You have to be really incompetent to write a typical Java service in a way where it is no longer possible to develop it at all,

Disagree. Anything can be made write-only.

> but in Clojure it is just enough to get couple of people that are intelligent enough to write macros but not experienced enough to understand the dangers of lack framework forcing the structure of your application.

Sentence doesn't parse on multiple levels.

Agree with him. Coming from an OO background I always cringed at the 15,000 line classes with 2000 line methods. Side effects everywhere.

In my naivety I thought functional programs with their emphasis on lack of side effects could help. I then encountered a project that was totally functional but written entirely by people with no functional experience. The entire application was unmaintainable even in the most basic parts.

> Doesn't parse

> Macros

Macros modify code structure at runtime so obviously that is fraught with danger.

> Framework forcing structure

Lisp languages in general have a do it yourself attitude not found in others. While there are frameworks it's common to find projects that have invented from the ground up entire web frameworks, db access and orm.

Imagine if every project you encountered in Java reinvented Rx Java, Hibernate, and Spring.

I'm a big fan of Racket, and while I don't write very many macros, my understanding is that in both Common Lisp and Racket, macros are essentially a compiler pass, with no modification at runtime.

https://docs.racket-lang.org/guide/macros.html

> Macros modify code structure at runtime so obviously that is fraught with danger.

You may have inadvertently misspoke, but macros operate at compile time, at least in the Lisps I have used.

If one uses a Lisp interpreter, macro expansion may happen at runtime.

CLISP example:

  [2]> (defmacro add2 (place) (print 'add2) `(incf ,place 2)) 
  ADD2
  [3]> (let ((a 1)) (dotimes (i 4) (add2 a)))

  ADD2 
  ADD2 
  ADD2 
  ADD2 
  NIL
  [4]>

As one can see the macro form is expanded four times at runtime.

This is a confusing example, because in a REPL steps of compilation and evaluation are interleaved.

Indeed, can you write a program for CLISP that works like this:

- takes one command line argument (a file name)

- reads in the given file, interprets it as Common Lisp code, expecting it to deliver a definition for the add2 macro

- then runs

   (let ((a 1)) (dotimes (i 4) (add2 a)))

> This is a confusing example, because in a REPL steps of compilation and evaluation are interleaved.

The CLISP REPL does not compile, thus it can't be interleaved.

> then runs

It will still be interpreted and the macro will still be expanded at runtime.

I don't know Lisp well, but I remember PG saying in one of his books or essays, that Lisp is a language in which you can compile and run at read time, and the other two possibilities, too.

In full Lisp "compile time" is part of application execution.

Now, Clojure is kind of impaired Lisp because it is written for a VM that was not intended to be used this way and so this is not that much pronounced (but you still get REPL, etc.)

Is there anything lacking from the JVM beyond tail-call elimination that Lisp needs? I know Clojure is very slow to start up but my understanding is that this is because it uses the JVM very inefficiently, and they don't seem to care much.

Macro's are evaluated inside of 'defun, however, 'defun is evaluated during runtime (when the .lisp file is being loaded into the implementation) so base698 is technically correct.

Excepting that it is not really "fraught with danger" unless you are redefining macro's that have already been expanded and cached in function definitions.

Yes, same thing noticed here. Incoherent sentences above, in your parent's comment.

Have you looked at Nubank? Largest neobank in the world by users and valuation, it's a clojure shop to such an extent they literally bought Cognitect

No affiliation, just a satisfied customer

There are examples of successful companies using any programming language. It doesn’t prove anything about the quality of the language. The world still succcesfully runs on COBOL. That is not a good enough reason to pick COBOL for your next project.

The difference here is that Nubank attributes part of their success to the use of Clojure.

I've read a few times that Java and enterprise languages are so horribly tedious for the purpose of slowing down armies of monkeys to avoid catastrophic design.

ps: what kind of clojure projects do you have in mind ?

That's a common meme but there's not much evidence for it.

Java has the syntax it does because it's based on C++. It is verbose because (a) C++ is verbose and (b) it insists on everything being inside a class. The latter is a reasonable choice given that the underlying VM needs some unit of linkage and scope ... sort of like criticising C and C++ for requiring that everything be inside a function, but there are reasons for doing it that way.

It's true that Java's designers are very conservative, and this is partly because there are many users who appreciate that their skills don't get obsoleted all that fast (a lot of whom are in enterprise roles). But that's more of an accident of a commitment to backwards compatibility and historically long release cycles. It's not like they set out to make an 'enterprise language', they didn't. It was originally meant for the embedded TV set top box world.

I really read that big companies, for logistics reasons, prefer verbose, boilerplate-full languages because it slows down mistakes. Not only because the pace of evolution in the language is better for long term careers.

I don't think that's really true. I think it's more the case that the sort of languages that have the features large teams need have tended historically to be verbose, and some people conflated the two together. For instance lambdas made Java less verbose and there was no pushback from enterprises, which you'd have expected if they actually loved verbosity. Kotlin is also doing great in big companies and conciseness is one of its selling points.

> This underlines the fact that Lisp projects can very easily end in spectacular disasters.

I've been following HN since 2014 and haven't seen any submission about such a thing.

The closest was some article or comment from someone who worked at Cycorp, about some horrors of the legacy code base.

Here it is: https://news.ycombinator.com/item?id=21783828

"... the biggest mess I have ever seen by an order of magnitude"

:)

Not surprising since LISP isn’t used much.

Still, for some example companies: https://github.com/azzamsa/awesome-lisp-companies

How do you find REPL integration in Common Lisp compares with Clojure? I find myself blissfully productive in Clojure, and I wonder if it's the same for all Lisps

Common Lisp invented the REPL, Clojure only provides a subset of it.

Have a look at, "The Interlisp Programming Environment",

https://www.computer.org/csdl/magazine/co/1981/04/01667317/1...

Or online demos like https://youtu.be/OBfB2MJw3qg for Symbolics.

Ultimately you can try the community versions of Allegro and LispWorks.

https://franz.com/enterprise_development_tools.lhtml

http://www.lispworks.com/

Yeah the REPL in your typical Common Lisp is much more integrated than nrepl/cider in Clojure. Things like restarts and the debugger are built with the REPL in mind.

> This underlines the fact that Lisp projects can very easily end in spectacular disasters.

This is not specific to Lisp, it happens to all dynamically typed languages.

And this is why "holding invariants in my head" doesn't scale. Your objects have a type. Why not put it in the code and ask the compiler to verify it, so that future developers on that code (including yourself) will have an easier time reading it and evolving it?

Common Lisp allows to annotate code with type information, which can be checked at runtime. Also it's object system works on classes and thus generic function have arguments for those classes.

Something like Common Lisp is a dynamically typed language, but where implementations can use knowledge about types, either at runtime or even sometimes at compile time.

SBCL

  * (defvar *foo*)       ; a global variable *FOO*
  *FOO*

We tell SBCL that the type of the variable is an integer between 0 and 10.

  * (declaim (type (integer 0 10) *foo*))
  (*FOO*)

Now we try to set it to 30:

  * (setf *foo* 30)

SBCL detects the problem:

; in: SETF FOO ; (SETF FOO 30) ; --> SETQ ; ==> ; (THE (MOD 11) 30) ; ; caught WARNING: ; Constant 30 conflicts with its asserted type (MOD 11). ; See also: ; The SBCL Manual, Node "Handling of Types"

That's amazing. I want that.

I like how clear it is about what it's for, which is the sort of thing one does all the time in other languages that lack a vocabulary for it. I'm going to assume you could put any predicate you wanted in there?

An excellent analogy would be the metric system vs imperial measurement system, at least in the USA.

A productive and profitable country of feet and pounds and inches, but significant progress is usually made in labs full of meters and liters.

One could say the constant translation problems between the systems are something like an intelligence filter which provides its own rewards when not actively translating.

> I have trouble finding enough people, mature enough to be able to even propose working projects in Clojure.

It has nothing to do with maturity and very thing to do with maintenance time and maintenance costs of using languages that do not have a robust pipeline of talent.

Using languages like closure do give you a great amount of job security if you manage to sneak it in though.

> Using languages like closure do give you a great amount of job security if you manage to sneak it in though.

I am not after job security. In fact, I treat "I am not after job security" as one of the important points when presenting to potential employers.

I try to do good job and think for the outcomes for the employer first.

I value trust and I try to make it clear to my boss that they can trust me to always work with best interest for the project and company.

For example, I always make it clear when I make mistakes and then we try to figure out how to solve them.

I think, overall, this brings much better job security than trying to sneak technology to create a project that only I would be able to maintain.