Clojure is Capable

The best answer I could think of was, "don't manage huge projects." Instead, divide projects as early as possible in development into supporting libraries, each capable of being independently tested and maintained.

In my experience, applications developed with this attitude coalesce themselves into a small ecosystem of mostly generic library code with a domain-specific apex "application" entrypoint to bind them.

In an application factored like this, type checking isn't something you want as much anymore, since the constituent libraries interface with one another at a level above. They present and consume "public" (intra-application) APIs, the behavior of which are defined not just by types but by at least documentation and tests. The thing that static types let you do -- have huge amounts of code, woven together with structural dependencies -- is no longer a thing you need or want. It would seem bad, actually, since what you'd have is an inscrutable mass only a machine can understand.

I haven't worked enough with large non-Clojure codebases to say whether or not this approach is superior, but I can say that it scales. Another nice thing about it in a business setting is that supporting libraries lend themselves to open-sourcing, and a relationship with an open source community is a good thing for a company's development efforts/hiring/contracting in my experience.

To answer your specific questions:

* I don't think I write more unit tests (I should probably write more tests in every language I use)

* I don't have any experience with clojure.spec

* All of the features of Lisp work together to make it awesome, including macros. In particular, in the context of an "ecosystem of libraries" approach, macros are a fantastic tool for exporting a DSL from a supporting library. They're also use for DSLs in the apex code, where application-specific behavior can be defined in terms of a DSL.

* I don't really wish I had static types - I love Lisp the way it is. But I have found various languages for structural checking useful with certain kinds of sub-program, like receiving data from the outside world, and for writing compilers. If you write a compiler as a series of transformations, it's nice if the shape of the code between transformations is well-defined somehow.

You're definitely smart enough to use Lisp. I think the key to success is not necessarily smarts, but attentiveness to modularity or cognizance of the difference between "library" and "application" types of code.

* There's a Paul Graham essay related to this idea: http://www.paulgraham.com/progbot.html

* I think this approach is applicable not just to Lisp, but maybe to any dynamic language with a sufficiently rich set of generic collections and with the concepts of a "module" or "package".

* In refactoring, I almost always make mistakes in Clojure that Scala will catch. I feel empowered rather than limited by the Scala compiler 9 times out of 10.

* Homoiconicity makes me feel great, but I find I don't actually miss it (directly) in Scala. That said, Scala has a complex grammar, and I miss the overall consistency of s-expressions.

* I always miss static types. I'd love to try the new spec improvements in Clojure 1.10; debugging mistakes prior to that are a nightmare. I don't need static types to be happy, but I do need real tooling support when things go wrong.

Clojure has a lot of lifting power and I would like to keep using it. I find that once I get past type errors, programs work very well. Why would I spend time getting past those if I don't have to?

During refactoring, do you feel more flexible, or less? I can see both sides to the argument. On one hand, static types are more rigid, and getting something to compile during a refactoring can be a pain. Often for the right reasons, but often it's due to some annoying technicality in the type system. But once you get there, "if it compiles, it runs" is a cliche that is pretty accurate in my experience. On the other hand, with a dynamic type system, you can slowly evolve a code base and see incremental progress during a refactoring, but I can never get over the dreadful feeling of "what did I forget?". Has clojure.spec worked out in terms finding bugs during refactoring/maintenance and general QA?

I think there is good and bad to this, but in my experience, using a lot of types seems to be an anti-pattern. Since most web-based projects really only deal with strings, basic numbers, lists and dictionaries, I really don't see the entire benefit, even in strongly-typed languages. I work with typed languages as well and generally feel ambivalent about types.

A bad refactor is a bad refactor and bad code is bad code no matter what language you are using. If you can't understand what the code is doing, don't touch it until you can test against it or test against what it's really supposed to be doing. Types don't alter this fact.

Do you make use of the Lispy-ness (Homoionicity, macros, code is data, etc ...) or is it generally an anti-pattern to use these language features in daily development?

Homoiconicity is the very definition of Lisp, so there is no way to work around that.

Macros are generally eschewed in Lisp. It's glaringly obvious when and if you need to use a macro, and honestly, most people can go their entire career without pushing a macro into production.

Code is data is pretty hand-wavey in my opinion. All it's really saying is that, if you call a function with a value, you get a value back. There is no mental quantum leap needed for this.

Do you wish you had static types? If so, have you tried gradual typing like Typed Clojure? And if you don't miss types, why? I remember someone saying of Typed Clojure "We went to Clojure to get away from types!". And I'm still trying to understand the mindset.

I don't think the community has any strong for-or-against opinions on types. In general, it's bad practice to use multiple-type functions. The perspective that's generally embraced is that each function does exactly one thing, which is why you see so many Lisp projects with many short functions, and if you pay attention, the functions aren't doing a lot of acrobatics based on possible types.

What the community doesn't like is someone coming along and demanding something that isn't compatible. This is a little more nuanced, but it boils down to: "Add types if you want me to use it" or "Create a RoR system for me to use it." The community would rather you take the time to embrace Lisp and really try to understand what it is attempting to do before pushing your judgment on it.

I often have the feeling that I'm not smart enough to use Lisp. If I had 2x the brain power, I would be able to use macros to conjure up some black magic in 100 lines what would take a mortal 5k in a lesser language. But even if I manage such a feat, I couldn't imagine bringing on another developer to that codebase.

Anyone can use Lisp. Most of what you do is very simple. The entirety of Clojure is written on a single webpage. https://clojure.org/api/cheatsheet

No one is building up systems with loads of macros and wild abstractions that cause a 5KLOC code base to shrink to 100 lines. This just isn't possible, and anyone that claims it is has no understanding of what a macro actually does.

> During refactoring, do you feel more flexible, or less?

Both. More flexible because in Java you have 5x more code (was worse before lambdas). Less because I sometimes look up the code and don't know (and can't immediately infer) the types anymore. Usually I have a handful of integration tests that show me immediately when there is a problem but I wish there was a way to show the inferred types in my IDE.

The problem is though usually only a big one for the one big state where you have lots of nested maps, vectors, arrays and so on and only if you modify the structure of this state. For smaller things, fixes are fast and easy, even though they still take 10 minutes while they could take 5. I feel it's kind of the price I pay to use Clojure but then again there's probably a reasonable solution to it - Clojure infers the types anyway, so why not show them somehow.

I use clojure.spec partly, often when I need to throw useful error messages (e.g., a user facing REST API). It's a little bit tedious but it's always a good decision in retrospect.

> Do you make use of the Lispy-ness?

Yes and no. on the positive side: I don't ever need to look up the syntax of anything (I still make mistakes in Java switch for example) and using case and multi-methods I can really interpret quickly a DSL in a web server (e.g., binary arithmetic operations on OHLC bar time series) and I wouldn't want to implement that (safely) in another language.

> Do you wish you had static types? If so, have you tried gradual typing like Typed Clojure?

I tried for a brief two days gradual typing and didn't get it. I wish I had static typing exactly when refactoring or debugging.

> I often have the feeling that I'm not smart enough to use Lisp.

Oh, I'd even say the opposite, i.e., I am smarter because I use Lisp.

I know Sapir-Whorf is "deprecated" but I feel I am thinking faster because I am thinking in much less convoluted structures compared to when I programmed in Java.

- During refactoring, do you feel more flexible, or less?

A lot of it depends on how the code has been written. There are definitely parts of our code base that I really dread going near and there have been plenty of times when I have broken stuff by making a change without fully understanding the code. It is quite easy to write incomprehensible lisp code.

With Clojure being more pure and opinionated things are probably not quite as hairy as it can get in CL. But if you write your code well CL can be fairly easy to refactor. Generally preferring to use CLOS (the object system) rather that just property lists and hash tables will give you some typing information that can make life a lot easier. Avoid nested lists as data structures - there is nothing worse than code that does stuff like `(dosomething (cadar list) (cddar list)`

- Do you make use of the Lispy-ness?

Macros tend to only be used for writing framework code. There is generally no need to use them when writing business logic.

The aspect of Lisp that is often not mentioned is the iterative development. When code is data it becomes possible to evaluate code against a running system. This is, in my opinion, the nicest feature of Lisp. No compile times, you can change code on the fly, evaluate and test functions as you are writing them. Development becomes a real conversation with the compiler. You can even connect into the live server if needed to debug problems.

- Do you wish you had static types?

Yes.

1. "Has clojure.spec worked out in terms finding bugs during refactoring/maintenance and general QA?" - Spec worked out as a much more useful library than we expected. You could describe any data structures with it, then validate, generate, regenerate it. I think spec is a must have for any Clojure programmer from 2018.

2. "Do you make use of the Lispy-ness" Macros are super useful sometimes, but in general we try to avoid them, because we prefer to solve everything with just pure functions. Homoiconicity and the "code is data" aspect are must haves in any programming language after you realised their power.

3. "is it generally an anti-pattern to use these language features in daily development?" macros are feel like cheating sometimes, and we fear the consequences, but homiconicity is useful for us on a daily basis.

4. "Do you wish you had static types?" We prefer to create our own "types" up to complex data structures. A built-in type system seems like a downgrade from here.

5. "I often have the feeling that I'm not smart enough to use Lisp." I think you are just scared from the unknown, Clojure is an alien technology compared to most of the currently popular languages.

Any kids could learn programming in Clojure due to its simplicity and the REPL driven workflow. Its an interactive experience and very likeable.

I'm not sure I can answer your questions. While I do use CL at work for smaller jobs I've only used it for bigger personal / solo projects, so I'm not sure that counts.

About refactoring: it's not a thing that comes up very much. Not in the sense that it was a thing like when writing enterprise Java. Maybe because I try to keep my code composable I don't have to change it in many places? But this isn't limited to Common Lisp for personal projects: also bigger projects with multiple persons at work in C++, Python, etc. refactoring isn't such a big thing that I'd use it as an argument when chosing programming languages.

Also once your code settles down, when you're going from the exploratory phase to the "let's make this ready for maintenance"-phase, CL has the option to annotate functions with types. Not at the level of Haskell but close enough to catch most things.

That's what I like about CL: it's very practical and multi-paradigm. It doesn't force into a specific paradigm like object oriented or whatever fad comes next.

I never have that feeling "what did I forget?". I'm sure I do forget things, but by building up code function by function and testing those functions one gets a pretty good feel for the code base. BUT, I've never worked on CL projects with multiple people, so I get where you come from.

I love being able to use macros and the Lisp syntax is one of its main appeals to me. It is so simple and straight-forward. Algol-like languages with their curly braces, brackets, statement-ending, etc. etc. feel like such a chore. I try not to overuse macros, in fact I rarely use them but they're there when I need them and make the language very powerful and, if used correctly, more readable and easier to understand.

"Code is data" … Do you mean something like this?: http://www.moserware.com/2008/04/towards-moores-law-software...

I guess I've been using Lisp long enough that I don't quite get what it means but I use it automatically.

"Do you wish you had static types?" — Not really, like I said one can add garnish to the CL code to get something that is good enough, but even then I rarely use it. However, I only use it in mission-critical stuff and I don't write much of that.

A hardcore Clojure developer will probably tell you that Spec is, in some sense, even better than having a static type system. I haven't used Spec, so I can not say. But I do find, when I'm writing Clojure code, I write pre-conditions on every function, so every function has a contract. This goes a bit beyond what you can get from static type checking, in the sense that I'm testing all of my assumptions, including key names in maps, or at least, this is less work.

There is also the advantage of testing against real world data, instead of mock data. I've had the problem, when using Java, that all the unit tests passed, but the app died in the real world, because the real world data (perhaps from a 3rd party API) was different from our mock data. In such a case you need run time contracts to help find the problem, and static types offer no help (depends on the project really, but the more your app interacts with 3rd parties, the less useful static types are).

I'm not proposing this as The One True Way, but I would propose this as the Really Easy Way to write sane code. I don't have to think about it. It's like writing a comment for the function, but it gets enforced and can not possibly go out of date without giving an error. It documents everything the function is expecting.

An example:

    (defn push-sentence
      [current-database-connection permanent-id map-of-sentence]
      {:pre [
             (not (nil? permanent-id))
             (not (nil? map-of-sentence))
             (map? map-of-sentence)
             (:sentence map-of-sentence)
             (:belongs-to map-of-sentence)
             ]}
      (mc/update
       current-database-connection
       "reports"
       {:permanent-id permanent-id}
       {operators/$push {:sentences map-of-sentence}}
       {:upsert true}))

    (defn push-transcript-processed
      [current-database-connection transcript]
      {:pre [
             (not (nil? transcript))
             (map? transcript)
             (:permanent-id transcript)
             ]}
      (mc/update
       current-database-connection
       "reports"
       {:permanent-id (:permanent-id transcript)}
       {operators/$set {:processed :true}}
       {:upsert true}))

             (not (nil? map-of-sentence))

             (:belongs-to map-of-sentence)

I keep the functions short, which feels natural in Clojure.

Again, this isnt' the One True Way. Realy Clojure programmers will tell you to use Spec, or maybe Typed Clojure.

But I will say, heavy use of pre-conditions gives me the right balance between the contract enforcement I want, while still giving me the flexibility of a dynamic language. I like Java, but often the verbose quality of a static typed language is just not something I need.

- refatoring

I don't really find it to be problematic with dynamic typing. I tend to rely on specification tests and spec to test functionality at the API level. If the API behaves according to the specification, then the code can be said to be working as intended.

- homoiconicity

Absolutely, my team gave a talk about one of our projects recently https://www.youtube.com/watch?v=aXBe6hoi-Mw

The project heavily relies on homoiconicity as the business logic for each form is encoded using EDN and stored in the database.

- static types

I've primarily worked with statically typed languages prior to Clojure. I've worked with Java for close to a decade, then used Haskell for about a year, and some Scala.

My experience is that dynamic typing is problematic in imperative/OO languages. One problem is that the data is mutable, and you pass things around by reference. Even if you knew the shape of the data originally, there's no way to tell whether it's been changed elsewhere via side effects.

What I find to be of highest importance is the ability to reason about parts of the application in isolation, and types don't provide much help in that regard. When you have shared mutable state, it becomes impossible to track it in your head as application size grows. Knowing the types of the data does not reduce the complexity of understanding how different parts of the application affect its overall state.

My experience is that immutability plays a far bigger role than types in addressing this problem. Immutability as the default makes it natural to structure applications using independent components. This indirectly helps with the problem of tracking types in large applications as well. You don't need to track types across your entire application, and you're able to do local reasoning within the scope of each component. Meanwhile, you make bigger components by composing smaller ones together, and you only need to know the types at the level of composition which is the public API for the components.

The other problem is that OO encourages proliferation of classes in your code. Each class is its own DSL with an ad hoc API, and knowing the behavior of one class doesn't tell you anything about that of the next. Keeping track of that quickly gets out of hand. With Clojure, you're working with a common set of data structures and using standard library functions to manipulate them.

REPL driven development also plays a big role in the workflow. Any code I write, I evaluate in the REPL straight from the editor. The REPL has the full application state, so I have access to things like database connections, queues, etc. I can even connect to the REPL in production. So, say I'm writing a function to get some data from the database, I'll write the code, and run it to see exactly the shape of the data that I have. Then I might write a function to transform it, and so on. At each step I know exactly what my data is and what my code is doing.

Clojure idioms put those black magic macros or even complex ones in the "last resort" category. It is very much focused on representing problems as data then composing general functions to work with them. That is ranked as more elegant than the macro driven dsl style and similar ones. They are still used and are very useful.

I find types are one thing among many, all affecting each other, when it comes to how productive I am and how much I enjoy it. Clojure is unreasonably effective and spec solved the class of problem that I would have liked to have gradual types for so I don't miss them no. Gradual typing is cool though, I'm sure I'd make use of it.

Furthermore, I think that writing software using monolithic design is bad practice. It creates coupling within the project, and it makes it harder to reuse code. It's harder to onboard new people, and so on. However, the biggest problem is that it creates a lot of communication overhead for the team. The more people you have working directly with each other, the more emails, meetings, and other kind of communication you end up incurring.

Splitting things up into small components that are each maintained by separate teams of 5~6 people is a far more productive way to maintain large software projects than to create monoliths.

Clojure + REPL is save states / time travel

(could be clearer though)

- write code

- compile & run, see if it works

Just keep repeating this until you achieved what you want.

In REPL this is exactly the same you just have a much faster way of trying out a concept.

I think this is pretty simple to conceptualise.