Tour of our 250k line Clojure codebase
tech.redplanetlabs.com
tech.redplanetlabs.com
This is a great insight no matter what language or framework you're operating in. Laziness has its virtues, but invariants, validity checks, run-time type checks, etc. should all be performed as early (and often) as possible - it's much easier to debug issues when the are proximate to an object or structure being created or modified, then when that data type is being used much later.
For example when writing an api endpoint to create a task I would typically deserialise the json into a CreateTaskRequest. If the object is created without exceptions I can be sure it is valid. CreateTaskRequest implements the ToTask interface. The service layer takes only objects of this interface and converts into a Task object that gets persisted. The persisted Task then gets converted into a TaskResponse so only valid JSON comes back out.
Lots of classes and interfaces but they are all small and with a single purpose.
Indeed. This feels like a just-before-the-moment-of-realization situation.
The endless cycle between "more dynamic" and "more static" continues it seems.
I wonder if there is any correlation between experience in the field and static vs. dynamic vs. "fail fast dynamic".
(I'd say Erlang falls in the latter category and it has a pretty good track record for reliability, but so does Python. It's an imperfect axis for sure.)
That's the side effect. You ultimately end up with more code and not less even though it saves you from type checks.
There are trade offs for both.
Hell, in the extreme, perhaps types in general could be generalized as a set of tags?
(See also, https://news.ycombinator.com/item?id=27168893)
For example a MutableList<T> can be dropped down to a List<T> which is not mutable. And a generic conversion from Collection<T> to NonEmptyCollection<T> should be trivial to write as an extension method.
TaggedType<std::vector, NonEmpty, Sorted>
TaggedType<std::vector, Sorted, NonEmpty>
fun <T> doSomething(values: T) where T: Sorted, T: NonEmpty {}
<T extends Sorted & NonEmpty> void doSomething(T values)I'm not even sure if my "types as a set of tags" idea makes much sense - perhaps it decays to what is typically understood as types. Or perhaps it hits computability problems.
I did some mental experiments on a "set of tags" type system last night, and I quickly realized the complexity will be around deciding when to keep a tag (property) on a type, and when to drop it. With a set of tags being open, a function could not possibly know about them. This creates a problem.
Consider a function like: Sort([List[T], ...], [Function([T, T] -> Bool)]) -> [List[T], Sorted, ...]. Takes a list and a comparator, sorts the list, attaches a "Sorted" tag to return type, retains all other tags. Looks like a reasonable definition. Let's look at the use cases below (with P being some applicable predicate):
- Sort([List[Int], P) -> [List[Int], Sorted] -- OK!
- Sort([List[Int], NonEmpty], P) -> [List[Int] Sorted, NonEmpty] -- Awesome, exactly what I want!
- Sort([List[Int], Foobar], P) -> ...? Should it be [List[Int], Sorted, Foobar]? But what if Foobar designates some business rule that depends on the element ordering? How is Sort supposed to know?
Without knowing a tag, we'd sometimes want a function to drop it, and other times want the function to retain it. I have no idea how to solve this at compile time (or even at runtime, without crippling the tagging system).
Contradicts sorted foobar
Independent sorted nonempty
Independent foobar nonempty
A large part of the point of these systems is that you can write code which generalizes across the different subtypes (different units, tainted vs untainted values) yet it still passes through invariants by default (values calculated from tainted input are themselves tainted, a unit-quantified value multiplied by a scalar retains its unit, and so on).
I've done similar things with syntax trees in compilers. With multiple passes, you have a representation for the output of the parser, then you might do type annotation, rewrites due to coercion etc., symbol binding and overload resolution, constant expression evaluation, and so on in separate passes. If the input and output trees for each pass have different types, it's easier to keep track of what's going on, of what invariants hold for any given tree node, or indeed which node types are permissible to exist in the input or output of a pass (e.g. you don't want unbound overload calls after overload resolution).
One problem is that tags probably aren't enough when you're dealing with data structures rather than simple zero-dimensional values. Some methods shouldn't be called, or some fields shouldn't be accessed, if a value has a type with the wrong tag. But if you have a wholly different type, then you can end up reimplementing a lot of the same logic, once per type, simply to get the types to flow through.
One solution I've used a couple of times for the tree problem is tree grammars. That is, a data structure which encodes a description of a valid tree and can encode invariants like nodes of type T need to have attributes of type Y with values satisfying predicate P, and between N and M children of type U, V or W, and so on. The grammar is defined in terms of an abstract supertype, and the concrete subtypes have the grammar specific to their phase baked in. This is a hybrid between static and dynamic typing, a compromise necessary for languages like Java without much expressiveness in the type domain.
Exactly.
Personally I like that bit of extra code because it gives every class one reason to exist. There are no conflicts so the type system can be used fully without ambiguity.
I’ve always disliked the way early rails promoted fat models that combined serialisation, deserialisation, validation, persistence, querying and business logic in the same class.
My experience here is mostly from Kotlin which is a great language for this. Nullability, extension methods, (reified) generics, data classes, delegates, etc can all help reduce boilerplate.
While the total LoC written might be higher, the amount of written logic in which you can introduce bugs is less.
Some languages make this less likely (Haskell, Rust) but most mainstream languages will happily let you introduce this bug.
Without some kind of custom validation system in place (JSON schema, property attributes, etc), that doesn't tell you much. With most serialization libraries I've used the default settings would let you deserialize {} into any class without an exception and all the properties would just have the default value for their type. Using stricter setting gets you a little more but definitely no guarantee of a logically valid state. If I want to go even one step past what little validation static typing provides, and I usually do, I'd rather just take the type noise completely out of my data and move all validation to a single place.
My CreateTaskRequest class which is used for deserialisation only does not have defaults (unless intentional) so it throws when required values are not present in the source json. It also takes care of dateformat/uuid parsing. The output is always fully valid typed or it throws. It’s a 1-to-1 mapping on what is described in the API docs.
The service layer that takes CreateTaskRequest and converts to Task for persistence is where the business logic validation happens (valid foreign keys, date ranges, etc, unique checks). Cleanly separated from deserialization.
For reference: I use Kotlin with Jackson which has great support for this.
FWIW, this sort of thing can be done with a dynamic language, too. It's true that some static languages happen to be really far ahead of the curve with this sort of thing. But it's also true that some dynamic languages put you in a better position on this front than many of the most popular static languages do.
(And for those of us who really do love an intractable quagmire, there's always JavaScript.)
For my part, I tend to find this debate to be mostly a distraction, because the influence of the language's type discipline is quite small relative to the influence of the programmer's coding discipline. In the group I'm working with currently, the most committed fans of static typing tend to also be the ones who have the greatest tendency to assume, "If it compiles, it works," and proceed to check in glaring bugs. I don't bring that up in order to point fingers at static typing proponents (I tend to prefer static myself, though my preference is not particularly strong) so much as to point out that we should be wary of memes that subtly encourage us to become complacent.
var flag_is_unset = flag_is_set
I generally land more on the dynamic side of things, but there are certainly problem domains where I love types. The more closed and "mathy" the domain, the better I think types fit. I just wish it was less an all-or-nothing choice, and that people were less religious about it.
Optional static typing, like what you get in Typescript or MyPy, is interesting, but defaulting to dynamic and making static opt-in undermines a lot of the potential benefit of static typing. I don't think that it works the other way around, though. My hunch, which I am not particularly prepared to defend, is that, at least as long as you've got good type inference, defaulting to static and making dynamic opt-in does let you keep most of the practical benefits of dynamic typing.
I guess it's kind of like unsafe code blocks, also from C#: Pointers and weak typing can be very useful in some circumstances, but I'm generally much happier having the compiler try to guarantee as much as it can, and then sometimes be able to tell it, "Nah, hang on, I got this one."
Yes, there's certainly truth to this. However, I write a lot of TypeScript and it provides an _exquisite_ on ramp in that you can add types gradually and telling the compiler not to worry about it can be super useful, especially if you're dealing with poorly behaved third party stubs that may have diverged from the actual implementation. The fact that you can use TypeScript as a super-duper linter _or_ try to maximally leverage its powerful type system is a huge strength. It's also very helpful for adoption. As frustrated as I get sometimes with TypeScript's unsoundness and the lack of pattern matching, the fact that I get to use it instead of JavaScript (and have also got half the company using it!) is a huge win.
[1] the caveat is that it somewhat sucks to change these restrictions and therefore the types. Compilers can hopefully make these refractors easier, but they may still suck. In languages like clojure, you mostly have to try to write programs/tests to be resilient to any reasonable changes to the data structures that you can imagine.
In the ML-family of language, you have a few key things:
1. Primitive types like ints, bools, strings, floats, arrays, not much else.
2. Product types which are records/triples of other types
3. Sun types which are proper tagged unions of types. Including things like optional or result (aka or-error) types, and also list (= nil or cons) types.
4. Abstract types which are types whose representations are hidden. You can have an abstract type called “hour_of_day” which is secretly backed by an int but which you can only interact with by using conversion functions or eg something that adds two values (mod 24).
5. Polymorphic types: you can have a list type which can be a list of units or a list of floats but not really a list of a mix of arbitrary different things.
The idea is to represent with these types a model of the world in such a way that only valid states of the world can be constructed. A user’s bank account balance isn’t an int, it’s a positive_dollars and if you try to do a transaction to make it negative, that isn’t possible as you can’t construct a suitable positive_dollars value. This can have annoying difficulties for maintainability because it is tedious to invert or change a one-to-one or one-to-many relation (eg previously 1 tax_number per person, now 1 person per uk_tax_number and one or two people per us_tax_number) and hard to represent with types a many-to-many relation like “every person has at least 1 bank account, and every bank account is associated with at least one person, and the bank accounts associated with person A have A amongst their associated persons.”
The promise is that the type system makes the practice and correctness of these refactorings easier.
In Clojure, the data model is more like:
1. There is a rich set of primitive, atomic types, eg strings, ints, but also dates and symbols and keywords and fractions and Uris and so on
2. There are collections like lists (of logically unalike objects), vectors (of logically alike objects), hash tables, and sets.
Data is built up out of e.g. hash tables of keywords to objects. The language has a rich set of features for acting on these types so one can do a lot with hash tables, whereas in an ML system only a few operations are available with a record type (eg constructing, reading fields, maybe updating them) and functions that do general things with any record can’t really be written. Closure is a language for manipulating data in general more than a framework for writing functions to manipulate your small, strict data types.
Clojure tries to model the world in a metarational way, accepting that it is unlikely that one can write a strict scheme capturing all and only valid states and instead programs should try to allow for the possibility of extra or missing information. You don’t want to care about whether you have a us_person or an eu_person so much as whether your data has a :person/preferred-name field. Issues with relations come up less because those relations are not trying to be forced into rigid types (they may be enforced by a database though—another cultural difference between Clojure and ML-family languages).
Fundamentally, I think the differences stem more from philosophies about modelling the world than type systems.
It seems like people mistake the battleground for the countries. If you only saw the US and Japanese battles in early World War II, you'd think they were tiny nations of ships, soldiers and aircraft fighting over a handful of islands. It wouldn't be clear that there were full countries with big populations backing all this, and that they were thousands of miles apart.
So Rich Hickey gives a talk criticizing types, and a Haskeller responds with a blog post, and you think that types are the difference between the camps. But it's not. It's just that this is the spot that's close enough to home to reach, but contentious enough to fight over.
It's far, far superior to the idiotic God classes most Java devs tend to write - the ones covered in attributes that are sometimes populated, other times not, and annotations for ten different purposes. Just no. Don't use the same single class to parse incoming requests, persist records at the innermost db layer, serializing the outgoing json etc etc etc.
The sane way to do it is to have a dedicated class to parse incoming requests, another dedicated class that represents a validated and decorated record to be persisted (that doesn't have an id attribute, because it hasn't been persisted yet), another that represents a persisted record (with an id now, that is guaranteed to never be null), another that represents the outgoing response etc etc.
A common criticism of this style is that it's wordier, there are too many types, and it can be hard to follow if you're not used to it. The thing is, that's the price you pay for more accurately modelling what's actually going on at each step. The "use a single God class for everything" alternative is "easy to follow" only because it omits important differences that are actually there at each step, which doesn't mean they're not there - they are there, they're just hidden.
Django-Rest-Framework does that with view + serializer + validators.
A more recent example, with a leaner and cooler implementation is FastAPI, where type hints are also use to declare validation on your end point, while a model is used for serializing the response.
A class full of strings feels like a code smell - but that is the proper representation of serving the request. And the alternative is needlessly restrictive. Ultimately it feels like pointless ceremony.
You shouldn't get to the point of having a CreateTaskRequest with a 0 that shouldn't be there.
Is that substantially different from what you suggest?
For example, if a request has a field I didn't expect, that's not an error, I simply ignore that field because I obviously didn't need it. Likewise if requests start using a more specific schema, such as now sending only positive integers. If, however, the schema changes such that a field I rely on is dropped or changes type, my code will eventually fail anyway as soon as it makes an assumption about the data. Why not fail at the point of entry instead of saving it for later?
Having worked on numerous large-scale distributed systems, I strongly disagree (and I would think the author would too).
There is a tradeoff to strictness, which is coupling. You want to validate and type check as much as necessary—but not more. If you are over-coupled in a distributed system, it can make it difficult to change your system incrementally, as one must also do with distributed systems. A canonical example of this is closed enums, which don’t tolerate unknown values. Good luck adding a new enum value if every part of your system is strictly validating that. Each part of the system should validate what it needs in order to guarantee that that system can operate correctly, and no more.
Easier said than done. Each part of the system can't know what it needs, since you've just added a new thing it might need to contend with.
The point of closed enums is to force these checks, even if all you're doing is adding a new fall through to ignore it, since at least that means you've considered it.
Yeah it can. It needs what it needed before.
If you have a system that is shipping stuff to customers, it doesn't have to care about the new childhoodPetName Field that you've added for a different part of the system.
This is especially relevant if you handle any kind of description of the real world, e.g. in robotics or medicine, and have multiple distributed systems and codebases all communicating with each other.
The inability to gracefully ignore cases that don't fit the expected data model is a weakness most contemporary programming languages and type systems share.
It's a matter of correctness enforced at language level, and why languages like Rust's match statement enforces all checks at compile time by default.
By definition the field is ignored by your program, therefore even if there is any malicious text in there, e.g. raw html, the system in question can't do anything harmful with it as it is ignored.
Sanitation occurs on a type level, so if the field contains data of the type DangerousRawHTML, and there is a component in your system that just blindly renders any field besides those of type SanitisedHTML, then that is the location of your vulnerability. A type system like Rusts would also catch this.
Don't conflate "correct rust program semantics" with "correct real-world model semantics".
I don't think there's a silver bullet here, or some rule of thumb that can save us from these schema issues. It's messy and imperfect; we have static type systems that can help us locally but they are not global silver bullets. Unfortunately we still have to use our judgment to find the right balance, because the "perfect" solution requires knowledge of the future.
Actually this sounds quite horrible.
This is Nathan Marz we're talking about though; he's on the short list of people who can both accurately identify the need and create something workable to fill it. If he told me it was necessary for a particular problem domain, I (personally) would think long and hard before I disagreed.
It isn't much harder than regular core. `macroexpand' is your friend! Particularly when wrapped by your IDE[0] into a "macrostepper" tool, which lets you macroexpand into an overlay[1], step by step.
Ultimately, macros are just functions - if slightly peculiar ones (they receive unevaluated arguments, and their return values are treated as if they were the code at the point of invocation). You can unit-test them just like any other function.
That's not to say there aren't macros that are very tough to debug. But the problem isn't macro-heavy code, as in code containing a lot of macros. The problem are heavy macros - monstrosities with complex internal logic, expanding their inputs into large amount of code generation, hidden state, etc. Complex macros like these take skill to write[2], but they're also rare to see.
--
[0] - By which I mean Emacs, though I guess there are Clojure IDEs now too.
[1] - I.e. a read-only view replacing the code you're macroexpanding on screen.
[2] - First rule: minimize the amount of code within the actual macro definition, move all logic into functions to be called by the macro, and unit test those extensively.
what a company needs is good leadership at every level. not unique tools, and not "10x engineers".
a complete head case, nevertheless he's a million times more successful than I so perhaps I'm the one who's nuts.
This can go either way. Yes, it can be abused. DSLs can be very useful and powerful.
In any case, I think it is better to not to rush to judgment for this particular case.
You understand the use case well enough to criticize it?
Could work out if its essential to offering a 10x service, but I doubt the multiplier is that higher to warrant such an engineering cost of developing and maintaining another language on top of Clojure.
Also, there would be many layers to get to the machine code (Custom lang -> Clojure -> Byte code -> JIT -> Machine code), would this impact performance in a way that makes the program too slow and needs re-writing?
I guess they could make it work in any language but judging by the description clojure is indeed a great fit, due to the macro capabilities, flexibility and solid runtime via JVM.
The doto macro was a relief back in the days when Java APIs insisted on being designed around stateful setters and getters rather than the builder pattern, it allowed you to operate on such unfortunately designed objects in single logical blocks and simulating it all being an expression.
Proxy allowed you to instantiate anonymous inner classes implementing only the methods of an interface that you needed, the rest you could omit; in Java you have to put them all, empty, which necessitated that you use an IDE to generate them, and back then IDEs were not as nice as today.
Those two alone made interacting with contemporary Java libraries so much easier.
It also was convenient to go from Java collections to Clojure collections and vice versa.
-- Guy Steele
1) default immutability (same simple data structures used in every library -> ecosystem composes better)
2) portable code across multiple host platforms (jvm, node, browser)
3) metaprogramming
IMO, in 2007, immutability on the JVM was a competitive value prop, but in 2021+ it is nothing special. It is the combination of the three things which is a competitive value prop today. Metaprogramming in particular is a mostly unexplored frontier, because it is very hard to do well, and very easy to do badly. Default immutability is kind of a necessary starting point to do metaprogramming well.
Can you elaborate on this? What do you think has changed in that time to make it "nothing special"?
I've got some experience with functional languages, and a good amount with functional features in OO languages. Started learning Clojure this week, and was pleasantly surprised with how quickly I could get working projects up and running. Less upfront learning curve than Elixir, which was unexpected.
The Clojure team has always done a nice job expressing the rationale for specific language features, and these rationales often lean towards solving problems that system designers historically faced.
Oftentimes, I think folks think of modern languages in regard to their syntax, tooling, ergonomics, etc; however, to me, the more interesting benefit in adopting a modern language is in how its inbuilt features address design problems that earlier generation languages exposed.
For a real world example of what I'm talking about, you can google "clojure expression problem" and find compelling articles about how Clojure solves this with protocols.
Providing a toolkit for attacking categories of problems inherently gets people focused on the fact that these problems exist in the first place when they may not even recognize them otherwise, and regardless of the choice of language, it leads to better design oriented thinking in the context of larger and more complex systems.
Strictly speaking the expression problem is only defined for _static_ type systems because its concerned with _static_ type safety and extensions without recompilation.
There are several motivations for protocols:
Avoid the 'expression problem' by allowing independent extension of the set of types, protocols, and implementations of protocols on types, by different parties
I agree that it's a concern in static type systems, but the same issue rears its head when defining methods which operate on specific types of strongly typed data, so no, it's not always trivial, nor is it a problem exclusive to statically typed languages, and if it was, there wouldn't be a need to introduce a new abstraction for which addressing the problem is a key goal.
When I compare Clojure to another functional language I know well, Haskell, one of the things I really feel it lacks is proper pattern matching and currying; yes there are libraries that you can use, but it’s just not idiomatic to do in Clojure. I would not, however, assert that it’s an imperative language.
Could you care to elaborate on what exactly you find is lacking in Clojure?
[1] https://clojuredocs.org/clojure.core/let#example-542692c7c02...
(let [a 10
a (+ a a)
a (+ a a a)]
a)
(let [a 10]
(let [a (+ a a)]
(let [a (+ a a a)]
a)))
(+ (+ 10 10) (+ 10 10) (+ 10 10))
And this is different to the imperative style, because in imperative you can do:
int a = 10;
a = ++a + ++a;
(let [a 10
a (+ (inc a) (inc a))]
a)
This is the difference between the variable substitution evaluation semantic of lambda calculus and the imperative semantics.
(let [a (take 5 (range))]
(let [{:keys [b c d] :or {d 10 b 20 c 30}} {:c 50 :d 100}]
(let [[e f g & h] ["a" "b" "c" "d" "e"]]
(let [_ (println "I was here!")]
(let [foo 12]
(let [bar (+ foo 100)]
[a b c d e f g h foo bar]))))))
The difference is that when you say `(let [a 10])` the variable `a` is a constant now, you can effectively replace all occurrence of it within the scope by `10`, and you can do this at compile time.
That's why people say the symbol `a` is bound to the value 10, and not the variable `a` points to the value 10.
Effectively you cannot change `a` within the scope anymore, all use of `a` in that scope will be equal, and so within that scope you can change their ordering freely.
What I think is confusing you here is that `let` gives you syntax sugar so all the scopes are flattened, but each new binding pair is actually inside a nested scope, and so `a` is still immutable. In this case, it matters almost never, but as I showed, it still can, because in an imperative language you could still try to mutate the variable even in the same expression, and that is impossible to do in Clojure.
Can you point me to something that demonstrates this? I've both decompiled this form and traced its compilation:
(defn foo [a]
(let [b a
b (+ b a)]
b))
(let [a 10
a (+ (let [a 20] a) a)]
a)
Let me try and better explain myself. For me, the qualification here is that the form exhibit semantics that are compatible with functional programming, and can be evaluated using a lambda calculus computational model. Those semantics are in turn incompatible with the imperative ones, since it prevents you from doing some things that imperative semantics would allow, as I demonstrated with my prior example.
The let form does dictate a series of expressions to be executed sequentially from top to bottom, but the binding of their result to a name is done functionally, not imperatively.
This still models a data-flow, and the fact it lets you shadow prior names doesn't change the functional nature of it.
The form simply expresses a composition of functions and how their inputs/outputs connects.
That means it doesn't actually require any separate mutable memory location, even if the Clojure implementation for it might use some.
That's inherently the conceptual model of a functional computation, and it is not that of an imperative one.
(let [a 1
b 2
a (+ a b)]
a)
1 2
\ \
(+, , )
\
3
If you look at the multitree, at each level of the multitree, you are free to evaluate the nodes in any order, but if there is an input/output dependency, then you must guarantee that ordering.
At the end of the day, we could argue forever that we just have different taxonomies. I consider imperative programming the computational model where you give instructions in sequence which dictates mutations on memory locations.
I consider functional programming the computational model where you declare a composition of expressions that can be reduced using variable substitutions.
With this in mind, let qualifies as a functional construct, because it defines such a composition and can be reduced through substitution.
Now in practice, the Clojure let form reduces over this sequentially both left to right on a per-form basis, and top/down between each pair of bindings. And since Clojure allows you to mix imperative anywhere, you can have a println in it and know that it'll run after all that came before, and before all that comes after. But semantically it is still functional. And that's why you can't modify the bindings, because they do not conceptually represent memory locations which you can mutate.
(let [a 1
a (+ (inc a) (inc a))
_ (println a)
a (+ a a)]
a)
((fn[a]
((fn[a]
((fn[_]
((fn[a]
a)))
(println a))
(+ a a))
(+ (inc a) (inc a))))
1)
And like I said previously, you can reduce it with variable substitution, which gets rid completely of all the variables, at compile time, and the evaluation will give the same result:
(do
(println (+ (inc 1) (inc 1)))
(+ (+ (inc 1) (inc 1))
(+ (inc 1) (inc 1))))
((fn[_]
((fn[]
(+ (+ (inc 1) (inc 1))
(+ (inc 1) (inc 1))))))
(println (+ (inc 1) (inc 1))))
So here what happens you reduce that function into a do-block (which is Clojure's imperative form):
(do
(println (+ (inc 1) (inc 1)))
((fn[]
(+ (+ (inc 1) (inc 1))
(+ (inc 1) (inc 1))))))
(do
(println (+ (inc 1) (inc 1)))
(+ (+ (inc 1) (inc 1))
(+ (inc 1) (inc 1))))
(do
(println (+ 2 2))
(+ (+ 2 2)
(+ 2 2)))
(do
(println 4)
(+ 4
4))
(do
(println 4)
8)
Ok, and this last bit is very important, this is what people mean when they say that in functional programming the order of execution doesn't matter. The side-effects must still be sequenced in their correct order, but all the computation can happen in arbitrary order, because the computation doesn't rely on a sequence of instructions like it does in the imperative programming paradigm, instead it relies on this "reduction" process I described which as you see you are free to reduce each part in whatever order you want, you'll always end up with the same thing in the end.
The declarative nature is that you're declaring that you want `a` to mean 10 within some scope. That's why you say "let `a` be 10 in current scope", that's your intention here, for `a` to be 10.
After you've declared that, it holds true no matter what.
Where as in the imperative style you say: put 10 at place `a`. This is variable assignment, there's a place which is refered too as `a` and inside that place you can set values and change them at will.
At any point you can instruct the language to change what value is at place `a`, there are no restrictions to the instructions you can give.
It's a bit of a oversimplification to claim that imperative programming is distinguished from declarative programming by assuming a lack of ordering in the latter.
Functional programming is not prevented from expressing order, rather it is less able to express random accidental order at the operational semantics level.
At the end of the day, it's very much about the computational model, imperative is based on Turing model, and Functional on the lambda calculus. Because the latter doesn't depend on a global mutable running state, it is said to be declarative, in the sense that what you see is what you get, you don't need to keep track of what the memory currently has to proceed to the next step.
That said, you're right that Clojure supports some forms of imperative programming as well, but let isn't one, your parent commenter pointing to Atom was more on point. That's what you'd need to do to get back a more imperative `let`:
(let [a (atom 10)
a (+ (swap! a inc) (swap! a inc))]
a)
The println in the example is imperative, as it has side-effects, but a let block having an ordering of bindings is not inherently imperative. You can replace any let block with a set of nested functions:
(let [foo 12
bar (+ foo 100)]
[foo bar])
((fn [foo]
((fn [bar]
[foo bar])
(+ foo 100)))
12)
Language is hard.
I've worked in multiple Clojure shops and it has always been amazing.
All the core code and business logic etc. (kernel) is implemented functionally, and the interface with the outside world (shell) is implemented imperatively.
Clojure being a horrible language is a really hot take and it being based on hearsay makes your coomment look like it's not in good faith.
I do not know where your views are coming from, sounds like 2nd hand experience rather than 1st one.
But passing contexts I can see. There is a not uncommon pattern that one can use of keeping a large map with state in it.
However it's completely compatible with pure functional programming.
There's nothing "imperative" (as commonly understood) in 98% of Clojure code out in the wild.
"Contexts" or "context updates" are also very rare, unless required by a non-Clojure JVM or JS library.
Can discuss further if you reference a specific open-source code example with "context" or "imperative style".
But, since we all make opinions from others, let me provide you a balance of opinions by giving you mine.
I'm a senior engineer and I currently use Clojure professionally. I find it to be a lovely, fun and productive language, my favorite one to date actually. I have prior professional experience with C++, C#, ActionScript 3, JavaScript, Scala, Kotlin and Java. And of all of those, Clojure is my favourite to work with.
The "context' pattern you may have heard of, I believe I know what it refers too, and it is not an imperative pattern at all, let me explain.
There's often a case where a piece of functionality will be implemented by multiple functions composed together. For example, an API handling a request will delegate to sub-functions which could in turn call down to more sub-functions.
In that scenario, it can happen that the sub-functions need data about the context of the call to the parent function, or they need data generated by the prior sub-functions that were called. In my example, lets say the request object to the API dictates various options and has the request details, and each options is relevant to different sub-functions, maybe one needs the username, where the other needs the cart-id both of which were passed on the request, but maybe the other function also needs the user-permissions which was obtained from the prior sub-function.
Ok, so say:
someAPI({username: "foo", cart-id: 123}) {
permissions = get-permissions(username);
retrieve-cart-items(username, permissions);Now as you have more and more of these forming a coherent whole, in Clojure there is a pattern where people will say, all this initial data and the data generated by the intermediate steps becomes the "context" data for the operation as a whole.
(defn some-api [context]
(-> get-permissions
retrieve-cart-items
:cart-items))
{:username "foo",
:cart-id 123}
{:username "foo",
:cart-id 123,
:permissions [:can-view-cart]}
{:username "foo",
:cart-id 123,
:permissions [:can-view-cart]
:cart-items [:pen, :paper]}
And generally in Clojure you'd use the destructuring syntax to indicate what in the context map you depend on:
(defn get-permissions
[{:keys [username]:as context}]
...)
But, this is still fully functional, because none of the functions share a mutable reference to a shared context, they take an immutable context as input and return a new immutable context as output which will just happen to also include all the key/values of the context they received.
Sometimes people also pass in dependencies using this context pattern, which is a form of dependency injection through parametrization, but you combine all dependencies into one parameter object.
For the sake of the less experienced, I'd point them to these substitutions in particular:
Compojure: Reitit (which they used in the front end too, so migration maybe in progress) would be my preference for backend routing.
Component: Integrant takes Component's ideas, but prefers the flexibility of multimethods operating on plain data to the typed records approach for defining systems.
Schema: Once very popular, but superseded by clojure.spec (bigger in scope) and to a lesser degree Malli for data schemas.
Potemkin: Avoid, handy for some internal code organisation purposes but hostile to tooling and debugging IMO.
Clojure in general being very stable and backwards-compatible makes it even more easier to just continue using older libraries. So what if the library is "dated"? If it works well, why not continue using it?
(I speak from my own experience)
Too many "I'm very clever" functions that are hard to understand and also have subtle bugs.
When you say you have too many "I'm clever" functions, do you mean within code your team wrote, or in the ecosystem at large?
the "I'm very clever" defns and macros that I'm referring to were written by our team.
Most of the 3rd party libs are pretty good. Except for them being not maintained and not keeping up with the latest versions of databases, etc.
Also yea everyone wants to be so damn smart having macros within macros within macros that no one knows what the original intent of the code is anymore.
The repl based development I find also breeds a really bad mentality of forgoing building a deployment process and instead people just repl in and make a bunch of changes and prod rarely matches whats checked into github.
If you're writing macros on a daily basis, you better have a really good reason for it.
Its maddening to just be constantly mentally unpacking this stuff and searching through docs only to find out "Oh its not a clojure thing, someone wrote a macro for this"
They are used in certain libraries like https://github.com/ptaoussanis/timbre but for things that are simply not possible without macros, for example (timbre/spy (+ 1 1)) will actually print both the expression and the result:
DEBUG [ss.experimental.scratch:1] - (+ 1 1) => 2
This is one of my favorite debugging tools. IMO beats a debugger hands down! Printing can also be disabled with a global library flag without code modification.
Perhaps if the macros are "simple" they can be unpacked relatively easily. I do understand how mentally challenging that can be for somebody who's just starting with Clojure. I've been using Clojure for a long time and only just recently became more comfortable with macros after I made a conscious effort in that direction. I'm still far from an "expert" in them.
One thing to keep in mind is that Clojure has excellent interop with Java, and it is trivial to write a small Clojure wrapper that just does critical thing for your dependency injection system e.g. startup and shutdown, and just pass through a reference to the Java object when you need to make function calls.
Another thing to keep in mind is that Clojure code is extremely stable, so pure Clojure libraries often require minimal maintenance, and code written for Clojure 1.3 runs on 1.10 99.99% of the time with no modifications.
The other large issue in packages ive found is using some performance and monitoring tools to our clojure apps such as DataDog. The java tracing brings the app to a crawl based on how it instruments the compiled clojure code. Since clojure is a dependency itself the core language ends up having instrumentation code injected and the app grinds to a half. Something the Datadog team has not been able to solve.
Overall unfortunately I've just found Clojure isn't really particularly good at anything. It can hog memory, lib support seems iffy, Codebases never seem to have this "wow" factor that lots of people mention about clojure's elegance. I just don't see the promised land.
We also don't suffer from any of the problems mentioned above, had no issues with libraries and no one abuses macros or makes prod repl changes willy nilly, etc.
I also fully agree with this:
> Detecting an error when creating a record is much better than when using it later on, as during creation you have the context needed to debug the problem.
I made it a rule to perform integrity checks as early as possible: when creating, accepting or transforming data. I use spec (but schema would work just as well here), and have lots of pre/post conditions in my code.
I tend to settle on "simpler" solutions. I use mount, rather than component, because it uses the namespace hierarchy and requires less code and management. I use very few macros, and try to use simpler tools rather than more complex ones.
I noticed that these days roughly 30-40% of the code I write deals with integrity checks, anomalies and anomaly handling.
Love to hear more about this.
My approach has been to convert errors into data and have behavior that deals with conveying these errors in different ways, but it's always felt too complex for the task. I've just got a lot of stuff dealing with handling errors in the different environments (jvm / browser / nodejs / rn) and across async and non-async code.
https://github.com/weavejester/integrant
I haven't used Clip a lot yet but my next project is defiantly going to be with Clip.
For validation I have started to use Malli (https://github.com/metosin/malli). I really liked the idea behind clojure.spec but some of the implementation was a bit clunky and a bit too 'Rich Hickey', not sure how to describe it otherwise. Schema was again the first serious attempt at a library like that for Clojure so it was really nice when it came out. Malli sort of combines what is great about both.
Having said that, the article also exposes some flaws of Clojure that I also found.
1. Clojure overly obsesses with expression and non-procedural style coding. While in reality many large scale Clojure repo deploy their own macro to bring procedural style back (like letlocal in the article).
2. The builtin abstraction tools are almost always too simple to be useful.
But those are not big deals IMO.
At the same time, I also wonder if this article mention any feature of Clojure that is truly unique to Clojure compared to other Lisp/Scheme languages. I wonder if the article will still make sense if we simply substitute all "Clojure" to "Racket" (obviously I know ecosystem is not comparable).
Not lambda as we know it now popularised by AWS, but an architecture for stream processing where batch views from expensive and slow batch jobs are combined with speed views from stream processors into the final live result.
At best I would expect a small niche collection of very specific tasks to be improved, but definitely not applicable to general productivity.
Point out where I said "impossible"? I said skeptical, which is an absolutely perfectly position to take. I'm all perfectly happy to be wrong.
Storm is also not very well regarded in the stream processing community due to its restrictive model, poor guarantees, and abysmal performance [3].
I have nothing against Marz, but I do think skepticism is warranted until we see what they’ve built.
[1] https://blog.twitter.com/engineering/en_us/a/2015/flying-fas... [2] https://blog.twitter.com/engineering/en_us/topics/infrastruc... [3] I worked at Twitter for 3 years, then at Google on Millwheel and Streaming Dataflow.
$10M, the new cost will be $100k, or $10k, or less.
$1M, the new cost will be $10k, or $1k, or less.
I have a lot of experience with Python/Javascript now, and spent many years in C/C++/Objective C and Java. Also have some Go.
If you're into interactive kata-style problems, there's 4Clojure [3].
Also join the Clojurians's Slack [4] for the community.
[1] https://www.braveclojure.com/clojure-for-the-brave-and-true/ [2] https://www.oreilly.com/library/view/living-clojure/97814919... [3] https://www.4clojure.com/ [4] https://clojurians.slack.com/
One of the best "why Clojure" talks that I've seen, especially for somebody coming from an OOP perspective, like myself.
Can someone please explain this to a novice like me?
In practice the only benefit is very similar to checked exceptions - can't forget to check that value is in range.
E.g. a colleague implements de-serialisation for your type but adds an empty constructor to make their life easier. You might not learn there's a hole in the boat before your first bug.
If yes, then I agree this is super helpful (and magical).
If not, how is this different from a normal constrcutor (with runtime input validation)?
But your distinction about the theoretical limitations vs the practical capabilities of current type systems is also a good point.
https://clojure.org/about/spec
If your impression is that this is like sugary unit tests: It is not. You can run specs during development, while running a in-editor REPL, code gets evaluated while you type it so to speak.
It is way more expressive than a type system and it is opt-in, but it doesn't give the same guarantees obviously. It is also not meant to be a type system but rather a tool to express the shape of your data. It is used for obvious things like validation but also for documentation (over time) and generative testing among other things.
Names in Clojure codebases and libraries are pretty reliably annotated with trailing exclamation marks, looking like: `save!`.
To that end, running Clojure code blindly to test it and its types is a fairly practical practice, coming up in cases like Ghostwheel [1], which uses this for generative testing of clojure.spec types, which can be much more sophisticated than what is commonly used in static systems, even with refinement types.
[1]: https://github.com/gnl/ghostwheel#staying-sane-with-function...
Can you elaborate on this? I have some experience with Clojure, and have been relatively unimpressed with spec. Everything it does I can do with (refinement) types (I think). Reading over the Spec documentation it constantly talks about predicates... which is exactly what a refinement type is.
Spec/Clojure has the problem where it validates... but doesn't parse. See: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
For example Ada's type system, which has equivalent of Common Lisp's SATISFIES construct, which implements a runtime type assert that can use all the power of the language.
>Most static type systems that can do what clojure.spec can do tend to include runtime assertion and type checks and do not erase type data from runtime (what some static type zealots call "uni-type" approach).
When you don't erase the type data... you're gonna have more than one type. How is this a "uni-type" approach?
Extremely glad to see this resource published! Thank you to the Red Planet team for releasing this!
Aren't there many Clojure projects out there, or is the language generally not used for large projects? I have a JavaScript frontend written in Vue that is 150k+ lines of code, without any tests. Which would be 25k lines of code more than they have, disregarding their tests.
I'm not saying that more code is better, I just found it odd that 250k was considered one of the largest Clojure codebases in the world
I was honestly skeptical, reading all of this clojure-macro criticism (constructive for the most part), but trust was restored when I saw the author. Well done, sir! Excited to see the details.
Spec is also still in alpha, with spec2 still under development.
Ask who?