A Year in Clojure
blog.taylorwood.io
blog.taylorwood.io
There is literally an arity of reduce which skips the initial state. Check out `(doc reduce)`.
> No tuple type? Vectors are tuples. “What if one of the vectors is of a different length?” I gasped as I clutched my pearl monads.
`s/tuple` is the answer to that question.
> This ties into another concept that was new to me: applying a sequence of arguments to a function.
You're not applying the sequence to the fn. This is "function application", so you're applying the fn to the sequence. In Clojure, you have some data, you apply a fn to it, then you have some transformed data. Rinse and repeat. Also, check out `(doc apply)`.
> I’ve read/heard the Clojure designers encourage the use of namespaced keywords, but I rarely encounter them outside clojure.spec definitions.
As other people have done in the comments here, I make a case for namespace keywords here [1].
> I can’t talk about data in Clojure without talking about clojure.spec!
Please be sure to check out Clojure spec's instrumentation, along with Orchestra [2] to further improve the experience. My team uses `defn-spec` for basically every fn in both our front-end and back-end code; all of the data is spec'd, with shared specs between front-end and back-end, and we get full instrumentation during development and testing. Thanks to macros, that can disappear completely in production.
1: https://blog.jeaye.com/2017/10/31/clojure-keywords/ 2: https://github.com/jeaye/orchestra
Java 8 isn't awesome, but it was good enough to kill Scala and Groovy.
You don't notice their utility in code examples, or small projects, but in large projects they are very useful.
{"signup/username": "John",
"signup/password": "pg4pres"}
More on the topic I can say that I don’t really see anything compelling that you can’t easily do in F# with a lot more safety in this article...
> I’ve felt this pain a few times and the only conclusions I’ve drawn are to use clojure.spec for non-trivial code, especially around domain boundaries, and try even harder to solve very small problems in ways that can compose to solve larger ones.
This seems reasonable to me. A lot of where refactoring gets painful is when you have bits of code being reused across wide swaths of the program. Keeping modules small limits the opportunity for that to happen, and makes the scope of impact for a change easier to conceptualize without leaning on a compiler.
The problem I see there is, it seems to be exceedingly rare that you can get a whole team of developers to be so disciplined - and all it takes is one lapse in discipline for it all to fall apart.
Any project of this size is too big and is better off being broken up.
You need tests no matter what kind of typing you are using.
The best thing about Clojure is that it is immutable by default. Start writing pure functions that transform you data step by step until you send it on its way with a side effect. I don’t miss types much at all - immutable maps lists, sets and pure functions get the job done and it is fun to program.
Static typing fans hardly ever acknowledge how difficult the types make it to read the code. Too much focus on that one time when someone made a typo and there was no unit test.
This has been argued infintum, nobody is changing their mind. WTF am I doing here?
let add n1 n2 = n1 + n2
I think types make code much more understandable. I like them to be explicit so there's no mental overhead of thinking "what type might this be" - and the larger the codebase, the more this pays off.
There were really 2 main points to java (as I understand it). The first was to have a statically typed language that had a relatively fast runtime that could easily be ported to many platforms. Indeed, the killer application for Java, IMHO, was the idea of implementing that run time in hardware (and it's a shame that Sun never really pushed that angle very hard).
The other main point of Java was to codify a set of "best practices" as a kind of "defence against the bad programmers". The intent was to reduce the difficulties of hiring programmers by making the programming language less expressive (compared to C++, for instance). IMHO, this largely failed as programmers are creative and will always find ways to be expressive. So on one had we got C++ template crack and on the other hand we had abstract factory factories.
Basically, though, I don't think there was ever a real intent to allow Java (the language) to make it easy to scale development to larger teams, or larger code bases. The success that Java had in getting into large enterprises just meant that the normal Java user was interested in that, because it's the environment that they were working in.
The idea is attractive to enterprise people because they already have a lot of people and they want to get a lot done in a short period of time. Most managers (and many influential programmers) do not understand the loss of productivity that accompanies communication overhead from having larger teams. The large code bases are the necessary result of programmers trying to isolate themselves from the hundreds or even thousands of other programmers that they are working with. By building complex boundaries, they can reduce the communication complexity at the cost of increasing the program complexity. This is usually a good trade off if you are stuck having to work with an insane number of people.
BTW, in case you think that it is only non-technical people who fall into this trap, I will provocatively mention 2 words: micro services. The way micro services are usually designed and implemented, they are usually the very definition of premature subsystem decomposition. This premature decomposition is chosen so as to allow a few people to "lock in" design decisions and reduce the need to communicate with a whole bunch of people who might ruin their architectural vision. Basically, same problem, same result. Technical people (who didn't live through the CORBA years) drink it up like the yummy, yummy cool aide it is, though ;-)
Technical people understand that a technology that can drastically reduce LOC is a game-changer: you can hire far fewer programmers - these need to be better than your bucket-brigade Java folks of course, but that's OK because you're saving money on the quantity of them; you need fewer managers to coordinate everyone; the product itself will be better because it's had fewer fingers in the pie, and internal comms during development will have been better because the team was so much smaller. No, the big problem with this is that management in big corps simply do not believe that these sorts of savings are possible - it just seems too good to be true, and they are too risk-averse to rejig their organisations to the required extent; they also fear being beholden to a small priesthood of very clever technical people from which there is no way back - eg we can't ship our Clojure codebase back offshore to some commodity programmers because they won't be able to make head or tail of it.
I find the counter-arguments to your discomfort in this thread very strange. Some big problems just require big solutions. A language might indeed be best suited for smaller problems, but that doesn't mean that bigger problems are a "cultural disease". For Clojure in particular, I recall Hickey saying something to the extent that for the problems where Java is successful, Clojure should be successful too. Maybe this rhetoric has changed, but I think it's also related to people's experience that what takes 750k lines of Java takes under 1000 in Clojure, for some problems, so you get this bias that 750k lines of Clojure is "wrong". No, it's just that the equivalent Java would be a factor of 10-100 bigger. If a language can support modularity, it should be able to scale.
More at addressing your discomfort, I'd be curious to know what you mean by "language without any type safety". Dynamic typing isn't the same thing as untyped, and some dynamic languages (like Lisp) support a notion of compile time that lets you have type warnings, optimization, etc. that you might be used to from static compilers... as far as testing goes, the big testing pushes in industry have come from people dealing with giant C++ and Java projects more than the dynamic language crowds, because more energy for testing is needed in general. Whether dynamic languages need more or less of it could be interesting, but because of the variations in what we call "static languages" and what we call "dynamic languages" it might not be useful to compare things so broadly. Instead we could compare language against language, and take into account usual dev practices. Having a REPL to leverage while developing, testing, and debugging changes things in a big way, for instance.
Common Lisp was designed to be a successor of such a Lisp and was supposed to support both complex and/or large applications. Code density can be a bit higher than C++ (say, factor 3) for larger programs.
In the end 80s / early 90s Evans&Sutherland developed a high-end 3d design software in mostly Lisp - the Conceptual Design and Rendering System (CDRS) - many of the cars at that time were designed with it (from Ford, Jaguar, German brands, ...). A designer work environment did cost most than a million USD per seat - including a high-end graphics system from E&S. The 3d software was written in atleast 150kloc C++ and 400kloc Lisp.
Other CAD systems written in Lisp should have been much larger like iCAD or the design system from PTC (years ago to be said at 7MLOC Lisp). iCAD for example was used at Boeing for the development of turbines, wings, internal stuff - it's even said that there was a complete model of some Boeing in iCAD - which used kind of an object-oriented language on top of Lisp for the parametric construction of technical things. Similar applications and languages still exist - for example GenDL: https://gitlab.common-lisp.net/gendl/gendl
There are a bunch of things which makes this possible: Lisp compilers with various compile-time warnings, extensive runtime error handling facilities, object-oriented programming, good development environments, ...
I guess one issue might be that most developers that get introduced to Lisp, do so in primitive ways.
So they never get to learn about those toolings.
Even with Clojure, not everyone is willing to pay for Cursive.
For 10 years I've seen many developers who have only really used statically typed languages like C++ and Java, now finding the "joy" of dynamically typed languages and all the "freedom" they give you, only to realize within a year or two that, wait a second, verifying those types at compile-time was actually pretty useful. I've also seen mass migrations away from languages like Ruby, Python and Clojure towards OCaml, Haskell, and F#, because people really enjoyed the functional programming aspects but wanted their static typing back. After writing the 10,000th unit test that only exists to ensure that you spelled some function/method/hash-key correctly, it gets a little tiresome. Personally I've settled on TypeScript, which has nice IDE integration in VS Code so that it validates your types live while you code, while still giving you the flexibility of a dynamically typed language (it's all still JS at the end of the day).
https://code.facebook.com/security/fighting-spam-with-haskel...
Lest you forget, both fight against Java: https://trends.google.com/trends/explore?date=today%205-y&ge...
Industry is not doing a 'mass migration' to clojure, haskell, or any other language not in the top 5.
So the OP was wrong to claim a 'mass migration' in the first place
OTOH, when I'm doing more "building infrastructure" type work - implementing a data store, writing a compiler or interpreter, stuff like that - I start getting more interested in rigidity and formality. Static languages treat me well in these situations. The more rigid, the better - I'll prefer Scala to Java, for example, specifically because it gives me more tools in the type safety department.
I recently rolled my own (verg tiny and specific) build/deployment system for a project. Initial prototyping was a breeze, but once I had things more or less nailed down, I started adding types and, for example, was able to leverage the type system to make sure it can only be deployed listening on IP addresses in the private range. The next step is to rig up some sort of Zerotier integration to ensure I only make it available on my private networks.
It is an implementation of much larger messaging idea - message receiver only takes care about data it needs and nothing else.
so instead of
data Person = Person String String Int Float String String
you have :person/name ....
I’m not sure what these comments are arguing against, but it’s not compile-time analysis.
Seriously: you’re describing a feature that only exists in a small subset of languages and only a small subset of contexts at that. OCaml and Go do this but not for data, just for methods.
The language used in this thread so far has made it seem like runtime specs are the solution to this “only care about part of the bag” problem rather than what I consider a general feature of dynamic typing that certainly can be expressed statically as well.
Runtime specs absolutely also validate. And you can run them at test time. And you can put arbitrary code in them if you want. And sometimes you can statically validate them.
What I’m saying is you get ad hoc structural typing on everything, including data, which you’ve already said is a good thing, so I’m not sure what we’re arguing about or if we’re even arguing :)
EDIT: ah! You edited the paragraph. Simple question: where does this ideal static typing thing ship? Like, how do I do this on my machine right now? What package do I install?
People were talking about using runtime specs to annotate and validate the subset of information a function may care about. Which is certainly something I wish I could do in more languages, though ideally as much at compile time as possible.
Of course, runtime specs in Clojure let you make more assertions than just structure, so structural typing as seen in TypeScript, for example, certainly don’t go all the way. It just helps address what I thought was being discussed.
What is the ideal middleground? I think one possibility would be if Clojure was statically typed to begin with yet had the syntax of spec/schema.
Though I’m not sure we’re talking about the same thing anymore.
All of a sudden you can stop worrying about "what am I getting in this function" and actually concentrate on the business logic.
(No, tests can't cover everything. Yes, we still have to write tests, but now they are all useful. Yes, the thing that types hinder you from doing changes is a lie, it's quite the opposite, much easier to change things since the compiler covers your ass)
I think what you're saying is something like "this function guarantees that the thing I get here conforms to this shape" and I do like that. Something I don't see a lot of people express is just the simple "I know what the shape of this thing is by looking at the signature". Whenever I find myself reading other people's Clojure code, it feels like I end up spending more time figuring out what is being passed into a function. I have to look around to find the origin of a value. I wanted to tear my hair out the first time I tried reading Ring's source code to figure out how it worked. With F#'s Suave, I can tell what each function gets and what can be done to it--it's self documenting. In Clojure, as the original author you can play around in the repl full steam ahead because the types are in your head at the moment. Those who come after you have to assemble the puzzle with less information.
utop # let avg a b = (a +. b) /. 2.0;;
val avg : float -> float -> float = <fun>
utop # avg 2 4;;
Error: This expression has type int but an expression was expected of type float
utop # avg 2.0 4.0;;
- : float = 3.
user=> (defn avg [a b] (/ (+ a b) 2))
#'user/avg
user=> (avg 2 4) 3
user=> (avg 2.0 4.0) 3.0
I think it is a matter of taste which one you prefer. With property based testing and specs you can go very far in term of pre-runtime checks and there was some data about the number of bugs per line published by Github where it turned out that Clojure did not do too bad in comparison to statically typed languages.
avg :: Num a => a -> a -> a
Where your "a" is a number (that is, a type that implements the Num typeclass)
This would work out of the box on floats, ints, etc
If you have operators that are typed like + and +. how can you make this work with Num?
λ> :t (+)
(+) :: Num a => a -> a -> a
λ> :t (/)
(/) :: Fractional a => a -> a -> a
import Data.List
average :: (Real a, Fractional b) => [a] -> b
average xs = realToFrac (sum xs) / genericLength xs
One of the things I liked about Lisp when I first saw it was that, since it uses kebab-case and (usually) full words, I could just turn on my standard English spell-checker. After all, if "Programs must be written for people to read, and only incidentally for machines to execute", then it makes sense that I should be able to run a normal spell-checker on them. The only reason that was infeasible in languages like C is because they used (for both technical and cultural reasons) names like "memcpy".
Whatever field I'm working in already has its own vocabulary. I don't need to create a new <domain>-in-<proglang> vocabulary for a program about it. Or if I do invent an abbreviation that's so useful that I want to use it in my source code, I'll probably also want to use it in my documentation, webpage, emails, etc.
Most large programs that I've worked on, in other languages, have at least a couple misspellings in func/var names, which even Haskell's type system won't catch. "Referer" was clearly not written by someone with their spell-checker turned on for source code!
Indeed one of the "joys" is that one can apply that verification a la carte, and possibly with different approaches [0][1]. Similarly, one may desire to dispatch on type [2], but might sometimes want arbitrary dispatch [3].
> After writing the 10,000th unit test that only exists to ensure that you spelled some function/method/hash-key correctly
This too is solvable with a la carte tooling [4].
Everything has trade-offs. Static type systems give a fixed set of benefits with a fixed set of costs. Some folks prefer to use their experience and judgement to choose when, where, and how to pay the associated costs (and that freedom of choice itself has a cost). It's okay to have different preferences.
[0] https://github.com/plumatic/schema
[1] https://clojure.org/guides/spec#_entity_maps
[2] https://clojure.org/reference/protocols
I have found too that when you are programming functionally as in Clojure you don't get bit by shared state things not being what you expected as everything you are writing should be data in, data out and easily reasoned upon. Nil handling in Ruby is a huge problem as people are chaining methods on mutable objects which becomes complicated, and this is the source of most bugs in Ruby. This is somewhat mitigated lately with safe navigation operator.
That is why TypeScript is appealing as you can use typing when you'd like to but still take in "unsafe" JavaScript or write in JS. Unlike if you were using Elm and had to write a wrapper to conform all JS code coming in.
Spec in Clojure is very nice as you can enforce and conform types where you need it, and also make generative testing for free. If there is some mass migration to OCaml, F# and Haskell I have yet to see it because very few businesses use them and the communities are much smaller.
The thing I've _consistently_ seen with every large Clojure project I've been a part of is that as the project grows, it becomes harder to make sense of the types used across the project. On the last couple of these projects I've been on, Schema was used to annotate many types, but I've never found this to be quite enough and I still remain firmly convinced that gradual typing just doesn't work well enough generally. I think that with a _very_ highly disciplined team of experienced developers it could probably work well. But we don't all get the opportunity to work on such teams. What I've personally seen is developers are all too willing to either add a partial type spec/annotation (with probably some s/Any's lazily thrown in where there should most definitely _not_ be), or just flat out not adding any at all ("Why would I need it here? This is very simple code and it's obvious what is happening here!" ... sure, maybe it is to you now... what about in 6 months? Or what about to one of the other developers on the team?).
When discussions about static vs dynamic typing come up (which is almost always a fruitless argument in my experience), I've noticed that the dynamic typing proponents tend to get overly focused on the idea of "correctness". This has always disappointed me. Correctness is certainly an important benefit (though I think it tends to get exaggerated a bit much), however, to me the idea of documentation in the form of a type definition is _much_ more valuable over the longer term, especially so for large projects. Developers are usually quite lazy, especially when it comes to documentation, and often when looking at a piece of code in a large code base, the only documentation about the values being passed to a function is going to be the type definitions (aside from reading the code itself of course).
In this sense Java is actually dynamically typed: there's no way of knowing at compile time whether an object is actually an instance of that object or is null. In languages like Kotlin, Haskell and Ocaml, which specify in the type system whether or not something can be none and force the user to check before using something that can be null, null pointer errors don't happen.
That is an extremely unusual definition of dynamic typing. If I have an Integer it might be null but it won’t be a BeanFactory (unless someone did a reflective call or wrote some bytecode or otherwise subverted the type checker).
Instead of that, it would be better to respond directly to the specific points the OP made, rather than move the argument to a bunch of new topics.
OP did not say anything about other languages, be it Java or otherwise. OP is just describing experience with closure.
In order to understand clojure better, it would be good to not shut out feedback.
I hope java gets native support for @nullable, but for now we have stuff like error prone and @nullable annotations.
Ex: Facebook and adding types with Hack & Flow. Dropbox's work to add types to python. Uber used to be a python and nodejs shop, and now new stuff is done in golang and java. Many nodejs shops go from pure js to typescript or flow.
Gradual typing is useful for companies that don't want to immediately migrate everything at once. If you have the right policy of 'type everything you touch', enforced with a linter, a codebase gets migrated relatively quickly.
Independent modules are easier to reason about, they're reusable, and they can be maintained on independent schedules. The biggest advantage though is that this approach allows you to split teams up into smaller groups responsible for the individual modules. One of the biggest problems in maintaining large projects is communication overhead, and typically it's very difficult for teams with over 5-6 people to be productive.
So, if your project outgrows limits of dynamic typing, chances are it should be split up instead of using static typing as a crutch.
Maintainable in what sense? And large in what sense?
I typically work on 1-10 million lines of Java code solutions split up into maybe 20-50 seperate components.
These 20-50 seperate components have untyped boundaries between them.
Setup correctly these untyped application boundaries rarely cause problems - of course you have to put a bit more thought into architecture, documentation, and verification but it’s not particularly onerous and we have been doing it successfully for decades now.
I’d say anecdotally errors caused by this lack of typing are well under 1% of total errors.
Are there languages you've seen that don't show an ugly side as projects grow to a large size?
Assuming, for the sake of argument, that that's true, it means that we've got to think about how you define "programmer". Does it mean anyone who uses a programming language, or does it mean something more like "software developer"?
TIOBE, for example, effectively measures popularity among the former group, not the latter. Which means that, in the case of Python popularity, it's going to be picking up on all sorts of things: analysts and BI people ditching Excel in favor of Pandas, ops people ditching piles of shell scripts in favor of Ansible, graduate students choosing Scipy over R or Matlab, 7th graders with Pi-tops, ...
Starting with the fact that most dynamically typed languages today are progressively adding type checking (spec in Clojure, gradual typing in Groovy, etc...) while no statically typed languages are going the opposite way.
Of course, no-one actually thinks of C# as being a dynamically typed language despite its dynamic keyword, nor does anyone think of Apache Groovy as being a statically typed language despite its @CompileStatic annotation. In the everyday world, C# is used for building performant systems and Groovy is used for glue code and build scripts.
Languages tend to go both ways nowadays. Go first shipped with both static typing (though without generics except for builtin datatypes) and dynamic typing (though using the wordy interface{} marker).
Static languages added dynamic capabilities with reflection and run time code generation a long time ago.
At the same time, industrially-popular static languages have added dynamic escape hatches. The trend is really toward optional static type checking, with dynamic languages moving to opt-in typechecking and static languages moving to opt-out.
You can use Apache Groovy for just the glue code and Gradle build scripts, and use a statically-typed language like Java, Kotlin, or Scala for the actual system you're building -- that's what most programmers do.
Compile time verification is nice if you know all your inputs at compile time. At some point, data from elsewhere comes in. Specs are an answer that preserve the composability goals of Clojure (who cares if I have a bit more information than I strictly need?) and allow you to validate them at runtime with useful error messages.
spec has incidentally made Clojure's compile term error messages better! But that's just because one program's compile time is the compiler's runtime, and the compiler is running spec verifications on your code (which is after all, just data!) during its runtime :-)
My experience is that dynamic typing is problematic in imperative/OO languages. One problem is that the data is mutable, and you pass things around by reference. Even if you knew the shape of the data originally, there's no way to tell whether it's been changed elsewhere via side effects. The other problem is that OO encourages proliferation of types in your code. Keeping track of that quickly gets out of hand.
What I find to be of highest importance is the ability to reason about parts of the application in isolation, and types don't provide much help in that regard. When you have shared mutable state, it becomes impossible to track it in your head as application size grows. Knowing the types of the data does not reduce the complexity of understanding how different parts of the application affect its overall state.
Immutability plays a far bigger role than types in addressing this problem. Immutability as the default makes it natural to structure applications using independent components. This indirectly helps with the problem of tracking types in large applications as well. You don't need to track types across your entire application, and you're able to do local reasoning within the scope of each component. Meanwhile, you make bigger components by composing smaller ones together, and you only need to know the types at the level of composition which is the public API for the components.
[REPL driven development](http://blog.jayfields.com/2014/01/repl-driven-development.ht...) also plays a big role in the workflow. Any code I write, I evaluate in the REPL straight from the editor. The REPL has the full application state, so I have access to things like database connections, queues, etc. I can even connect to the REPL in production. So, say I'm writing a function to get some data from the database, I'll write the code, and run it to see exactly the shape of the data that I have. Then I might write a function to transform it, and so on. At each step I know exactly what my data is and what my code is doing.
Where I typically care about having a formalism is at component boundaries. Spec provides a much better way to do that than types. The main reason being that it focuses on ensuring semantic correctness. For example, consider a sort function. The types can tell me that I passed in a collection of a particular type and I got a collection of the same type back. However, what I really want to know is that the collection contains the same elements, and that they're in order. This is difficult to express using most type systems out there, while trivial to do using Spec.
alternatively, I'm always receptive to reports about experiences.
I'm very skeptical about demands to produce "empirical evidence" for or against any sociological programming practice, because it tends to be a method to place the burden of proof on the opposing side, while reserving the default position for own own side - which just turns out to be a way to normalize one's own beliefs.
Furthermore, fixating on static typing as the one defining feature of a language amounts to concept space reduction. In practice, there are many complex factors that contribute to overall quality of the language that have to be considered holistically.
This is a huge factor I'd like to emphasize and write about in a follow up post, having worked on many Clojure projects at different companies now... Another Year in Clojure :)
I've maintained some large Clojure codebases I didn't write, and the easy ones shared this quality. The more painful ones were reminiscent of large OOP codebases and honestly I would've preferred they'd been written in Java.
https://medium.com/wit-ai/open-sourcing-our-new-duckling-47f...
Programming is not my day job, so please correct me if I'm wrong - but every project I was involved in so far - having access to a production DB is a big no no.
Or does the project you refered to not contain any sensitive data, so having direct access to the production system is deemed 'safe'?
I'm curious how Clojure avoids this. In my large Clojure projects, there aren't "types" in the sense of static typing, but the shapes of data still exist, the types are there, but they are in my head or in other forms than that which a compiler can check. I still have to remember that a map is a certain "type" of data with certain keys/values that are also of certain types, and a large project might have many of these "types", even if they are not explicit. That is equally hard to maintain for me -- harder, really, since it's too easy to make mistakes.
I also find that you learn patterns for structuring code that make it easier to track the types. For example, I avoid renaming keys or restructuring data except at the edges of the application. Adding Spec or Schema at component boundaries is also helpful, as you can see what the shape of the data was at the API level and work your way through the helpers from there.
There's also a big benefit of working with data as opposed to objects. Each class is its own ad hoc DSL with custom behaviors. Knowing what one class does tells you nothing about what another does. This approach doesn't scale well for large applications where you have thousands of classes with each with its own behaviors.
It wasn't horrendous (and it's not like Scala is absolutely amazing for us), though maintenance was something of a pain point. We might use Clojure again, especially with some new blood who are Clojure enthusiasts.
These days I'm less and less certain of what proactive role a language plays in team success. That is to say I think the arrow of causation is usually pointed in the wrong direction. A well-functioning, excellent team will, given sufficient latitude, converge on a language that suits a team well. This language is often different for each team. Introducing a "good" language is unlikely to save a team.
Crawlers naturally want to be stacks with pipelines and expressing them as tail recursions over URLs curried into transducers, etc was easy enough conceptually, by difficult in reality because you want crawlers to be stateful. I think in clojure you end up hacking state by just attaching stuff to the outputs of your functions and building of more complicated objects down the line.
I discovered that stuff which is cake in SQL or a language with data frame support is often typically hard in Clojure. Eg eg joining data , aggregation, etc. The few clojure libraries for this sort of thing are complicated to use because they have a mechanical sympathy with how clojure executes. Eg https://github.com/nathanmarz/specter/
So I wrote some of my own libraries to get over this hurdle. I then realised that there isn’t a single decent statistics library that still maintained, and that the java bindings were far from simple to us.
At that point I gave up on clojure for my use case. That’s not to say that there isn’t something for which it’s awesone. But in my opinion it isn’t data processing.
You can certainly use SQL, and it's not any worse than any other language (and in some ways a little better), but IME it's not Clojure-simple, either. It's still the most awkward part of all my Clojure programs.
Clojure excels when you can avoid mutating state, and I think that's why people especially love it for web apps. It fits almost perfectly into that model.
I kind of found a similar problem with Scala. In theory yeah, you can use any Java library from Scala. But what was happening was I was continuously hitting the impedance mismatch between the scala collection and data types and the Java ones. Sometimes it was happening implicitly and doing automatic conversions of massive amounts of data without me knowing and I had to do all kinds of non-idiomatic stuff to get Scala to perform well. My Scala started off much prettier than my Groovy code usually is, but by the time I finished it was an abomination. At that point I gave up and went back to Groovy because while it lacks many of the functional features, idiomatic Groovy "just worked" and gave me most of what I was looking for in terms of a "better Java".
> Sometimes I think I should just use Kotlin, but then I would miss the ability to drop seamlessly into dynamic mode
Have you considered Clojure for this use case? That might alleviate your problems with using Kotlin and Scala, without having to resort to a single-target language like Apache Groovy. After all, unlike what Clojure, Kotlin, and Scala all offer now, we're never going to see an edition of Groovy that targets Javascript, Android, or native CPU.
I guess single target isn't a particularly big problem for me; the JVM runs everywhere I need it to.
On the data analysis side things aren't rosy on the JVM. I'd delegate that part to something which actually has all the tools built-in, e.g. Julia/Python+Numpy/R. Apache Arrow[4] is a nice project to facilitate the dataframe interop.
[1] https://github.com/ztellman/manifold
[2] https://github.com/stuartsierra/component
(defmacro let (bindings &body body)
`((lambda ,(mapcar #'first bindings) ,@body)
,@(mapcar #'second bindings)))
(define-syntax-rule
(let ([id expr] ...) body ...)
((λ (id ...) body ...) expr ...))At the end of the day, every line of code you write is some data in a bespoke data format. Most programming language implementations leverage this, and academic compiler courses make you do the same work. A program lexes, parses, modifies... that data structure and eventually rewrites it to some other programming language, usually something lower level.
If you exposed that underlying structure to the programmer, you give them a lot of power! The simplest way to see this is that any time you write a highly repetitive bunch of code, wouldn't it be great if you could ... write some code that wrote the code for you? You know how to systematically express the thing you want: you can probably say it a lot more crisply in a few lines of (especially functional) programming languages.
Once you get rid of simple repetitive stuff, you realize that you can do so much more with this. You want advanced pattern matching? Sure: you can go implement it as a library. async/await? Library. Type checker? Library. New convenient way of defining functions? Library. New syntax for matrix math? Library. Et cetera :-)
People in other programming languages have figured this out too. That's why we have code generators. The difference is that code generators are far enough removed from what you're actually doing (because e.g. they're glorified string concatenation) to be quantumly less useful. Generally speaking advanced IDEs will do this sort of thing: actually parse the code you're working on and let you do fancy tricks with it. Lisp is like having that power, all of the time.
It takes a symbol and figures out the name of a C function, how to most efficiently call it (with type hints for perf). If you screwed that up in one location it might have security consequences. But I _can't_ scre it up, because instead of copy pasting that code a gazillion times as I would have in Python, I just figured out what I want once and then use that functionality a gazillion times :)
When you actually start using these languages, you find yourself tempted to leverage this property and try writing lots of macros. But then you go back to the community and everyone tells you "the first rule of macros is you don't write macros." It turns out that if you use macros all over the place then your code becomes impenetrable. The problem is that macros allow you to circumvent the expected order of evaluation and produce your own novel syntactic structures. This puts lie to the old claim "Lisp has no syntax." In reality, Lisp has tons of syntax, it's just informal and buried within the definitions of macros.
It parallels the problem put forth in Jo Freeman's famous piece, "The Tyranny of Structurelessness" [1]. Any group that purports to adopt a structureless organization ends up having a hidden, informal structure. This is the way of Lisps as well.
I actually had heard of at least KRC and Miranda even before the first time I've read the paper, but even if I hadn't I would immediately be able to think of several languages with some or all of the additional features over the Lisps he was comparing them to for which he describes them as superior to Lisp for the purpose, including some modern Lisp derivatives.
Pattern matching, mathematical notation, and static typing with use defined types are all features that have become more, not less, common in languages since the paper was written.
Equivalent arguments exist for Ruby. Ruby isn’t bad because someone wrote a DSL that should’ve been a method and maybe a hash.
I happen to own a copy of HTDP which I bought for the course. It's a suggested textbook but not required at all. I haven't had time to go through it though.
I learnt loads translating the Lisp to Clojure.
Comments should only be needed when things are especially complex and if they are everywhere their average value goes down. Most codebases of good quality that I've seen have not had particularly detailed comments--except where you really need them. The mere existence of a comment already hints that the code that follows needs to be carefully understood before you can touch it!
There might be boilerplate comments before each function to describe the function and explain the parameters but other than that comments are only reserved for the tricky interactions. But in many cases also this information can be embedded in the name of the function and its parameters!
Good code reads well because it's built on top of lower level blocks that are simple and well-defined. And those are simple and well-defined probably because they are also built on top of even simpler things. This removes the need for a lot of mundane comments.
I don't think Clojure is more exceptional with regard to this than other languages. But I think a higher percentage of Clojure code is of good quality, and that probably is because Clojure attracts good programmers who care about these things.
Yes, you are. Outside of example code, this seems reasonable to me. If you can't understand the code just by reading it and a short docstring, it's a sign the code is too complex. There are cases where such complexity can't be helped, but they're not common.
As a trivial example, something like:
(int (* 1000 seconds)) ;; seconds -> milliseconds
(seconds->milliseconds seconds)
Ideally, comments should mostly be used to explain 'why' rather than 'what'.
It's definitely overwhelming in the beginning but after a while docstrigs + reading code seems to hit a very good tradeoff for me. And just using comments for weird or unexpected behaviour.
There are alos efforts to put more focus on documentation though: https://cljdoc.xyz/
Maybe someone would some more google fu or simply a way to query news.ycombinator could do a search for the string in the header...
a = [C, C++, Closure, Python....] look below for a bigger list
{Moving from <a> to <a>}
{<a> is considered harmful}
I don't want them to stop or anything it just would be interesting to see someone with like an sql database of news.ycombinator posts to see how many are posted in the last year or so and look at them all so I can binge read them because quite honestly they are the better posts on this site.
Bigger list of A here https://gist.github.com/brianherman/64a2800ebd1907c6bba4f6eb...
https://github.com/golang/go/wiki/FromXToGo
The best article of this kind that i have seen is by far
http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...