Problem Solving and Clojure 1.9 with Rich Hickey
case-podcast.org
case-podcast.org
I like it how its a Lisp, and ever since I've learned Common Lisp, I've been wanting to use Lisp at work for everyday projects, and Clojure seems to fit just perfectly well.
As with these new things you are always excited to explore further and see what's about to come next. After searching, I was not able to find anything on the Clojure roadmap anywhere on the internet. I understand that Clojure is largely an Cathedral kind of an open source project, largely developed in private at Cognitect and released to the world. But it was still a little strange to not see anything on Clojure's future plans. Sometimes you also see the Clojure repo on Github go without commits for weeks, for example the last commit as of today is July 4, 2018. Which is like strange for a programming language of this size.
On the ClojureScript side there is also a lot of movement. I follow https://twitter.com/mfikes on Twitter to get a heartbeat.
But I went through a complete journey of reading ANSI Common Lisp, The Little Schemer, The Seasoned Schemer, On Lisp and various other scheme material on the internet.
So syntax and other Lisp related stuff wasn't the problem. The Joy of Clojure book showed the Clojure way of doing things really well.
If you have to start on absolute basics of Lisp, I'd recommend watching the SICP lectures and reading PG's ANSI Common Lisp book.
I didn't finish the course as I moved to Elixir, although I liked Clojure very much. Elixir is also functional, much better for newcomers, lots of support, people, documentation, examples, books, courses, growing very fast... And one doesn't need to wait several seconds to start the virtual machine.
https://www.amazon.com/Clojure-Programming-Practical-Lisp-Wo...
A lot of tips and insight of how to do and not to do things in Clojure without too much hype. "Practical Lisp for the Java World" - the title does not lie :).
When I was first getting into lisp/functional programming a few years ago I would find that after writing a 6-8+ line solution on some of the tougher problems, the top users would have a way of doing it in 1-2 either due to a standard library function I didn't know about or they just took a better approach overall.
Most of Clojure's current development is around tooling [1] and some changes to move clojure.spec out of alpha. I believe they want to accomplish this for 1.10, but I'm not sure where I remember hearing that.
Alex Miller has mentioned that Cognitect built and continues to work on a tool that tracks all of your functions with spec definitions and generatively tests them only when the function or one of its dependencies changes. This would be extremely helpful since generative testing can be expensive to run, especially if you want to generate huge amounts of test cases. It'd be nice to say "test all of my codebase's functions 10 million times" and then have that information stored locally with the code.
In Rich's "Effective Programs" talk [2] he outlines the "10x problems" that Clojure was targeted at solving. In it there are two 10x problems that weren't solved by Clojure: resource utilization and libraries. There's not much Clojure can do to solve resource utilization as it's dependent on the host (JVM), but the library problem (especially Clojure libraries with spec definitions) is solvable (though there seems to be some disagreement on whether only allowing growth in libraries in a practical).
Pure speculation: All of their tooling projects (spec, codec, tool.deps, and the above mentioned generative test runner/tracker) are going to come together in a tool that attempts solves the library dependency problem. Rich outlines a lot of his thinking about how to accomplish this in his "Spec-ulation" talk [3]. He wants Clojure to "become the first community to make [dependencies] great". I hope that we'll see more tools and guidance from the Clojure team on this soon (and maybe improved out-of-the-box error messages)!
[1] https://github.com/clojure/tools.deps.alpha [2] https://youtu.be/2V1FtfBDsLU?t=29m53s [3] https://www.youtube.com/watch?v=oyLBGkS5ICk
The Clojure core team does not get anywhere close to the praise it deserves.
That's not terribly surprising. Clojure is incredibly stable. It's a pattern you'll see across the ecosystem. There are libraries that haven't seen a change in 2+ years which still see heavy usage and downloads. In that regard, it's unlike any other ecosystem I've seen. Don't let it dissuade you-- after a while, you'll get used to it and see it as a strength, and by comparison most other languages will seem insanely churny.
The low number of commits is somewhat intentional in that they actively avoid providing too much functionality in core and relative to Go which has the might of Google backing it there's fewer people with hands on who are incentivised based on innovation. It would be nice to see more frequent integration of commits from the community but ¯\_(ツ)_/¯.
Also:
> That is the number one problem in programming - the time [thinking] is not spent, and people flounder around.
That's a summary worthy to be framed if I ever heard one.
It's something a lot of libraries could learn from. Most users want to build software for the long term and tend to avoid upgrading libraries too often.
Fwiw, this is a large part of Rich's Spec-ulation [0] talk.
Knowing a set of functions I rely on did not change (at all) provides more peace of mind to me than just knowing the function signature is still compatible.
Versioning (or at least understanding changes) on the function level in addition to the library level sounds so powerful to me that I wonder why this concept isn't more mainstream yet. Are there other examples?
[edit: I found codec: https://github.com/Datomic/codeq#codeq]
I do think it can be very difficult to know if you, as a producer, are breaking any consumers. I'd love to see some sort of OSS build tool and CI system that automatically rebuilds and tests all consumers of your library to give you, the library author, more information about the changes you make before releasing.
There's also this to look at on the subject: https://lkml.org/lkml/2018/8/3/621
Reddit discussion: https://www.reddit.com/r/linux/comments/95b1hf/linus_torvald...
You just can't break backwards compatibility.
This was a big mistake Python 3 made.
Think about it like totally new language
At least Python's backers had the guts to release a breaking version 3, unlike some other dynamic languages, specifically Ruby and Apache Groovy. Ruby's backers have tentatively slated 2020 for a (MRI) Ruby 3 release after two more Ruby 2.x versions. As for Groovy, 2 months ago its backers canceled its version 2.6 release which was to backport the planned features of Groovy 3 into a JDK 7 compatible release. Looks like we won't be seeing version 3 of either language anytime soon.
The backward-compatibility breaking Ruby release often held up as a parallel to Python 3—Ruby 1.9—was released December 25, 2007, about a year before Python 3.
If Python had as much guts as Ruby in this area we'd be seeing a backward-compatibility breaking Python 4 around 2021.
I remember attending JSF Days in 2009 and how we would be all writing Groovy JEE Beans in the near future, show in different ways across a few sessions, and here we are.
I don't suffer from XML allergy and am pretty fine with Maven in what concerns Maven projects.
Actually I was pretty happy with Eclipse + Ant + ndk-build as well.
To this day the NDK already went through a couple of Gradle build variations and it still doesn't support NDK libraries across NDK projects, as AAR only support binary libs for Java apps.
I follow Fuchsia with attention, but it might end up just as Brillo.
If you have to absolutely must break backwards compatibility. Break it once, and never again, in a way you fix your language permanently. Like Perl 6.
Python broke backwards compatibility for a print statement.
* Ruby had better dependency management. Python's only recently introduced Pipenv which provides a standard approach for managing development and prod dependencies. VirtualEnv's been around for a while but didn't handle scope.
* Ruby wasn't used as extensively as Python in stuff like system libs and start-up scripts so system packages weren't constrained in the same way.
* Ruby's ecosystem was more focused around web development. Rails in particular was fairly early in adopting new versions of Ruby within about 1 year of the release of Ruby 1.9. Django on the other hand was about 5 years before it had it's first Python 3 compatible release.
* "microbenchmarks" generally got better with Ruby releases whereas Python 3 seemed to have gotten worse from what I remember. I don't think microbenchmarks are terribly useful out of the context of an actual application but many people use them as indicators.
* subjectively I think the Ruby community was more committed to unit testing which made "fearless upgrades" a little more palatable.
Similarly, TeX shows the halmarks of stability. I can reliably rebuild any document I've ever written. However, that means that the styles I used to use to write documents have to still be supported. Contrast that with HTML, where they constantly give new methods to do something because they changed their mind about how it should be done. Worse, they no longer support the old ways, because they were supposedly "never supported."
I think there is some merit to change driving progress. It is just frustrating when so much change is effectively user hostile.
It's weird to me when people claim clojure is dying, yet the ecosystem is evolving rapidly and people are making money with it. It's not a model that's easy to starve, it's too lightweight to collapse under it's own load, and there's no way to kill it, so how is it dying?
Mindshare ebbs and flows, but being THE programming language was never clojure's goal.
During the release of 1.9 there were some issues relating to spec, libraries, and build tools in the pre-release versions. So I'm not sure I would say the rule is followed to the letter such that you can consume Alpha releases are safe. It was prerelease candidate so no biggie. I know there were some discussions on the mailing list of a formal security audit of the Clojure but I'm not sure where that went but I'd be interested in seeing the results.
That's also an argument for keeping security holes around. By definition those are bugs and users (albeit malicious ones) depend on them. How dare you fix a security hole and break a perfectly working exploit!
Now, there is a tradeoff between security and convenience. That is also why fraud techniques haven't taken away credit cards as a thing. To that end, sometimes it is better to engineer a detection/mitigation strategy instead of removing the convenience.
And life seems to be full of more strawmen then makes sense. So, yes, you can easily find examples for this in either direction.
1. improving/providing secure defaults. 2. reducing the API surface so that usage is more evident. 3. refactoring that eases maintenance. 4. improving performance where appropriate. 5. probably others I haven't thought of.
In RFC style I would say it's a SHOULD rule rather than a MUST.
Using a concrete example relating to security; libressl maintained much of the API surface that OpenSSL provides. In essence they aimed to provide a "drop-in" replacement. However there were whole families of algorithms and functions which they deemed "unsafe/unfit" for purpose (e.g. FIPS algorithms). I think that's a perfectly valid exception to the rule. It acts as a canary in the coal mine and you have options; fix your code or defer upgrading.
I would advocate for thoughtful deprecation cycles over ossification of poor APIs and algorithms.
From the interview: “But there are really two very distinct kinds of change - there are breaking changes, where your expectations have been violated, and there are accretions, where there's just some more stuff where there wasn't stuff before. And in general, accretion is not breaking.”
To paraphrase stuff I’ve heard him talk about elsewhere, your library should never change a function to require more input or provide less output. If you decide to do that, call it something else and keep the old function around. He’s talked about function-level versioning too, touches on it here a bit. If I use library functions x y and z and the maintainers only broke (the contract for) functions a b and c, then I probably don’t care and can grab the latest version for its other new goodies. But that decision still requires me to read the library source or at least a changelog, even though it’s simple enough for the right tool to be able to report it to me. Spec looks like a step in that direction.
However, I have the feeling that its popularity has been decreasing over the past few years. I very rarely see job descriptions mentioning it. Have people observed the same thing?
My personal impression is the opposite.
I also think I saw that it ranked pretty low on various "popular languages" studies. I would be very happy to find out that I am mistaken TBH.
I really love Clojure, but these days I'm still mostly a Java/Kotlin guy. Haven't seen any Clojure job or Clojure being used in production yet, unfortunately.
Sadly, that is a common fate for Lisp(-like) languages nowadays. People are scared of unusual syntax.
Anecdotally, there are 5 startups within a 1 mile radius of my office (in Charlottesville, VA) whose primary stack is clojure.
[0] https://clojure.org/community/success_stories
[1] https://news.ycombinator.com/item?id=14302762&__s=r8nvw64mp3...
[2] https://github.com/clojure/clojurescript/wiki/Companies-Usin...
[3] https://www.indeed.com/q-Clojure-jobs.html
[4] http://hnhiring.me/ (search "Clojure", top right)
There are a few other clojure focussed companies in Bristol too.
Personally I would have loved Clojure to absolutely win, but being an elite language for senior devs isn't a bad situation either (I'm much enjoying my latest Clojure jobs and the peers I could find there).
Installation on Windows: Not yet available - see Leiningen or Boot instead.
That was a disappointment to me, seeing as the language has been around for more than 10 years now. Also, when I went to try the Leiningen route I found that the installer wouldn't work on Windows 10:https://github.com/technomancy/leiningen/issues/2412
So I decided to try another language instead... But eventually I returned to Clojure and managed to get it up and running by using the fix that's suggested in the GitHub issue above. I'm really enjoying the language, but I fully agree that it can be frustrating to install on Windows.
I also came across (not a Lisp-devoted series admittedly) but an episode of another podcast which focusses on Racket [https://softwareengineeringdaily.com/2015/11/04/racket-with-...].
I haven't found any podcast series devoted to (Common) Lisp or Scheme so far though.
This[0] is a better way to learn, in my opinion.
Then after that, maybe check out Brian Will's videos[1].
Then definitely check out this helpful video on web dev[2]. There's a better video somewhere but I can't find it anymore. It's just a guy walking his friend through how to make an isomorphic clojure web app. Very nice pair programming walkthrough guide. But I guess it's forever gone.
[0]: https://kimh.github.io/clojure-by-example/
[1]: https://www.youtube.com/playlist?list=PLAC43CFB134E85266