Clojure's machine learning ecosystem
codewithkira.com
codewithkira.com
> Smile 3.x is GPL-licensed, which poses some potential conflicts for some end users...The community consensus is converging around moving away from Smile due to the GPL-relicensing issue, focusing instead on Tribuo...
(tribuo is developed by oracle)
It's a really great thing that the java community has a high performance and well accepted (~5x stars than tribuo) ML package that's GPL. CF python where the top two libraries are developed by google and facebook. The GPL protects individual, independent developers.
I don't think it's right to recommend that new users move away from the package because of licensing issues; the fact that it's GPL now is a good thing for everyone except corporate users (probably a great part of readers). The people who might have GPL problems already know themselves when they'll have a problem.
I was going to chime in to agree but then I saw how this was done - a completely innocuous looking commit:
https://github.com/haifengl/smile/commit/6f22097b233a3436519...
And literally no mention in the release notes:
https://github.com/haifengl/smile/releases/tag/v3.0.0
I think if you are going to change license, especially in a way that makes it less permissive, you need to be super open and clear about both the fact you are doing it and your reasons for that. This is done so silently as to look like it is intentionally trying to mislead and trick people.
So maybe I wouldn't say to move away because of the specific license, but it's legitimate to avoid something when it's so clearly driven by a single entity and that entity acts in a way that isn't trustworthy.
I really like the package manager, I like that there’s an IDE with a visual step debugger (seriously, why do almost no lisp people value this?), that it’s multi paradigm, and that the language continues to improve and evolve.
Clojure just has so many finicky or annoying things about it. The design of deps.edn/clojure CLI and the lack of user friendly tooling is forever frustrating. Laziness and its implications being invisibly core to the language is annoying as well.
We also have a debugger in Emacs/CIDER.
Your other complaints read like you haven't dug really far, or they are outdated.
;; on Linux and OSX clj -Sforce -Sdeps '{:deps {} :aliases {:dev {:classpath-overrides {org.clojure/clojure nil} :extra-deps {com.github.flow-storm/clojure {:mvn/version "RELEASE"} com.github.flow-storm/flow-storm-dbg {:mvn/version "RELEASE"}} :jvm-opts ["-Dclojure.storm.instrumentEnable=true" "-Dclojure.storm.instrumentOnlyPrefixes=user"]}}}' -A:dev
;; on Windows clj -Sforce -Sdeps '{:deps {} :aliases {:dev {:classpath-overrides {org.clojure/clojure nil} :extra-deps {com.github.flow-storm/clojure {:mvn/version """RELEASE"""} com.github.flow-storm/flow-storm-dbg {:mvn/version """RELEASE"""}} :jvm-opts ["""-Dclojure.storm.instrumentEnable=true""" """-Dclojure.storm.instrumentOnlyPrefixes=user"""]}}}' -A:dev
————
I hate this so much
You can run a project with a library in a repl with a one liner - seems handy. Don't even need to create a file
What projects are you working on where you are using nested hash tables passed in as strings to the CLI that couldn't be written as data in a deps.edn?
What tools do you miss from lein (genuine question) moving to deps.edn was mostly painless for the teams I've worked wit
The “do-it-yourself” mentality that is at the core of Clojure dev culture results in a less rich library ecosystem. Not only you often end up writing custom code for a generic feature, the rest of your team cannot reuse their past experience but has to learn your custom code and maintain it.
"user friendly tooling" and ANY Scheme in the same sentence?!
I kid, but I always have held that Schemes feel like "tools for building a programming language" (the insanity that is (car ), (cdr ), (caadadr ) suddenly reveals its usefulness when you write an s-expression parser) and Clojure feels like "an actual programming language" with its benevolent use of brackets, build tools, and library ecosystem.
(defn my-func [arg0 arg1] ..) -- Clojure
(define (my-func arg0 arg1) ..) -- Scheme (of which Racket is one)
One thing I've always liked about Racket is its refusal to accept (if )s without else statements, insisting instead on using (when )s.
If you're familiar with Java, then you can think of Clojure today as going through its Apache Ant phase. Some people love it, some people hate it. I personally want to spend 0% of my time building build tools; thankfully there is a pretty solid build tool that is still maintained and will continue to work for the foreseeable future.
I started with lein, and still use lein for any 'production ready' project, but I'll use deps.edn for little personal scripts because in those cases lein feels like bloat. For me, using deps.edn was straightforward because of my previous experience with lein.
There is a lot of strange shade in the Clojure community; like that thrown at lein.
In addition to lein, the ones that get me a lot of negative attention are the community's opinion that (loop ) is an antipattern in spite of it being more performant than the comparable built-in functional patterns, and (first ) and (second ) being preferable to (nth )s in spite of (nth )s also being more performant.
1) The language evolving all the time is an anti-feature. Clojure is an oasis for the people who are tired of having their knowledge obsoleted by constant change that doesn't really improve anything. People can build whatever weird and wonderful features they want using macros in their own library.
2) deps/edn - the design seems to support toolmakers more than end users. I hated it myself (tinged by poor documentation at the time which has since been remedied). But for anyone else having similar issues; the answer is to jump to leiningen [0] which is by far an easier way to set up a Clojure project.
Deps is ok, but lein is what I'd recommend to anyone as a starting point.
Another thing that's nice with Racket is that it can produce binaries, and while the cross-compile story is kind of messy the language as such is actually, really, multi-platform. Even the GUI-library, which is quite nice, especially with gui-easy additions.
And I just love how easy it is to do everything in Racket. Creating an executable, adding a package, including a module, even spinning up a new file and putting it in SICP compatibility mode - all of these things require 0 memory or reference with a manual because they're so straightforward. Everything is very nicely designed.
We do, that is why Cursive exists.
The problem are the folks educated in SBCL + Emacs that never used anything else Lisp related, including the survivors of the Lisp hype wave like Lisp Works and Allegro Common Lisp.
The other issue with debugging is perhaps a "me problem", but it's not uncommon for me to have to re-discover where laziness is present in Clojure's ecosystem while debugging. I don't like laziness being implemented on core functions - I think they should be explicitly named alternate functions. Laziness everywhere + "super stable language" = forever annoying. I guess this is only an issue for others who seldom use Clojure as a hobby language - I'm sure people who use daily are able to cope better.
I'm grateful that I get to benefit from these community efforts.
I learned a new word today.
And I agree, but the end might be a bit optimistic at how often or efficiently that happens. In my experience, most are all too willing to accept the institutionalized choices.
Eventually ART, although here the process would be more convuluted.
And if one is willing to give away a couple a big piles of money, PTC and Aicas are still in Java town.
But is it just me or has it gotten awfully quiet around Clojure? I mean, it is of course expected that no novelty can stay in the lime light forever. But what has become of all of the excitement?
My impression is that Clojure failed to live up to the hype, but has found its niche - but it is a rather small niche. In the end, despite its attempts to differentiate itself from Lisp pitfalls, it has gone the way of all Lisps: it's this cool, intellectually stimulating language that in practice almost nobody uses.
Is that view wrong? Please feel free to correct me, and sorry if this post is about Clojure in general and not specifically about its machine learning ecosystem.
I would like to learn it and probably would like to work with it. Only downsides I see is reliance on JVM (which is also a strength) and not as good ways of recursion as other Lisps or Schemes (simply call the function, duh, because it just works and doesn't need extra keywords or syntax).
But I liked the talks and that is seems to have the functional datastructured one needs, if one wants to avoid mutation.
The problem is that an entire ecosystem is more than just a new language. Clojure has some really interesting libraries, but the rest relies on Java. It is a good ecosystem nonetheless. Julia is having the same issue. It has grown a very respectable ecosystem, but it is hard to grow further to catch up with Python in some fronts without massive corporate funding.
It is interesting to note that, at some point circa 2010, it looked like Clojure could revive Lisp-Stat with Incanter. But, eventually, the project declined in popularity and died. Reaching mainstream relies on getting a critical mass of developers at the right time and for the right domain.
I like Asp.net, but the thing I've run into several times in F# is there are breaking changes in libraries that the developers of F# can't keep up with.
- Giraffe doesn't support swagger docs, you have to use Asp.net directly, because the swagger generation isn't compatible with the way Giraffe does routing
- Microsoft Identity uses EF core by default. The EFcore.FSharp library doesn't support past .NET 6. You could use your own backend, but at this point, it's easier to just use C# interop
You could use C# for the routing of your app and F# for the business logic. But then there's the C#/F# interop story. Lots of little finicky type differences you have to convert. Whereas using Java from Clojure feels easy and natural to me.
And even if you use Asp.net directly and do interop correctly, you're basically using C# at that point. You're missing a lot of benefits of a functional language. It's not worth it in my opinion, even though F# is a wonderful language to me.
It's tough to learn the ecosystem in Clojure. But once you do, I've found the "small libraries" approach much better than the whole ecosystem tying itself to one web framework it can't even keep up with.
I personally don't have much issues using ASP.NET directly from F# (don't bother with Giraffe or anything like that); its not too hard. The routing layer is only a few classes/functions anyway - it isn't the majority of your program. I guess maybe I'm in a different position to you - I find for the few bits of ASP.NET config it isn't worth switching to C#. A few rules you need to follow but I would say that is a lot less than the Clojure learning curve and you get static typing still - that is just my opinion however.
My experience is that both languages have their pros/cons and tbh both have their challenges w.r.t adoption. My personal view is most of these challenges are not technical. The issues we complain about with the language as developers are IMO minor (i.e. things are still possible or decent compared to some other languages) and probably aren't the major factor in the wide scale technology choice away from FP in general.
For better or worse developers are often "languages takers" not "language deciders" in most big shops and this is where most of the jobs are - its only in niche elements/problem spaces (i.e small shops, dedicated niche teams, etc) where niche technologies can be worth it. This means not a lot of jobs; but when there is a job it can be higher paying. Big employers drive the market for technologies though when it comes to large scale usage and they want commodity fungible developers (i.e. resources), and are happy using a mediocre tech that gets the job done to do it. Unless there is a "killer app" where the language must be used to unlock the value most managements will stay clear of better tech - predictable, boring, commodity tech that I can get anyone in for (contractor, offshore person, etc) wins for them every time especially for the MBA/product manager/etc types. In the current economic climate developers are trying to meet the jobs that are being advertised as well (higher interest rates, less startup's, etc etc).
Like I said, if you want to use Microsoft Identity, you need to either write your own backend that doesn't use EFcore or use .NET 6 instead of 8. Why? EFcore is essentially a scaffolding library. You can't use it from F# without EFcore.FSharp. EFCore.FSharp doesn't support anything higher than .NET 6.
So now we have a scenario where the main auth library in asp.net doesn't work in F#. I've never had these issues in Clojure where I just flat out can't use a Java library.
It seems there is work in progress on bringing .NET 8 support here https://github.com/efcore/EFCore.FSharp/pull/162
Both identity and efcore both work fine with F#. What _doesn't_ work is the EF Core design-time support (having models auto-create your schema) and scaffolding razor pages with F# (razor pages just doesn't work in general).
Calling these APIs (authing your users, and interacting with the database) work fine.
I've used mostly database-first design, and tweaked the stock identity pages code anyways so these things haven't affected me a ton (though I _would_ like to use razor pages, since it seems nice and simple for the 80% use-case).
It's either "just the way I want it exactly" or "unusable", any smallest and most accessible but slightly different option from preferred way to make things work be damned.
I'm really sad to see a part of F# community being persistently vocal against .NET which makes F# viable and possible in the first place.
If you look in the database section of F# websites, maybe 20% of those libraries work. If you bring it up to people online, they'll say "yeah use Dapper, it's the only thing that works".
This just isn't the case in Clojure and I don't think the expectation is that it should be.
F# has had a good run with a lot of open-source efforts in different domains, but as some core parts of the domain evolve (asp.net), if there are changes made which make it more complicated or prohibitive to use F# (t4 templates, roslyn analyzers, custom code-behind behavior), it can seem like F# is on-track to become deprecated by the moving-target that is it's native platform tooling.
I don't have much of a doomer mindset about things, since from the evidence of the F# team's priorities, it's keeping up with CLR and C# specific features (check the F# releases for how many features are compatibility-focused) while also delivering on some very cool F# specific things (SRTP, nested record copy/update, etc).
In the end, I still ended up using F#, since it was only about 30 minutes of copy/pasting the classes/etc over once, and for the rest of the time in development I got to stay in F# land.
To be explicit, I did do what you say here. I scaffolded things out in C#, then just copied/translated that over to F# calls. All the APIs still work just fine.
Communities can't control themselves, always try to create an ecosystem on top of the ecosystem, some of them even are quite vocal against the platform that made their beloved language possible in first places.
Thus as the platform moves under their feet, everyone fails to keep up, to make anything usable one has to master now two stacks instead of one, and spend even more time tracking down issues.
Clojure's approach to embrace the host is a much better way to handle this.
I think there was hope that would happen, but Incanter simply didn't have the functionality or documentation needed to become a player in that space in 2010. It was a doomed effort because of the (intentional) difficulty of native interoperability on the JVM. They'd have had to rewrite tens of millions of lines of code to realize their goals.
However to be honest anecdotally I do think even if they aren't going down in user terms, FP languages are losing percentage market share. From my personal experience FP is no longer new; the hype of FP in general has died down. The hype has moved onto other things over the years (e.g. AI, blockchain, bare metal technologies that enable the above, etc etc).
I'm not saying FP doesn't have a lot of value. I enjoy, particularly when the program calls for it, FP languages like F#, Scala, etc. They still offer a lot of value IMO for all the usual spouted reasons.
IMO I just think the hype has moved from technologies that do things better (i.e. more efficient, less error prone, makes my job easier, etc) to technologies that do shiny new things. There are a lot of reasons for this, many of them aren't in fact technical but economic and political (e.g. management). Things like less jobs globally in tech (meaning my skills need to have wider reach), demand for specialist roles that aren't as language focused, etc play a part. Mainstream languages taking FP features as well makes the return vs risk of using a FP language even worse of a tradeoff - there's less perceived return for changing stack than before.
The language ecosystem reflects the smaller numbers of contributors.
The ideas behind Clojure (functional, immutable datastructures, homoiconic syntax, focus on simplicity, JVM interop) still stand strong IMO.
There are still some exciting projects done in Clojure: Electric Clojure and Rama come to mind.
Honest question: what domains?
The further you get from silicon valley the more clojure you see — anyone with a network at FANG spent the last 10 years trying to break in and get that huge salary, which means conforming to the recruiting process, i.e. spending your time off grinding leetcode instead of learning new PLs
I also don't buy that you see more Clojure the "further you get from silicon valley ..." without some actual (or even anecdotal) data to support that claim.
I'm not saying there are zero instances of Clojure in production systems. But it is hard for me to think that it represents a large number of professional employed software engineers. The 2023 StackOverflow survey show 1.38% out of 67,053 respondents in their "most commonly used programming language" section naming Clojure. That is less than 1,000 developers. The public university I work at graduated 500+ CS undergrads in 2023. I would bet $100 that zero of those got a first job that paid them to write Clojure professionally. I would bet $1000 for less than ten grads. I would bet $50,000 for less than 25.
I'm not trying to be cute but don't you think there are less than 10,000 software engineers in the US (maybe the freaking world) who primarily write Clojure code in their day job?
Because I'm curious, job searches at the company site of your list for 'clojure':
Nubank - enhum resultado com a palavra "clojure" encontrado (no results)
Guaranteed Rate - no results
Kroo Bank careers page - no tech listings currently
Griffin - open listing for software engineer (London or Remote within the UK, Germany, Sweden or Ireland) - description: Our backend stack is Clojure, FoundationDB, Kubernetes and AWS. Our frontend stack is CLJS, Reframe, Reagent, React, Stitches, Storybook, and Playroom.
Pennymac - Sorry, no jobs were found that match your search criteria.
Two Sigma - No jobs found
Dividend Finance - 0 JOBS FOUND
Treasury Prime - There are no current openings. (for software engineers)
Gravie - Lead software engineer position - description: Advanced programming experience in multiple programming languages ( Java, Kotlin, Groovy/Grails, JavaScript/TypeScript or Python) along with Clojure/ClojureScript or another functional programming language
I love Clojure - it has always "worked" in my head as a language to solve problems in. I was at the first Clojure/conj. I have deployed production code that I wrote in Clojure. Clojure made me a better developer in other languages.
But in the big picture for professional software engineers, Clojure might as well not exist.
It also seems Rich Hickey kind of stopped doing interesting talks, and some big bank bought into it and no one really likes banks.
My last few jobs were in clojure so jobs do exist, moreso than in Common Lisp and Haskell at least (probably). There aren't very many companies that use it relative to the programming world at large, so I'd say you're right on that front.
I'm happy using it and will continue using it. Sometimes I encounter interesting ideas and libraries that I end up using.
At the end of the day, it's a programming language. You can write interesting things on uninteresting or even subjectively awful languages and boring things on interesting languages.
Here's something I've been working on for the last two weeks: https://keyboards.justbuythisthing.com/. If I didn't tell you I wrote it in clojure, no one would know. It could very well be PHP.