Specter – Clojure API for immutable programming (2017) | Hacker News Reader

Specter – Clojure API for immutable programming (2017)

nathanmarz.com

152 pointsamgreg37 comments

Specter is tremendously useful and rarely lets on a case where it can’t perform a transformation. For ensuring that you’re getting the most efficient form of a transformation [0] it’s incredible —- and it is somewhat data-oriented, allowing easy saving and reuse of navigators.

Meander [1] is also wonderful. Its unification-based approach makes for nice expressions of intent. Its strategy-based approach to rewriting [2] is more flexible than most common walker libraries, too.

[0]: https://github.com/redplanetlabs/specter/wiki/Specter's-inli...

[1]: https://github.com/noprompt/meander

[2]: https://github.com/noprompt/meander/blob/epsilon/doc/strateg...

So specter is lens and meander is syb/generics. Nice

I know the author of meander, and I kind of doubt they'd describe it this way. I will ask them, though.

Specter is fantastic. It makes transformations on deeply nested data structures easy. Want to transform all the "datetimes" from JSON data into actual datetimes (or vice versa)? It takes about five minutes if you have to look everything up. Works with ClojureScript too.

Some production code:

    (defn transform-tagged-values [x]
      (sp/transform (sp/walker tr/tagged-value?) tagged-value->cljs x))

Give it a predicate for which values to transform, and a function to do the transformation. Couldn't be easier.

It really does remove multiple layers of complexity that usually stand between your mental model of what your transformation function should do and the code that actually implements it. With Specter if I can clearly articulate the transformation I want to do on a data structure, then I can almost always translate that one-to-one into the code that does the transformation.

When I started using it in one of my existing projects I was able to completely eliminate a few needlessly long and complex functions that were littered with comments and calls to helper functions. In their place are functions that are about 5 lines of code that barely even need a docstring because it's readily apparent what they do just by looking at the source.

I invented instar (https://github.com/boxed/Instar) which is quite similar to specter (imo a little bit nicer) because I also thought the built in API was weirdly clunky in these respects.

I have used both Spectre and Instar. I prefer Instar for straight forward simple transforms, but Spectre can do more complex ones. Spectre has a more difficult API to wrap your head around and that is its biggest downside, IMO. You really need to dedicate time to learn it rather than try to pull it out on demand.

What's the performance vis-a-vis Specter?

I have never run a benchmark and the specter guy really cares about it so instar is probably way worse :)

I was more trying to show that the API could be better and hoping this would put pressure on core devs to do it properly.

Very nice syntax, looks great.

Specter is always on my clojure toolbox. Not only it makes transforming arbitrarily nested data painless, it also is usually more efficient than the average hand-rolled code!

A great example of why macros in Lisp are powerful, and how they empower you as a user even if you never write one.

Previous discussion of Specter: https://news.ycombinator.com/item?id=10295060

also cf

Glom for Python: https://sedimental.org/glom_restructured_data.html

Hashdig in Ruby's standard library: https://ruby-doc.org/core-2.7.1/Hash.html#method-i-dig

So it turns out that Clojure is the ultimate data-wrangling tool yet there's no chance of earning a living with it. So much for innovation. Technology is a strange industry where conservative Java bondage and a 2-bit browser scripting extension with no stdlib reign supreme.

This is totally incorrect. Join the Clojurians slack and ask around about work. There's plenty opportunity out there. Apple, WalMart, DRW Trading, NuBank, Capital One, and the list goes on and on.

There are always a few outliers for any language but when you look at a larger sample Clojure's failure to gain adoption is undeniable. Take contract roles on IT Jobswatch, for example. 12 conracts in the last 6 months in the whole of London and 1 in the rest of the UK compared with Scala for which there were 279 in London and 128 in the rest of the UK. I'm not a Scala fanboy either. I remember back in 2013 when Clojure was hot and Clojure meetups in London were well-attended. Those days are over. London Clojurians has struggled to attract a handful of attendees since long before the COVID19 crisis.

> Clojure's failure to gain adoption is undeniable

What... are.. you.. talking... about? Do some googling around. Clojure today is the most popular in its category of the languages with strong FP emphasis and slowly and steadily growing. It has already generated more jobs, books, active podcasts, conferences. More than Haskell, OCaml, F#, Elm, Purescript, Erlang, Elixir, Idris, Julia, and has recently surpassed Scala and become 3rd most used JVM language after Java and Kotlin.

Of course, one can say: "actually, none of these have gained much adoption in the industry", and speaking in relative numbers, they'd be right. But those who can no longer bear the pain of writing awful, deeply nested Java class hierarchies or tired of debugging concurrency issues would always find projects to write and sometimes even get paid for, using their preferred tool.

I honestly don't understand people, when they say: "It's really nice, I love it, but I'm getting paid to use something else." Well, do you really expect someone to call you and say: "Hey, I saw your comment on HN, it seems you'd love to use Clojure, can we offer you a lot of money?", or something like that?

I for one, after only a week of playing with Clojure, immediately quit my job. It took me just a few days to find a Clojure gig. I switched jobs three times since then, never (even during the COVID-19 crisis, when I was laid off) was a question for me "Is the fun over? Do I have to start looking for Python jobs now?" Luck of course is a factor, but you have to herd, guide and push your luck forward. It's no one's fault that you are forced to use something you don't like.

"... recently surpassed Scala and become 3rd most used JVM language". If we're talking about real, paid jobs then you're not addressing the statistics I quoted. Scala adoption is an order of magnitude higher than Clojure. I'm not talking popularity amongst developers - just real paying jobs which can be relied on to put food on the table and pay mortgages. Sun and Oracle's marketing of Java in the late 90s/early 2000s is a juggernaut compared with which an individual's luck in finding a Clojure job pales into insignificance so, yes, they are responsible/at fault for Java's dominance of paid programming work.

Clojure’s just a Java library, friend.

I wonder how many fanfold pages of Java, complete with getters, setters, toString and hashCode, one of Nathan's examples expands into?

Not really. I put together a few github projects and was approached about two jobs over a few years just under 200k... I said no, because even though my github projects were simple when I wrote them, reading them later was insane. Reopening my java projects after years is the exact opposite - my IDE tells me everything I need to know and reminds me along the way. It's like a warm fuzzy blanket.

I'm almost 99% sure both jobs were places that bought into some dynamic developer who promised unicorns and rainbows and then left them with a huge clojure project have their eyes bleed to. No thanks.

Clojure is wholly inadequate to meeting challenges of the industry today. Not just because it's dynamically typed (which means tooling for it is close to nonexistent and it has terrible performance), but also because its ecosystem is a ghost town.

It takes more than just a language to create a productive development environment in 2020, especially one which is so flawed by designed to the point of not even being statically typed.

Your Google skills must be short. Clojure community is on fire with open-source and it’s IDE and build support is copious and stellar. Also it’s not statically typed on purpose; the reasons for this have been described & analyzed at length. I’d recommend you get out more.

Yeah, just like Common Lisp and Julia have terrible performance. Extremely slow, can't do anything with them.

> Clojure is wholly inadequate to meeting challenges of the industry today

Interesting point. Sadly, it seems engineers at Apple, Cisco, Walmart, NASA, CircleCI, Pandora, Grammarly, Funding Circle, Nubank and hundreds of other companies are amateurs who don't know jack-shit about challenges in the industry and keep building their sand-castles using their incredibly flawed tools. I'm sure, these losers would soon vanish from the radar and we would never hear about them. Just give it a few years.

Darn! This is sleek!

I have been learning Clojure off and on for almost a year now. I gotta say that it has changed my mind to a "Aha, I got lisp now" moment. Working with it is a joy, especially it integrates so well with the JVM and all available Java libraries out there.

I can't believe I have missed Specter during my learning journey. How could I not find out about this soon enough. I mostly deal with static typed languages and seeing the primary use of map and keywords as main data structure scares me. Rich Hicky said not to put too much emphasis in what in the map but the data/keywords you need, kind of like you do not care what in the UPS truck as long as it could deliver your package, right? But, navigating maps and nested data structures around is the pain. Specter definitely easies that pain a lot now that I know it!

This is why I am grateful for hackernews. Once in awhile, browsing the site and discover something like this makes me rethink the ways I do stuff. It also reminds me that I could be missing a lots of similar goodies on my way learning Clojure. I am prepared to have me mind blown again!!!

There's a lot of alternatives in this space

Meander was the last one I can remember:

https://www.youtube.com/watch?v=9fhnJpCgtUw

Like SQL for nested maps in Clojureland. Neat. The DSL [ALL :a even?] does feel a tiny bit un-clojurey to me as well, maybe making it look like [:all :a even?] would make it clearer.

Well there’s a reason it’s like that — ALL is actually a value, not just a sentinel or keyword. It’s found at com.rpl.specter/ALL.

Keywords don’t form a DSL in Specter, they just do keyword navigation into maps, the same way ‘0’ navigates into the first element of a list.

This touches upon a really nice element of Specter, which is its pattern for reuse. Here’s a very real but deliberately simple example of using it:

  (def chat-threads
    {[“sova” “tekacs”]
     [{:text [“hey” “you there?”]}
      {:text [“line 1” “line 2”]}]})
  
  (def ALL-TEXT
    [sr/ALL :text sr/ALL])
  
  (defn text-for [chat]
    (sr/select [chat ALL-TEXT]))

  (def ALL-CHAT-TEXT
    [sr/MAP-VALS ALL-TEXT])
  
  (text-for [“sova” “tekacs”]) ;=> [...]

Here the navigator into a list, then the :text field and then the list is bottled simply by putting several navigators in a vector.

Then, it can be used alongside other navigators in a first class fashion — [chat ALL-TEXT] is just a nested vector. :)

Well, not sure about that exactly. Consider a map like:

    {:all {:a 2} :foo {:a 4}}

Maybe namespaced keys would fit that usecase

In instar it's [* :a even?] which I think is prettier... being the author of instar :)

Specter is really handy for manipulating data structures, and have been happily using it.

I wonder, is it possible to select some subset of a map? Say you have these paths:

(def paths [[:some :path] [:another :deeper :path] [:yet :another :one] [:yet :another :one :deeper]])

And select from a map these paths. You could do it with get-in, but maybe Specter can do it faster? I couldn't find a way to do it with Specter...

I hope that more and more lens implementations should catch on in languages with immutable data structures.

They really are the key to make immutable data structures as easy to update and work with as mutable data structures.

TXR Lisp:

  This is the TXR Lisp interactive listener of TXR 243.
  Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet.

Give me all the .c files:

  1> (glob "*.c")
  ("abcd.c" "args.c" "arith.c" "buf.c" "cadr.c" "chksum.c" "combi.c"
   "crazyffi.c" "debug.c" "dict.c" "eval.c" "ffi.c" "filter.c" "ftw.c"
 "gc.c" "glob.c" "halfsip.c" "hash.c" "highest-bit-main.c" "highest-bit.c"
   "highest-test.c" "isqrt.c" "itypes.c" "lex.yy.c" "lib.c" "lisplib.c"
   "match.c" "parser.c" "protsym.c" "query.c" "rand.c" "regex.c"
   "signal.c" "socket.c" "stream.c" "struct.c" "strudel.c" "sysif.c"
   "syslog.c" "termios.c" "test.c" "tree.c" "txr.c" "unwind.c" "utf8.c"
   "vecho.c" "vm.c" "y.tab.c")

OK, which ones are over 100000 bytes:

  2> (keep-if [chain stat .size (op < 100000)] (glob "*.c"))
  ("arith.c" "eval.c" "ffi.c" "lex.yy.c" "lib.c" "match.c" "stream.c"
   "y.tab.c")

Clojure is already good at that- at conditional pipe / filter / aggregate semantics, bc it is a lisp. Specter is for a different problem. When you have a data structure that has some semantics and is deeply nested (and you aren't discovering the structure, like getting size from a node via a function call) Specter has a declarative set of operations to navigate and extract on that data structure, extremely efficiently.

Adding a bit of context - as Clj data structures are immutable, changing something that is deeply nested cannot be done in-place (like in a Java set of nested objects - navigate until you find the one you need, call setFoo() on it, you're done). So it gets painful real quick. Specter is like search-and-replace - first you find all the elments you need, then you can replace / change them at the touch of a fn.

  1> (defstruct node ()
       item
       next)
  #<struct-type node>
  2> (new node next (new node next (new node item 42)))
  #S(node item nil next #S(node item nil next #S(node item 42 next nil)))
  3> (call .next.next.item *2)
  42

I don't usually have a problem with you posting TXR comparisons in every lisp thread, but if you're not going to take the time to understand what the library does, this is just spam.

Reply on news.ycombinator.com