Programming at the REPL: Data Visualization | Hacker News Reader

Programming at the REPL: Data Visualization

clojure.org

104 pointstosh20 comments

One thing that's really underappreciated in my opinion is the ability to use the REPL as a general tool for interacting with the system. A lot of devops tasks can be done very conveniently from the REPL.

For example, you can start a Babashka nREPL by running bb --nrepl-server and connect an Clojure editor to it. Then you can do a lot of fun stuff from there. For example, if you have osquery installed, then you can start querying your system for all kinds of stuff, get data back in EDN and manipulate it using standard Clojure functions:

    (require '[clojure.java.shell :refer [sh]]
             '[cheshire.core :as json])

    (defn osquery [query]
      (let [{:keys [exit out err]} (sh "osqueryi" "--json" query)]
        (if (zero? exit)
          (json/decode out true)
          (throw (Exception. err)))))

    => (osquery "select * from routes where destination = '::1'")

    => ({:hopcount "0",
         :interface "lo0",
         :mtu "16384",
         :type "local",
         :source "",
         :gateway "::1",
         :netmask "128",
         :flags "2098181",
         :destination "::1",
         :metric "0"})

I find I often just leave a bb nREPL running with Calva connected to it and some scripts to do common system tasks in it.

https://www.osquery.io/

https://github.com/osquery/osquery

The difference is that each of those are their own little silo, instead of being part of a single language ecosystem.

I always mention that a shell is a poor man's REPL, based on REPL based OSes like the ones from Xerox and ETHZ used to offer their OS interaction workflows alongside the development experience.

Shells like Powershell and Fish are the closest to it, due to their integration with programming language and OS APIs.

Completely agree, having an interactive programming interface to the system is superior to using a shell in practically every way.

Why do you think Powershell and Fish are closer than EMACS Eshell to the REPL based OSes?

They aren't part of a text editor, and integrate with the complete OS.

Emacs is a shadow of the REPL experience from using a Lisp Machine.

I think the time is right to revamp a basic shell. Something that generalizes pipes to async DAGs. The idea floats in the air between reactive objects graph in a browser or systemd services, docker composer ..

It would reseat the base os user interaction to something cleaner and leaner IMO.

This is great! For any SWEs that would be more comfortable in a Python REPL, you can do the same thing like this:

    import subprocess
    import json

    def osquery(query):
      return json.loads(subprocess.check_output(["osqueryi", "--json", query]))

    #>>> osquery("select * from routes where destination = '::1'")

    #>>> [{'destination': '::1', 'flags': '2098181', 'gateway': '::1', 'hopcount': '0', 'interface': 'lo0', 'metric': '0', 'mtu': '16384', 'netmask': '128', 'source': '', 'type': 'local'}]

I hadn't heard of osquery - thanks for pointing me to that!

clojure.inspector is an interesting namespace, but in practice I haven't found it very useful.

- It is rare that this problem is the limiting factor. In Emacs, which has horrible problems rendering long lines, usually by the time I realise there is a visualisation problem my editor has died.

- There is a thin band between too-big-to-print and too-big-to-visualise where this tool is useful. Almost all practical data seems to fall outside this range.

That said, Clojure already has some good tools for solving this problem. Immutable data structures and good generic primitives that are easy to inspect make it fun to deal with inspecting objects.

> - It is rare that the problem is the limiting factor. In Emacs, which has horrible problems rendering long lines, usually by the time I realise there is a visualisation problem my editor has died.

Sounds like the problem is your editor, not the Clojure namespace :)

Maybe try a real editor, such as vim/neovim ;) It gets a bit bogged down, even on my desktop computer, but at least it never crashes.

There is nothing wrong with the namespace. I just never found situations where it was useful.

And I see I accidentally wrote "the problem" instead of "this problem". I fixed that.

As a user of Common Lisp but not Clojure, where does Clojure dev environment stand with respect to something like SLIME? What unique features do Clojure dev environments have?

A closer experience to the old Common Lisp commercial IDEs, when using cursive.

https://cursive-ide.com/

I’ve only dabbled with CL/SLIME, but Cider seems to me to be a very close analog (and is similarly very useful).

Similar to SLIME, there is Cider (https://docs.cider.mx/cider/index.html) for Emacs.

There are a lot of debugging and data visualization tools for the repl with features I haven't found in Common Lisp like :

- https://github.com/jpmonettas/flow-storm-debugger (which I'm actively working on)

- https://github.com/djblue/portal

These look like nice tools. In CL land I think we can find some of their functionality in other tools than Slime, such as SLY, a fork of Slime with more features:

> - Instrument any Clojure form

This would be Stickers: annotate any Lisp form with a sticker, run your code, interactively walk through the recordings.

https://joaotavora.github.io/sly/#Stickers

> - Provide a GUI to explore your values and execution flow

SLY has an improved trace dialog: https://joaotavora.github.io/sly/#Trace-Dialog Also LispWorks' visual stepper is easier to use than SLY or Slime's. There is also an in-progress portable visual stepper for CL: https://zenodo.org/record/3742759

> time travel stepper

ah. Nice job.

Nice, thanks for the links.

Yeah I don't know how stickers trace code, but FlowStorm tracing debugger takes advantage of the immutable data structures used everywhere and by default in Clojure. You can then instrument entire code bases, run them, and tracing is just a matter of retaining(leaking) pointers of every intermediate expression so the GC doesn't collect them. I don't think this is possible when working with mutable objects, since then only way would be to somehow serialize the objects involved at every step so you can analyze the data later, which is prohibitively expensive in most situations.

SLIME was used in the early days of Clojure but is long sinced replaced by Cider (also emacs) or other impressive tooling, i.e. add-ons for IntelliJ (like Cursive). I would say there is nothing that SLIME offers that was not replicated or improved upon in Clojure's tooling.

I'm curious what you would count as improved upon?

Reply on news.ycombinator.com