Scripting with Elixir | Hacker News Reader

Scripting with Elixir

underjord.io

210 pointslawik48 comments

I've been finding Livebook (https://livebook.dev/) really useful for iterating on a script or manipulating some data in a reproducible way. I'll often try out one solution and if that doesn't work, create a new section and collapse the old one to be able to go in a different direction.

I really want scripting in Elixir to be better - and the addition of Mix.install() is a great step forward - but there are still a lot of awkward elements. Elixir will require you to be in a module to define a function and then respect module namespaces when calling them - which makes sense in a normal context but I find it to be annoying while scripting. Likewise I don't think dialyzer, the Elixir / Erlang static type analyzer, will run on script files.

That said I think the underlying lack of data mutability really increases the potential for code reuse and scripting.

I don't see a problem with defining modules in Elixir scripts. It was a huge annoyance in Erlang though.

Even scripts require some structure and code quality. I organize my bash scripts, otherwise they quickly become unmanagable.

If you dislike typing long module names, and for some unknown reasons won't use short alias (alias My.Long.Module.Name, as: M) or import. Then the best workaround is to use "M" as a module name:

    #/usr/bin/env elixir
    ...
    
    defmodule M do
      ...
      def f, do: ...
      ...
    end

    ...

    M.f()

Anyway, I think there was Joe Armstrong's blog post, where he said that modules are just containers for functions. The way we decide which module to put a function is completely arbitrary. Sometimes it's very hard to decide, or maybe the same function could equally belong to several modules. I designing my toy programming language where functions do not nelong to modules or namespaces, but you can query them based on metadata like tags/types/docs/etc.

Agreed, I find this a bit annoying too. Makes me create a module earlier than I'd have wanted (but not too much trouble though).

One can still reuse code with "Code.require_file", which is nice to have.

you can define functions outside of a module using fn blocks

There's good odds it will come to elixir, since support just landed in Erlang

To be specific, it has landed in the Erlang shell with Erlang/OTP 26. Defining inline functions in source code, on the other hand, has been possible for years (like 20+ years).

There's also defining a module and then `import`ing it.

In the Ruby language you are implicitly in the context of the Main object (IIRC). If scripting was it's own context then something like this could perhaps work for Elixir. You're already in a module context implicitly.

sum = fn (a, b) -> a + b end

sum.(2, 3)

Maybe a but clunky but no module required.

Mostly gets really clunky for recursive functions.

interesting. As a data engineer I have been indoctrinated to think about things as various transformations and so the functional approach to things is very nice. I've also been wanting to learn/play with Elixir.

In the Python space, though not the same, the ergonomics of which kinda get you there, there's this: pipe https://pypi.org/project/pipe/

The pipes are really very nice.

They are actually even nicer under Livebook: https://livebook.dev/

It can let you drag-drop reorder, enable/disable, steps in a pipeline: https://github.com/livebook-dev/livebook/blob/main/lib/liveb...

It is really wild.

I like pipes but I'm not sure if it's pythonic. What if other people need to maintain/debug the code?

Granted, it's not as much of a learning curve as personal Lisp macros.

If you haven't played around with it - you might dig Livebook https://livebook.dev

Lost an opportunity to name it Pype

Of course in Elixir[1] reduce is more efficient (at least memory-wise) than map/flat_map, but in case of scripting it usually doesn't matter (unless this JSON file is huge, but it's fully loaded into memory and parsed there anyway).

I would rewrite the example from the post like this (NOTE: I didn't run it, so it might be slightly incorrect):

    #/usr/bin/env elixir

    "file.json"
    |> File.read!()
    |> Jason.decode!()
    |> Enum.flat_map(& &1["children"])
    |> Enum.map(&{&1["id"], &1})
    |> Enum.into(%{})

---

1. although map is much easier to parallelize/vectorize (e.g. in Array PLs)

It's slightly more verbose but I have mostly stopped using the & notation to produce functions out of thin air[1]. I find the following code is much more easily readable - more welcoming to the less experienced, less brain load for the more experienced:

    #/usr/bin/env elixir

    "file.json"
    |> File.read!()
    |> Jason.decode!()
    |> Enum.flat_map(fn x -> x["children"] end)
    |> Enum.map(fn x -> {x["id"], x} end)
    |> Enum.into(%{})

[1] The `&a_function_with_odd_arguments(&2, &1)` type notation can still be very useful and is (subjectively) a bit less symbol-salad.

I dislike Elixir's abuse of sigils (it's inhereted from Perl via Ruby).

The capture operator syntax in Elixir isn't a very ergonomic design.

It uses the same symbol for both fun and arguments, and also for MFAs (&M.f/a).

It doesn't allow nesting, so in these cases there is no choice but to use the full notation.

In case the intent isn't clear from the code, I'll also use the full notation.

Also, I think it's much more acceptable to use shorthand/sugar in one-off scripts rather than in "real" code.

NOTE: in your example you use "x" for argument, so it's not much better than "&1". Since you already using the full notation, it's better to meaningfull argument names, such as "item" and "subitem".

For readers not used to Elixirs operators - which I am - "x" is much easier to understand than "&1" for two reasons. First, because the syntax of the full notation resembles how anonymous functions are done in almost every mainstream language that has them. Second, the character "x" is very often used as a name for the first parameter of an anonymous function, which reinforces the first point.

If I see only only this: (x,y) -> ... I can guess quite well what x and y are used for, and I suspect I am not the only developer who recognizes this as a 'parameter list' partly because the names happen to use 'x' and 'y' which are otherwise non-descriptive.

Nevertheless, your point still stands.

If you alrady decided to use a full form, then the difference between

   fn x -> ... x ... end

and

   fn item -> ... item ... end

is not that far, but it makes a huge different for other team members and even for the future self.

Beside Elixir/Erlang, I also a kdb+/q (an APL family language) developer, and there the 1st,2nd,3rd arguments of the function are called x, y, z correspondingly by default. E.g.

instead of

    f: {[n] n*n}
    f 5
    => 25

you can write:

    f: {x*x}
    f 5
    => 25

or instead of:

    add: {[a;b] a+b}
    add[10;20]
    => 30

you can write:

    add: {x+y}
    add[10;20]
    => 30

There is a wide range of syntactic sugar and shorthands in PLs. If your PL supports multiple options, you need to choose the best one depending on the context or situation.

Having said that, I believe that modern PLs plagued with the flexibility syndrome, and I would prefer it if the PL designers selected a single approach and sticked to it.

I never buy the "for those not used to the language" argument. People are going to learn the language and then it won't be a mystery anymore. `x` is only clearer than `&1` if you don't want to get used to the syntax, but once you're used to it it becomes second nature and is much more concise in certain situations. I mostly only use it for things like `Enum.map(users, & &1.name)`. Basically I never use &2 (well, I probably have once of twice before, lol).

You can skip the last Enum.map/into and just use Map.new:

    "file.json"
    |> File.read!()
    |> Jason.decode!()
    |> Stream.flat_map(& &1["children"])
    |> Map.new(fn child -> {child["id"], child} end)

I haven't used Elixir for scripting yet, but I really like the idea of including deps at the top of the file. It's always been a pain to manage this in other scripting environments. I did scripts with Ruby in past and it didn't end up being easy to use because of external deps.

TBF ruby lets you use inline bundler. See https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...

Oh that's awesome. I didn't know about this (or not sure when it was released.)

edit: it's been out since 2015 it looks like. Not sure why but I didn't know about this before. It would have been useful.

It's not that well known, and it's awesome too. I used that last week on "fire rescue production" scripts, and it was quite useful.

I'm glad that my 2 favorite languages have that feature.

You can declare dependencies at the top of a script file in a few languages, though usually through a third-party tool. I have a page on my site with a list of such languages and tools: https://dbohdan.com/scripts-with-dependencies. For example, for Crystal there is https://github.com/Val/crun.

I wish more developers knew about single-file scripts with dependencies. People seem to only discover them accidentally through a language or runtime that has the support built in. It means Elixir will contribute to the awareness, but Elixir itself is niche. Maybe Deno and Bun, as JavaScript runtimes, will popularize them. Having a common term not specific to any language may help, as there doesn't seem to be one. It hinders search. "(Single-file) scripts with dependencies" is what I have settled on.

Edit: Added the Crystal example and the second paragraph.

I'll second @josevalim's comment, but I also wanted to mention how delighted I am to find out today that a joke[1] I probably spent way too much time 7 years ago has actually served a useful purpose!

1 - https://hex.pm/packages/leftpad

Haha. Thanks for writing it. I appreciate your and every leftpad porter's effort. :-) It makes for a perfect little dependency test with a standardized name. (Standardized up to /left[_-]?pad/i.) When a language doesn't have a leftpad package, I go looking for a concise way to make terminal text bold, which takes more time.

Plus, seeing code like https://github.com/colinrymer/leftpad.ex/blob/8d2230bf094eed... is just fun:

  defmodule Leftpad do
  
    @moduledoc """
    Remembering `String.rjust/3` can be difficult, so Leftpad provides you
    another way to easily left pad/right justify your UTF-8 encoded binaries.
    """
  
    @doc ~S"""
    Provides basically the same functionality as [`String.rjust/3`](http://elixir-lang.org/docs/stable/elixir/String.html#rjust/3)
    ## Examples
        iex> Leftpad.pad("foo", 5)
        "  foo"
        iex> Leftpad.pad("foobar", 6)
        "foobar"
        iex> Leftpad.pad("1", 2, ?0)
        "01"
    """
    @spec pad(string :: String.t, count :: non_neg_integer, char :: char) :: String.t
    def pad(string, count, char \\ 32), do: String.rjust(string, count, char)
  end

Look at that one line doing all the work, and even it delegates it to another function.

why not:

    defdelegate pad(string, count, char \\ 32), to: String, as: :rjust

Ha! I just found this the other day and thought it was for real, lol. I wasn't actually looking for leftpad, of course, but, well done!

This is a very nice resource, thanks for sharing!

good ol' nasty jupyter notebooks support this:

    %pip install foo~=1.5
    %pip install bar~=1.7

    import foo
    import bar

This would require you to install and run jupyter. Mix.install just requires Elixir being installed.

That's neat.

I've been a big fan of "Mix.install" since it was released!

I believe this made me use Elixir instead of Ruby (my natural scripting language) more and more.

I use it on a regular basis for my work on https://transport.data.gouv.fr/?locale=en (which is Elixir-based and open-source), as one can see at https://github.com/etalab/transport-site/tree/master/scripts.

What I like the most is that it helps me start ideas, experiments & data analysis with their own set of dependencies (without much care of "will this impact the main application?"), store those experiments in the same repo at the main application, and maybe later promote some of those experiments to the main application source code.

Concrete examples include:

- data wrangling https://github.com/etalab/transport-site/blob/master/scripts..., https://github.com/etalab/transport-site/blob/master/scripts...

- exploring a new tool https://github.com/etalab/transport-site/blob/master/scripts...

- gradually iterating to create scripts to generate XML queries https://github.com/etalab/transport-site/tree/master/scripts... (later incorporated in the application)

- comparing GTFS (transportation data) files via scripting https://github.com/etalab/transport-site/blob/master/scripts...

- learning how to compute the checksum of the internal content of a zip file https://github.com/etalab/transport-site/blob/master/scripts...

As mentioned in the article, I can definitely recommend to check out https://github.com/wojtekmach/mix_install_examples which has a long list of examples.

One last tip is that you can also use "mix run script.exs" on a file not using "Mix.install", in order to rely on the same dependencies & configuration as the main application (e.g. to run a Ecto query https://github.com/etalab/transport-site/blob/master/scripts...).

I expected a mention of escripts.

I didn't understand the use case. I had chatgpt try to find one, maybe dependency management(eh, I got conda for that) and parallel processing(eh, I got pool for that).

Seems like a no-brainer to go with python for the long term. Why would anyone take the time to use this?

> Overall I would recommend scripting in whatever language you are most comfortable in that is at least reasonably comfortable for scripting. A quick script benefits from a low barrier between thought and execution. Use what makes sense for you. I find Elixir surprisingly good as a scripting language ever since the introduction of Mix.install.

Because the author likes Elixir and its trivial concurrency story. But they also wrote the above: Use what you want, for them it's Elixir. Why complain about other people's preferences when they aren't actually foisting it on you?

If you already write a lot of Elixir for it's application-level superpowers, then being able to hammer out a script in a language you're fresh on is nice. Combine that with better developer ergonomics (package handing in Python is inane, for one), and it's not a bad choice.

No one should be learning Elixir solely for the purpose of writing short, one-off scripts, though.

On the other hand, I find that starting to learn a language with one-off scripts is a nice gateway drug. This is how I started using Ruby initially!

I mean, if you value popularity more than any sort of intrinsic quality of your tools then, yeah, sure, it's a "no-brainer". But that rather strikes me as a "no-brainer" way of making decisions.

Futzing with pythons terrible multiprocessing is one of least favorite parts of the language.

They’re removing the GIL in some upcoming version. But in general it wasn’t really a problem. Most people don’t actually need multiprocessing and if they did it was handled by the library they’re using anyway.

Which isn’t to say the change isn’t welcome or not needed. Just that the limitation was overblown in most cases.

If you know elixir, but don't know python, it seems like a no-brainer to choose language you know over language you don't. Plus the portability and dependency management of Python has traditionally been a nightmare. Elixir has great solutions to both of those problems for scripting contexts.

Another case, if your code base is in Elixir, picking an unrelated different language (python) would (IMHO) be as silly a decision as picking elixir for scripts when you have a python app, and I would strongly question that person's judgment and fitness for making sustainable and maintainable technical decisions.

two reasons:

1. you prefer functional style languages, such as elixir 2. you want to avoid `pip install` ing dependencies

One time we wanted to benchmark SAAS X simultaneous uploads/downloads (versus our product), and find the bottleneck. This was like four lines of code. Try doing that in python!

Reply on news.ycombinator.com