Single-file scripts that download their dependencies | Hacker News Reader

Single-file scripts that download their dependencies

dbohdan.com

66 pointsnetworked55 comments

It's nice to see Deno recognized here. The ability of scripts written for Deno to import their dependencies by full URL instead of being dependent on a separate package.json + package manager install step being done first helps a surprising amount with certain tasks. It becomes possible to post a complete working block of code that uses arbitrary libraries into a single immediately executable code block online for example. Compare to Node or Python where any code block casually posted online to be really complete needs to be paired with an explanation to first npm install some specific packages (with specific version numbers, if the code snippet is meant to be useful in the future). There's a difference between posting code that's usable on an end user's machine with a few manual often-implicit steps done separately vs one that's immediately executable with exactly the intended environment as-is.

I am not a fan of having a script pull down all manner of things in order to run.

The smart is of course to review the script and see what its dependencies are.

This is not always straight forward, esp. when it gets recursive.

In that NPM is the worst offender I have tried.

It can easily expand from 10 libraries to hundreds of dependencies (and dependencies of the dependencies)

Only hundreds? You must be pretty careful to choose libraries with few dependencies. (I should be more careful...)

It's a topic close to my heart since that is one of the key thing that we solve in the OSS project windmill [1], letting users define simple single-file scripts in python/deno/go and handle the dependencies for them.

For deno, as the article points out, it's easy, deno has this feature baked in and we just have to deno run the scripts, similar for go where the imports are pretty much a fully qualified pointer to their source and version.

For python, we have to do a bit of machinery: we parse the AST, look at the imports, use the heuristics that most import names correspond to their pypi package (we maintain also a list of exception-mappings), pip-compile [2] all the imports, and then get a requirement file that we attach to the script, pip-install it before running the script (then we do a LOT of magic to cache the dependencies so one doesn't actually have to install them 99.9% of the time).

Note: if you're interested in a collection of single-file scripts to many APIs, that you can just copy-paste and call the main function of, that's what windmill's hub is for [3]

[1]: https://github.com/windmill-labs/windmill [2]: https://github.com/jazzband/pip-tools [3]: https://hub.windmill.dev/

Python actually has support for this with `pip-run`. There's a PR to get similar behaviour into `pipx`, which is effectively the `npx` of Python world. I think the single-script-with-dependencies workflow has been somewhat ignored until now; it would be nice to see that become easier.

If I'm not mistaken, looking at the docs of: https://pypi.org/project/pip-run/ it doesn't parse the AST to infer the dependencies, you still have to declare them (either in script with # Requirements comment (which we also use as an escape hatch https://docs.windmill.dev/docs/reference#dependency-manageme...)) or externally declared.

So it's close but not exactly the same.

The nix part seems quite useful. It basically allows one to run a script in any language, with it being self contained and without polluting the global path with all kinds or runtimes. With the only user installed dependency being nix.

Although I'm wondering about how fast it starts on the 2nd run after the dependencies have been download and how it deals with language specify dependencies, e.g. npm/pip packages. Read somewhere thst it can be a pain deal with those in nix.

Nix is really cool, and gets even cooler with NixOS.

To answer your questions: On the second run all packages are saved in the local store, so it's super fast. Language specific dependencies are treated like all other Nix packages, so Python dependencies can depend on Node, etc.

It actually creates a directory with hashes of the packages, so if you run another install that requires a package within that was previously downloaded, it just skips the download too.

Evaluation can be slow, which is why flakes-based tools usually cache the evaluation results. (There aren’t going to be duplicate downloads in any case.)

There’s no shebang support in nix3 (the new, flakes-native CLI with a single “nix” command). The closest you can get is probably “nix run”—can’t be distilled down to a single file, but can fetch and cache even online Git repos (use “nix registry pin” to freeze the commit hash).

What would be an example filetree structure for the nix run to match #!-interpreter via flake? Cannot all the definitions fit into a single "flake.nix"? Also, would it be possible to use some other name than "flake.nix"?

As far as I know, flake.nix is hardcoded, so you’re going to use a separate directory for every script anyway. And then there’s flake.lock, which you have to go out of your way to avoid creating, and you’ll be spammed with obnoxious warnings if you still do.

So would you want to fit everything into just flake.nix? I don’t see why. Could you? Yeah, sure: http://ix.io/4ll5 is an example for a single shell script.

> There’s no shebang support in nix3.

This is a pain point, but there appears to be discussions and a PR for Shebang support with flakes: https://github.com/NixOS/nix/pull/5189

Python is pretty easy. You just add something like '-i python3 -p python3Packages.click' or whatever to the shebang and it normally works and you can keep adding '-p ...' flags (assuming the package in question is upstream.) I don't know enough about the state of node to comment.

AppFS [0] solves a similar problem but with much lower learning curve and overhead for using it, since it removes the install step altogether.

[0] https://appfs.net/

1. Not sure how AppFS could have lower overhead than Nix in terms of removing the install step.

2. AppFS appears to be written by a user named rkeene, and I am replying to a comment by rkeene2. I think a disclosure would be helpful when comparing AppFS with other tools.

Nix has an install step to use a package, AppFS does not -- not having a step is easier than having one. If you want to use a package, you just use it.

Additionally, I'm the author of AppFS.net just redirects you to AppFS.rkeene.org -- I'm not sure what kind of disclosure is needed when discussing it in this context though.

> Nix has an install step to use a package

Not true. See "Using Nix to run software with no installation steps" https://determinate.systems/posts/nix-run for example.

This feature has been one of the main use cases of Nix.

> Nix has an install step ... AppFS does not... I'm the author of AppFS.net...

I appreciate the disclosure, as it helps readers (like me) to judge the merits of your claims, especially regarding AppFS vs other options, and whether those claims could be biased (due to differences in familiarity, for example).

> I'm not sure what kind of disclosure is needed...

Over Hacker News and other forums, when the author of a tool wants to promote their work (nothing wrong with this, I learnt a lot with such promotions) in the context of competing tools, the following disclosures are common:

1. Disclosure: I wrote AppFS.

2. Author of AppFS here. (Just like what you did.)

3. Shameless plug. (Note: I personally do not see this as a shame, but I just want to give examples of common disclosure styles in such contexsts.)

From the linked docs, it's not obvious to me how I could use this tool in the way TFA uses nix-shell to resolve script dependencies.

You just use the dependencies, for example:

  $ cat /tmp/hello.py 
  #! /opt/appfs/rkeene.org/python3/platform/latest/bin/python3
  print("Hello, world!".rjust(20, "-"))

  $ /tmp/hello.py     
  -------Hello, world!

The really nice thing about the nix-shell example is I can specify external python dependencies within the script itself, and even the python version itself, all within one file, and nix-shell just takes care of it.

If you change `latest` to the version [0], then you'll get that exact version.

Additionally any external dependencies can be specified by just accessing them the same way you would in `nix-shell`, just using the path instead of installing. For example, if you wanted to use the "awscli" (the only Python package I already created within AppFS) you would do:

  $ cat /tmp/aws.py 
  #! /opt/appfs/rkeene.org/python3/platform/latest/bin/python3
  import sys
  sys.path.append('/opt/appfs/rkeene.org/aws-cli/platform/latest/lib/python3.6/site-packages')
  import awscli
  
  print(awscli)

  $ /tmp/aws.py     
  <module 'awscli' from '/opt/appfs/rkeene.org/aws-cli/platform/latest/lib/python3.6/site-packages/awscli/__init__.py'>

[0] https://browser.appfs.net/rkeene.org/python3/linux-x86_64/3....

You can do something similar in F#. This creates a cross-platform desktop GUI.

    #!/usr/bin/env -S dotnet fsi

    #r "nuget: Avalonia, 11.0.0-preview4"
    #r "nuget: Avalonia.Desktop, 11.0.0-preview4"
    #r "nuget: Avalonia.Themes.Fluent, 11.0.0-preview4"
    #r "nuget: JaggerJo.Avalonia.FuncUI, 0.6.0-preview4"

    open Avalonia
    open Avalonia.Controls
    open Avalonia.Controls.ApplicationLifetimes
    open Avalonia.FuncUI
    open Avalonia.FuncUI.DSL
    open Avalonia.FuncUI.Hosts
    open Avalonia.Layout
    open Avalonia.Themes.Fluent
    open System

    module Main =
        let view =
            Component(fun ctx ->
                DockPanel.create
                    [ DockPanel.children
                          [ TextBlock.create
                                [ TextBlock.fontSize 48.0
                                  TextBlock.verticalAlignment VerticalAlignment.Center
                                  TextBlock.horizontalAlignment HorizontalAlignment.Center
                                  TextBlock.text "Hello world!" ] ] ])

    type MainWindow() =
        inherit HostWindow()

        do
            base.Title <- "Hello World!"
            base.Height <- 400.0
            base.Width <- 400.0
            base.Content <- Main.view

    type App() =
        inherit Application()

        override this.Initialize() =
            this.Styles.Add(FluentTheme(baseUri = null, Mode = FluentThemeMode.Dark))

        override this.OnFrameworkInitializationCompleted() =
            match this.ApplicationLifetime with
            | :? IClassicDesktopStyleApplicationLifetime as desktopLifetime ->
                let mainWindow = MainWindow()
                desktopLifetime.MainWindow <- mainWindow
            | _ -> ()

    AppBuilder
        .Configure<App>()
        .UsePlatformDetect()
        .UseSkia()
        .StartWithClassicDesktopLifetime(Environment.GetCommandLineArgs())

Suave is really cool for this too.

Here is a hello world server that fetches all of its own dependencies - just 3 lines!

    #r "nuget: Suave, 2.6.2"

    open Suave

    startWebServer defaultConfig (Successful.OK "Hello World!")

Ruby has this too:

  require 'bundler/inline'

  gemfile do
    gem 'httparty'
  end

  p HTTParty.get("https://acme.com/")

You should "drop him a line" (if you haven't already)

Thanks for the suggestion. I replied about Ruby here: https://news.ycombinator.com/item?id=34389323.

I think they missed the best one: https://github.com/teaxyz/cli

It doesn't even need tea itself to run, it'll make a bash script that'll get it too if absent.

Tea seems interesting, but the antagonistic tone in the README is very off-putting. Just tell me what it does without ranting about every other tool I currently use, and presenting yourself as the next best thing. Chances are you actually aren't.

Style points for using leftpad as the example.

Babashka (https://babashka.org/) example:

    (ns malli
      (:require [babashka.deps :as deps]))

    ;; This adds malli (https://github.com/metosin/malli) to babashka's classpath:
    (deps/add-deps '{:deps {metosin/malli {:mvn/version "0.10.0"}}})

    ;; Now we're able to load malli:
    (require '[malli.core :as malli])

    (prn (malli/validate [:+ :int] [1 2 3])) ;; true

It seems the complicated part of using a computer is not adding numbers together but organisation and preparing an environment for computation.

To run a computation I need code and data to be in the right place. If you analysed the number of add instructions of an average compiled program, I feel we would be surprised how many instructions are actual computations and not copying or moving. I'm excluding address calculations and referring to "add" or "subtract".

Preparing an environment for computation is copying data to the machine and preparing dependencies such as docker, package managers, compilation and other work. It's also moving data between registers in preparation for a calculation. It's similar to a factory processing raw ingredients that need transportation.

How do you represent all this movement and logistics efficiently and untediously? I feel computer science could probably expend some effort on representing logistics efficiently.

I think the entire field of computer programming might be described as various methods of organising and representing logical structures efficiently.

You're right.

Within a codebase there's a arbitrary number mental models that that particular code base uses to solve problems. Perhaps it uses a calling convention, or a REST API or a C FFI API. Or Spring. The problem is ultimately how do I get data from here to there so that I understand it for doing an organised computation on it.

When you go outside that codebase, you switch mental models of how to solve and think of problems. It's similar to switching between writing in Python and then writing commands in bash shell.

It's similar to switching between HTML, CSS and Javascript as a frontend developer. Or embedded C programming or the code of a compiler.

I feel the logistical problem is space or position based, which can also be computed with numbers or vectors as with language models.

How many of the problems that people solve with programs are the same problem, said slightly differently? BSD sockets standardised communication but we have so many solutions to positional+placement logistics: systemd, sysv, Tomcat/application servers, kubernetes, package managers, docker, wasm runtimes, virtual machines, firecracker, OSGI, LXC, processes, threads, databases and so on. Said another way: what is where.

https://en.wikipedia.org/wiki/KeyKOS was an OS that attempted to solve the problem of bringing code and data together when both parties don't wish to leak any information to the other. (example: the code is a third-party key management app, and the data are an org's private keys)

Unfortunately, having been in development from 1977-1988, it was an example of Solving a Problem Too Early.

Nobody ever mentions Ruby in these lists. :sad-panda: Check out inline Bundler:

https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...

Combine this with a shebang and you can get a heck of a lot of power into one file.

Does this install dependencies on a per-script basis? Can different scripts' dependencies conflict with each other? I think I saw this when I initially put together the list, but I didn't include Ruby because I wasn't sure how it worked.

I'll add Ruby if conflicts are normally impossible. (If conflicts are possible, the method is equivalent to shelling out to the language's package manager at the top of the script before the imports, which you can do in most scripting languages.)

I believe they get installed to a particular location that can be of course set in bundler’s config. I believe this is also per script, so they one script won’t interfere with another. I’ve used it before, and never had a problem

Thanks. I'll do a little research to confirm it and add Ruby if this is the case.

Conflicts aren’t an issue. Multiple versions of a library can coexist and bundler will install/activate the requested version on a per script or project basis.

The python example is missing the most important part, which is how to make your script work with external dependencies. Here is a working example.py with external dependencies:

    #! /usr/bin/env nix-shell
    #! nix-shell -i python3 -p python3 -p python3Packages.requests -p python3Packages.fire

    import requests
    import fire

    class CLI:
        
        def get(self, url):
            return requests.get(url).status_code

    fire.Fire(CLI)

Run with chmod +x example.py ./example.py get https://example.com

The author forgot JBang (Java, Groovy, Kotlin).

https://www.jbang.dev/

Static binaries.

Cosmopolitan libc for poly-OS static binaries.

In the Haskell ecosystem this is also achievable if you have the stack tool installed.

https://www.fpcomplete.com/haskell/tutorial/stack-script/

You can add Rust to the list https://crates.io/crates/rust-script

Seems like you could write a Python script that branches two ways, either makes a venv or runs.

PyPI packages don’t always have the same name as what you import. PIL comes to mind, as well as others I don’t remember right now.

pyyaml is a good example as well

Indeed: https://github.com/windmill-labs/windmill/blob/3f318b4ac4170... :)

Is that really a problem? Can't you just stuff a list of deps in your script?

This doesn't seem too different from the old 'setup.py' approach of Python packaging except that the 'one script' has both a 'setup.py' in it and the real script.

Sure. I skimmed the article and only really looked at the code for the deno one so I missed that they pretty much all had some special way to declare the deps that needed to be fetched.

These one file scripts are nice but it’s just as convenient to if the script is a package.

One of the reasons I like Go.

pipx might get this ability: https://github.com/pypa/pipx/pull/916

Interesting to see D in the list!

Reply on news.ycombinator.com