Shshsh is a bridge connects Python and shell | Hacker News Reader

Shshsh is a bridge connects Python and shell

github.com

68 pointsEntICOnc51 comments

I've always liked Python's "batteries-included" approach, and my take on libraries like this is that they introduce extra dependencies into scripts I write. And often, the juice doesn't feel like it's worth the squeeze.

For me, a little bit of syntax sugar (and maybe a line or two of code savings) isn't worth taking on the dependency burden, especially in production code.

Each and every dependency is one where I invite risk of requirements collision, or that the upstream goes away, or that the maintainer breaks compatibility, etc. It's more time spent installing dependencies, and more time spent processing lockfiles.

Why carry that burden when subprocess.Popen() exists and works fine?

I agree with you for the most part, I really like writing shebangable dependency-free python scripts.

But I 100% understand the motivation for this project. Like you said, subprocess.Popen works well enough, but it's so much less ergonomic than shelling out in Ruby, Perl, or even Awk.

Great points. Agreed.

In certain situations though, the readability/maintainability gains from using this library might outweigh the added complexity of a layer of indirection.

Situations like this: whether to build a utility solution completely [1] in shell, or invert things and use mix of Python and shell. E.g., wrapping 'git' to create a note-taking CLI, or shelling out to $WEIRD_INTERNAL_TOOL and scraping its logs and stdout/stderr.

I think infrastructure like this repo can result in more readable and maintainable code than an all-shell OR all-Python solution in such situations.

You'll be less tempted to start doing things completely in shell and then get locked into that lock minimum.

However, you MUST be prepared to understand the source code of all dependencies you take [except the language's standard library, or similarly mature external libs]. e.g., I pick bottlepy.org vs Flask for personal projects.

With a small/medium library like this, I might import it to get a headstart and will be ok with it ultimately ending up as an incompatible, heavily-modified fork of the upstream project. Which IMO doesn't reflect poorly on the user or the library author.

[1] "completely" means some non-trivial processing that makes you use hash tables, arrays, lightweight tokenizing using read, arithmetic evaluation, sed+awk to massage command outputs so Bash's lightweight data structures can handle them, os.path manipulations.

Also, `>> 'some-cmd --with --args'` is code that's more likely to end up in a deployment script than in a production app. Less likely to be a project that's being edited by many people at once, served to end users, etc.

I respect and often agree with the parent's hesitation to introduce inessential dependencies, but the cost/benefit of making a shell script more comfortable to read/write is very different than that of adding a mystery brick to the foundation of your app.

Deployment scripts are production code, same as whatever you're deploying. If your CI/CD docker images break, you're just as stuck as if your payload breaks.

I totally see the benefit of moving complex shell logic to Python (and still calling some shell helpers) but that doesn't mean you need to have a whole extra dependency for it.

You could write a two-line function that wraps subprocess.Popen.communicate (or subprocess.check_output), and use it instead for 90% of the benefit.

If the argument is "I don't want to learn it", or "too much effort", then I'd ask - is saving 15 minutes one time during development really worth a permanent dependency?

I remember Leftpad, and the lessons that it taught.

I've also had to deal with codebases where the build kept breaking because a large team of developers all Pip-installed whatever they wanted. A package here, two packages there. Overall a maintenance disaster, and you'd be surprised at how often those dependencies outlasted any code that used them.

Unfortunately, the subprocess package is too low level, and usually ends up making scripts excessively verbose.

I wish there was a stdlib package that sat between subprocess and something else with a higher level syntax for process management. Running binary `foo` takes 3 letters in shell, and many many more in subprocess.

Isn't that https://pypi.org/project/sh/ or https://plumbum.readthedocs.io/en/latest/#api-reference

No because they are not stdlib.

because subprocess.Popen() is a giant pain in the ass that's easy to screw up, and I have to read the docs every single time and I'm still not sure it's right

There are many reasons why one might want to call into the shell from a programming language - but ls, grep and cat are not among them. Python has libraries for this already?

I can also almost feel the spirit of Rachel Kroll (rachelbythebay) glaring disapprovingly at this, because the last time someone tried the equivalent of `exec('mkdir -p ' + path)` instead of doing it properly, it led to an SEV: https://rachelbythebay.com/w/2021/12/24/mkdir/

Yes, the page has something like "prepared statements" to avoid obvious injection holes, but it might not be obvious to the everyday user how important that is to get right.

Python definitely doesn't have a grep. The reason most programs have support for calling grep, ripgrep, or ag is because you can't just write or import a searcher that's as fast as these tools. You can find plenty a library that wrap them but none that compete with them with an independent implementation.

It seems better/easier if these tools just exposed an API so they could be used as a library directly without sticking exec wrappers in.

ripgrep does expose a library. It's just vast, complex and exposes an API that is lower level than what the CLI provides. It's a lot easier to just do `rg --json` and be done with it.

ripgrep has on the order of 100 flags. I don't have the bandwidth to maintain both a CLI and a stable library interface for all of that.

So while it might seem better to you, there are realities that make it difficult to do given constrained time and resources.

Tbh I'm surprised no one is already maintaining a decent abstraction (or maybe they're just not discoverable). Afaik openssl is the same way and there's some high-level libraries that are common (although I suppose some of that is built into language stdlibs)

openssl is likely categorically different. When you need SSL, there isn't too much choice. But when you need to search stuff, especially in a specific case, you often don't need the full functionality of something like ripgrep or even its speed. Often, just iterating over lines and doing a regex search for each is Good Enough. Bottom line is that a naive solution can get you a lot of mileage, where as there really is no naive and simplr solution to SSL.

On top of that, ripgrep has a `--json` flag with a pretty simple format. Running another process is often not a Big Deal, and thus, there is another off ramp to avoid the complexity of ripgrep's libraries that doesn't really exist for SSL.

Sure, if we could have every program built as a library that then had a front end wrapped around it that would be great. However, in practice that's not particularly common.

I mostly meant around ag/ripgrep since I thought these optimized search tools were pretty popular & widely used.

.. wat?

Have you not heard of `re` and its functions, search, match, and sub?

That isn’t grep

Grep is a regex capable search. re is more powerful than grep. Python supports "grepping" via re.search.

Absolutely is grep. If you need a shell tool to search for something in a python script, youre doin it entirely wrong

A simple loop over lines using Python's `re` package is unlikely to be as fast as an optimized grep. If your inputs are small, the difference might be negligible. But if they're large, the difference might be enormous.

And that's not even accounting for just regex engine throughput on its own. ripgrep for example uses rust/regex. Compare its performance with python/re: https://github.com/BurntSushi/rebar#summary-of-search-time-b...

Perhaps you want to interact with more complex command line programs, like kubectl to automate some tasks

Kubectl interacts with the kubernetes api and there is a python client for that.

> but ls, grep and cat are not among them.

Really?

Let me see you write this in native python with the same level of conciseness as the shell can pull off.

    from pathlib import Path
    
    for path in Path().glob("*test*"):
        print(path.name)

i think your python version of `ls` is missing a few options

This is a pure-python version of the first example in shshsh's README. The example uses ls without any arguments.

So that's ls Two characters instead of 3 lines

Where's grep ?

And where's the pipe ?

glob plays grep's role here. No need for a pipe.

If you want to run a shell pipeline that is more challenging to port to pure Python, then you could the shell directly:

    S.run("a | b", shell=True)

You could emulate it in Python using the subprocess module but it would be more verbose. It depends on specific use-case whether it is worth the trouble.

Calling the shell pipeline from Python gets you the best of both worlds: a shell one liner is used as a concise domain specific language to run commands , and more complex logic, proper error handling, etc is left to Python.

I have bookmarked/tried so many Python/Shell mashups over the years.

IMHO the following is about the only one that's tasteful in terms of project scope and not going off the deep end: https://github.com/hauntsaninja/pyp

I was really in to this stuff as well - eventually I just settled for awk, it works pretty well

I'd be curious on your thoughts on my entry: https://marceltheshell.org. (This website has lots of examples, so you can get a sense of how it works without installing.)

Personally I wouldn't use a full-on replacement interactive shell because I have a stacked .bashrc that I've constructed over a long time, and things I install from my package manager all come with bash_completion support etc, and even scripts written in Python can vend bash completions (https://pypi.org/project/argcomplete/). Also shellcheck's static analysis of scripts is very educational and that feeds back into interactive use.

I like that you support the scripting use case with vanilla Python syntax. That's something that made Xonsh a non-starter for me (I'm not going to write scripts when syntax highlighting and auto-formatting don't work because it's a superset of valid Python syntax).

Evidently my definition of "going off the deep end" here is giving up lots of working tooling and integrations.

Thanks.

Lately I've been finding it useful to write bash scripts with some marcel, e.g.

    #!/bin/bash
    echo ...
    pushd ...
    marcel <<EOF
    ...
    EOF

Thanks for posting. Interesting tool -- I'll take a look at this.

"Saving streams" is killer. I've been "meaning to build something to solve that" for a looong time. :) :) But you did. Thanks!!

https://www.marceltheshell.org/saving-and-recalling-streams

-----

One feature request to think about if it doesn't exist: defining multiline functions right at the prompt, which you later export to a .py file. So I think a decent inline editor with support for arrow keys might be useful. https://www.marceltheshell.org/functions-1

This is not a showstopper obviously. The current 'import' functionality should be enough for me for now.

Why inline vs. shelling out a dedicated editor: because replacing the screen will take me out of flow because I frequently refer to previous commands' output on the terminal when constructing/editing the next command.

Thank you! One change from the stream page: > and >> now redirect to files in the traditional Unix/Linux way. (That older behavior was just too ingrained and I imagine it is for lots of people.) Use >$ and >>$ to store in variables that don't survive your session. (The $ is meant to evoke an association with variables.)

Multiline functions can be defined right now:

    M 0.18.3 jao@loon ~$ f = (lambda: \
    M +$    lambda x, y: x + y \
    M +$    )
    M 0.18.3 jao@loon ~$ (f(3,4))
    7

So all that would be needed is a way to dump them into a file. That should be doable with another function.

One thing to note: You have choices beyond inline and imports for defining new functions. There is a startup file, typically in ~/.config/marcel/startup.py. You can define functions in there, or do imports. The REPL loop checks for modifications to that file, so if you change it, you pick up the changes for your next command.

For example, I just put this into my startup file:

    def hello(x):
        return f'Hello {x}'

and then without restarting marcel:

    M 0.18.3 jao@loon ~$ (hello('world'))
    Hello world

I have never heard of pyp before, looks neat!

I made something similar myself, pyxargs [1], which doesn't have all the magic of pyp, but although being geared towards more xargs like usage, it also functions somewhat similarly.

For example, although not as clean as the magic variables x, l or line, you can use '{}'

  ls | pyp 'x[:3]'  ==  ls | pyxargs --pyev "'{}'[:3]"

Overall the syntax of pyp looks much better, and the --explain feature is nice since pyp seems to do a little more for you under the hood (the closest pyxargs equivalent being --dry-run). I think I'll start using pyp myself when not running commands over multiple directories of files or in parallel.

[1] https://github.com/elesiuta/pyxargs

I like your take on it compared to pyp at first glance because it uses a clearer API (xargs instead of just wildcard pipe) and a more appropriate name (pypyp is nonsense) even if it has fewer features.

But yeah I didn't know either existed and I wish I did, seems like it could be really useful in the shell

Thank you! My original use case was for running ffmpeg in directories with encodings that broke xargs, and later added the py/pyev flags which basically swapped subprocess.run() with exec()/print(eval()).

I didn't really highlight it in the documentation but it can also run commands in parallel as multiple windows in a multiplexer so it's easier to checkup on their progress or interact with them.

What are the similarities/differences to plumbum? https://plumbum.readthedocs.io/en/latest/