IPython as a System Shell | Hacker News Reader

IPython as a System Shell

ipython.readthedocs.io

141 pointslaktak61 comments

Couldn't you just use Xonsh?

xonsh is absolutely the right way to go here. There are a lot of 'gotchas' when it comes to building a shell that isn't 'just' a novelty but actually usable.

Shells are most successful when you don't need to think in order to use them. Especially since everyone that's used linux in the past 30(?!) years has the same basic foundation for how to use the command line, the best Shell will feel familiar to sh and bash, but better.

Math, logic, string and value manipulation, all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas.

I understand that many people do know how to program in bash effectively. That doesn't mean it's the future. It's like the Perl vs Python thread from the other day.

Xonsh is intuitive. Xonsh is well designed. Long live Xonsh.

> all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas.

This made me chuckle. A few years ago, I went on a learning spree to finally memorize all of these sparse semantics so I could write bash scripts fluently. I ended up having a really good grasp, could write scripts with no flailing.

Six months later, I was back to searching "bash double quotes vs single". Unless you use them every day, for years, these semantics just fall right out of your head.

Yeah, having played with some scripting-languages-as-shells, the value that a shell adds over REPL is not insignificant... but haphazardly trying to mash together a scripting language and a shell can produce something like Powershell, which is incredibly powerful and expressive but infuriatingly inconsistent and full of gotchas.

> the value that a shell adds over REPL is not insignificant

Could you expand on this? I've been thinking about using a repl as a system shell. The piping issue is solved easily by sh.py. The main problem I can think of is startup latency.

In general it's the poor ergonomics of calling native command-line tools and manipulating their results that tend to disappoint, but I've mostly played with CSX (c# repl script) as a "repl as command-line" dabbling.

In shell, "if" just looks at the exit status of the next set of commands.

if ls /path ; then echo foo ; fi

works just as well as the sugar wrapping "test" ( [ ] ) or extended builtins.

That's fine. But simple things that should work in order to be painless, don't.

-- -- bash:~]$ if $var < 21; then echo "smaller"; fi

bash: 21: No such file or directory

-- -- BUT this works:

if (( $var < 21 )); then echo "smaller"; fi

smaller

-- -- BUT this DOESN'T:

if [[$var < 21 ]]; then echo "smaller"; fi

bash: 21: No such file or directory

-- -- BUT this DOES (added a space after '[['):

if [[ $var < 21 ]]; then echo "smaller"; fi

smaller

Plus the fact that having to type out 'then' and 'fi' and that semi-colons are semi-necessary are annoying. Compared to today's languages, very little of the non-posix parts of bash feel well though out. We can do better.

All these things make it necessary to understand bash. Most modern programming languages feel familiar. Bash and Sh don't anymore. They feel antiquated.

I'm not defending shell, just pointing out how it works.

One of my biggest annoyances with shell is that there are these magic programs such as [ that pretend to be part of the language but aren't.

If if $var has an exit status (because perhaps you insanely want to have var be some program set elsewhere) your if statement would work....

These other examples have operators (test or similar expanded builtins), and those set the exist status that the if block reads.

so grep -q hacker < /etc/passwd if [ $? -eq 0 ] ; then echo hackd! ; fi

is an indirect way of saying:

if grep -q hacker < /etc/passwd ; then echo hackd! ; fi

Shell is terrible, but the way people use it makes it even worse because the sugar hides what's actually going on.

Meanwhile in zsh

    $ if (( $var < 21 )); then echo "smaller"; fi
    smaller
    $ if [[ $var < 21 ]]; then echo "smaller"; fi
    smaller
    $ if [[ $var < 21 ]] { echo "smaller" }
    smaller

I'm curious , if I type

"ls -la"

how does it know if it is "variable ls minus variable la", or "command list with command argument -la"?

It checks if the variables 'ls' and 'la' exist, and if they do, it'll run it as python. If not, it'll try to parse it as bash. If it can't parse it as bash, it'll try again in python mode.

https://xon.sh/tutorial.html#python-mode-vs-subprocess-mode

> It checks if the variables 'ls' and 'la' exist, and if they do, it'll run it as python. If not, it'll try to parse it as bash.

This is horrifying. It can break in so many unexpected and context dependent ways... Do people really run this shell on purpose?

There are edge cases but IME I rarely came across commands that could be valid python and valid bash. And you'll keep those in mind when you're running things so that you won't ever name a variable after a bash command you might use.

I ran it for a couple of years while I was in school, it worked great. I wouldn't run it on a production server, this is really for experimental work

I've been using it for a few years. It has its warts, but the problem you highlighted has never bitten me.

Holy crap that was my exact reaction to reading that. It’s just like bash: 99% of the time it’s an innocent thing, until suddenly you source some script that redefines something, innocently thinking it shouldn’t be a problem. That’s how you get an rm -rf.

Yeah you wouldn't source a script written for a different shell in this language, if that's a part of your workflow you'll want to either audit them first or you'll just want to drop into a bash shell.

I'm not sure how you end up rm -rf ing anything. 'rm -rf' by itself doesn't do anything, and if you do 'rm -rf .' that's no longer valid python. Not to mention that most people writing python would write 'rm - rf' if for some reason they had variables with those names (why would you name it that?) that they needed to subtract. And for some reason, you're not doing anything with the result of that subtraction.

If you're sourcing scripts without reading them, the source of the danger is that behavior, not your shell

> most people writing python would write 'rm - rf' if for some reason they had variables with those names (why would you name it that?) that they needed to subtract.

I can see myself naming variables that way, when doing a geometric computation. Being a mathematician, all my variable names are single letters. But sometimes I have variants of a name that merit a second letter, like a sub-index. Maybe I use "r" for the current radius of an object and "rm" for the "middle" radius and "rf" for the "final" radius after some polishing process. Then I want to see how much material I'm losing from the object in the middle. Thus "rm -rf" (yes, the missing space is a typo).

You need two typos in that you need to forget that space, and you also need to have forgotten to set one of those variables. You also don't actually delete anything until you name a file, so you'll need to add more valid python that's also a valid filename

In marcel this is a command:

   ls -l

And this is arithmetic:

    (ls -l)

In general, parens delimit arbitrary Python expressions.

https://marceltheshell.org

IPython maintainer and Xonsh dev here. Yes, that is the right answer. I'll update de IPython docs.

[EDIT] Pr updating the docs https://github.com/ipython/ipython/pull/12894

I've never heard of this project before, but as someone from Florida, its name is causing me great distress! Conch is pronounced konk, not kontch. So I'm just going to call this project zonk.

(Otherwise, it looks great!)

IPython maintainer and Xonsh dev here, this is an acceptable pronunciation.

The question to me is what gaps exist in using IPython directly versus a more dedicated shell like Xonsh?

I've inadvertently found myself using the IPython interpreter as my shell more and more. Opening vim in IPython always makes me smile. But I don't try to push it's abilities as a shell too much out of a healthy fear that it'll break unexpectedly in the worst way when I absolutely need it not to.

Or take a look at marcel: https://marceltheshell.org

xonsh worked poorly with curses based applications when I tried it a while ago.

Last time I tried, it manually "passed through" nano to avoid broken keyboard handling, but you had to add micro, makepkg, sudoedit... to the list, you had to exclude every "top-level binary" that either supplied or invoked a console-based text editor, from xonsh's specialized handling.

having to whitelist and blacklist things which need to do cooked terminal IO is not a good situation.

I had this problem when I first switched to it, but the solution was in the Github issues. I don't know if it's a problem with the newer releases.

I came here to suggest exactly this.

I've been using amm on and off https://ammonite.io/#Ammonite-Shell

pretty nice if you know scala, still have to use regular shell(s) so I do not forget them

- slow startup

- no pipe/redirection mechanism

- high overhead in version/library control(pyenv, venv)

- no support for one liners because of python syntax - commands too verbose. a line with two lambda expressions would look terrible

(All my personal preference, obviously. But I'd rather have any other language than python, even lua)

Lua? Really? My first go-to complaint about Lua is exactly that same one - poor support for lambda-style programming for expressive one-liners.

When I said I'd even take lua over python, that's only if I was forced to pick a high level language for a shell. Lua is really easy to conjure up higher level algos, like javascript can.

But I'm not convinced any high level language is a viable replacement for bash. It would have to be more functional than bash whilst still having good support for standard unix tools.

One very interesting idea I found was a macro library for racket that turns s-expressions from deeply nested structures into a much flatter representation. Lisp might be a good option with toy with because of flexibility like that.

https://www.reddit.com/r/Racket/comments/mands9/fluent_unix_...

So, you want Scsh, then?

> But I'd rather have any other language than python, even lua

seriously? lua shell won't even evaluate bare expressions

I think you mean luajit shell?

  $ lua5.3
  Lua 5.3.3  Copyright (C) 1994-2016 Lua.org, PUC-Rio
  > 2*3
  6

  $ rlwrap luajit
  LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/
  JIT: ON SSE2 SSE3 SSE4.1 AMD fold cse dce fwd dse narrow loop abc sink fuse
  > 2*3
  stdin:1: unexpected symbol near '2'
  > =2*3
  6

Criminally slow startup (10 seconds on my system). Incredibly ponderous tab complete too. How is this even possible!? Non-starter as a shell for me.

Mostly agree, though if you need piping and redirection, you can use Python for that:

file_list = !ls

delete_me = b[0]

!rm {delete_me}

Though it can't fit into a single statement due to a greedy '!'

Used this a while ago and it's ok-ish, but I've also used Powershell and despite it not being as nice of a language as Python (imo) and also can suck in other ways, what is striking for this particular example is that Powershell doesn't need most of the explanation on this page because the topic is basically 'here is how to do grep/sed on plain text in a list of strings which contains file information and hope you extract the correct field' and Powershell did nail that pretty well by piping objects.

Is there a sh to powershell conversion tool? When I run a local environment in Google Colab, I find myself having to write !commands for MS and Linux filesystems. It'd be convenient to write once.

Not really (for a couple of often-used commands like cd and ls there are aliases in powershell), but I also wouldn't now how that would work. You mean to convert the complete syntax?

Although I like IPython, I wouldn't want my system shell to be vulnerable to sudden destruction upon updating Python packages.

> var = !cmd syntax

Is this a standard anywhere outside IPython? I would love to use this sort of thing (python with seamless integration with a system shell) in place of bash scripts. Yes, you can use subprocess but it's a lot less readable.

You might like Perl. The syntax to invoke and interact with processes is much more terse than Python's. And there's quite a lot of history of using it instead of shell scripts.

Edit: Things like:

  $output=`ls`;

  $output=qx#ls -l#;

  system('/bin/echo','avoid','subshell','invocation');

  open("APIPELINE","ls -l |");
  while (<APIPELINE>) {
     chomp;
     print "Got line: [$_]\n";
  }
  close(APIPELINE);

And access to child exit status in a way that's familiar ($?), etc.

That Perl is quite out of date.

These days, Perl has 3 argument open, and can use a variable for a filehandle. So:

    open(my $pipeline, "-|", "ls -l");

This is a good thing, because you can treat a filehandle as a standard variable, and can separate the special stuff from the command itself. This way if you're taking in user input, they don't get to mess with that. You can also do:

    open(my $pipeline, "-|", "ls", "-l", $directory);

Which lets you get away from the mess of correct quoting and just pass every parameter exactly as intended.

The bang syntax is specific to ipython. I'm not sure it's really desirable outside of that context. It lacks a lot of process control features you'd want in order to make things robust for general purpose scripting purposes.

It depends, for other python REPL/Env not really but it is the standard way to call shell command from ed [1] for example. Julia have also a shell mode in its REPL [2]. Personally, I like to use Raku [3] instead of Python or Shell script. It make it easy to drop one-liner, call shell commands [4] and you can super easily make a script a small CLI utilities by writing a MAIN sub [5].

[1] https://www.gnu.org/software/ed/manual/ed_manual.html#Comman...

[2] https://docs.julialang.org/en/v1/stdlib/REPL/#man-shell-mode

[3] https://raku.org/

[4] https://docs.raku.org/language/quoting#index-entry-quote_qqx...

[5] https://docs.raku.org/language/create-cli

It's also similar to a vim construct where prefixing a command with ! makes it run it as an external program instead of a vim command.

While I haven't personally used it, another commenter mentioned Xonsh[0] and that is what it appears to be: A very smooth integration of python into a shell that still feels like a unix shell.

0. https://xon.sh/

You might like invoke: https://www.pyinvoke.org/

That syntax exists in `git alias` too.

Anyone interested in a Ruby equivalent may like Pry's shell integrations: https://github.com/pry/pry/wiki/Shell-Integration

I tried this a few years ago when I was younger and more optimistic. But man it was not a great experience.

Having said that I think there is still room for modern shells which are easier to script in. For a while I am using fish and thought scripting would be at least as easy and straightforward as using lua. But the syntax is really awkward. For example comparing two variables needs to be put in a

math "2 + 2"

Urg.

If you want a full-fledged programming language in a shell, you might also be interested in Elvish (https://elv.sh). Start with the quick tour: https://elv.sh/learn/tour.html

can't say i don't like the idea, a certain howardism did similar things in emacs org-mode and live document / os code blend feels very right

I can hear the smalltalkers laugh from afar though

I've enjoyed using pyp[1] with fish to make some nice one-liners

For example, custom history command to make early commands at the top, scroll to the bottom, and show the date and command on the same line

function hist; history -t -R | pyp '["\t".join(pair) for pair in zip(lines[::2],lines[1::2])]' | less +G; end; funcsave hist

[1] https://github.com/hauntsaninja/pyp

Babashka has picked up a lot of adherents lately by using a more hybrid approach to this idea. Use clojure to where you want to succinctly and powerfully manipulate data structures (even simple things like (assoc ...) is alien technology in bash) User the shell for everything else.

It's too slow (1 second on an nvme machine). Command argument completion doesn't work either.

TempleOS uses (Holy)C as a system shell (and for everything else). It certainly has its limitations at the command line, but on the flip side it makes writing "shell" scripts a whole lot easier!

Reply on news.ycombinator.com