Xonsh, a Python-ish, Bash-compatible shell language and command prompt | Hacker News Reader

Xonsh, a Python-ish, Bash-compatible shell language and command prompt

xonsh.org

242 pointsngoldbaum61 comments

I'm a lead dev on the fish shell (http://fishshell.com)

xonsh looks very cool. Shell scripting languages are famously obtuse and underpowered, and we've thought about how to cleanly merge shell scripting with a real language. It's heartening to see an attempt at it.

Ambiguous syntax is a sticking point. For example, say the user runs this:

    [ 1 ]

is this constructing a Python list with the value 1, or is it invoking /bin/[ with two arguments? It's ambiguous! It looks like xonsh interprets it the first way. Any thoughts on this issue?

A related issue is the relationship between bash functions and Python functions. I'd be very interested to see how xonsh approaches this.

Also, here's some unsolicited advise, from one non-POSIX shell maintainer to another: The claim that xonsh is "bash-compatible" is delicate. Full bash compatibility requires your shell to be insane. E.g.

   echo foo | cd /
   cd / | echo foo

In bash, only the second line will change your working directory. Or consider this command:

this finds the first (for some meaning of first) file in your directory and invokes it with all other files as arguments. I tried it: xonsh doesn't handle either of these "correctly," which is to say that xonsh isn't yet insane. That's good news! On the other hand, this means it's not really bash compatible.

IME nobody really cares about full bash compatibility. What they want is for their existing bash stuff to just work. Bash shebang scripts will just run in real bash, but that leaves shell-integration scripts: virtualenv, rbenv, etc. Supporting that stuff is one of the most common requests that the fish shell gets, and one of the most commonly cited reasons for not using fish.

Here's a piece of rbenv's bash integration: https://github.com/sstephenson/rbenv/blob/master/libexec/rbe... . It's short but there's a lot of features being used: set, shopt, IFS, functions, etc. If you're aiming for bash compatibility, that's your target!

In the meantime, I suggest changing the language to express that xonsh is bash-like, not bash-compatible. Otherwise users are going to be confused and disappointed.

Thanks for posting this @ridiculous_fish! I am the main author of xonsh, and I really appreciate fish. Its awesome and I am certainly going to borrow ideas :). I agree that making shell languages into 'real' languages is strong goal that we should all have. And like with real languages, many can co-exist happily.

I'd like to point out though, that the goal is not to have bash-compatibility. As you point out, bash does some insane things and that is exactly what I want to avoid with xonsh. I should be able to give xonsh to a programming newbie and they should not experience any gotchas once they know the language.

I have been pretty careful to say BASHwards-compatibility in the docs and elsewhere, for exactly the reasons you mention. The goal isn't to be bash with some python-isms, but to be python with the useful parts of the shell syntax. Like any good technology, it should interface as well as it can (ie when not insane to do so) with the previous technologies. For example, read in the bashrc or take advantage of the bash completion functionality. That is what BASHwards was meant to mean. @ngoldbaum seems to have submitted the link here, under 'bash-compatibility.' Oh well. This probably is proof enough that I should change the 'BASHwards' language elsewhere. Thanks for the tip.

Keep doing great things with fish!

FYI, I interpreted "BASH-wards compatible" as "BASH-compatible" as well. The subtlety seems a little lost on me.

"BASHwards" is a bad word to use for this, since the natural interpretation is that it is a combination of "backwards" and "BASH" meaning thatn xonsh is an "upgrade" that is backwards-compatible with BASH.

It seems to be more intended to be not backward-compatible, but to have some attention to minimizing the pain of transition, which is valuable, but needs different language to communicate.

       echo foo | cd /
       cd / | echo foo
    
    In bash, only the second line will change your working
    directory.

Um... in Bash, neither of those change your working directory (unless `shopt -s lastpipe` is set, in which case: the first does).

Whoops, you're right! Egg on my face.

you two should get a room, and fail to emerge until you are ready to bless the world with one shell to rule them all (and in the darkness bind them)

> that leaves shell-integration scripts: virtualenv, rbenv, etc. Supporting that stuff is one of the most common requests that the fish shell gets, and one of the most commonly cited reasons for not using fish.

shameless plug, but I made a utility to solve the problem: https://github.com/edc/bass

So you can write `bass source /usr/local/bin/virtualenvwrapper.sh ';' mkvirtualenv env1`, which of course can be shortened with a function.

I'm testing this out later this week... if it works... you're my new hero!

I'm pretty sure it said "BASHwards-compatible" when I submitted the story to HN, so I guess one of the admins changed the title.

> Shell scripting languages are famously obtuse and underpowered, and we've thought about how to cleanly merge shell scripting with a real language.

It was called REPL (Interlisp-D, Lisp Machine, Cedar, Oberon), Workspace (Smalltalk), but then UNIX won.

> Shell scripting languages are famously obtuse and underpowered, and we've thought about how to cleanly merge shell scripting with a real language. It's heartening to see an attempt at it.

I used tclsh for a while. IMO a shell language needs the "function arg1 arg2" function-application syntax to fit with the way programs are called. Fortunately that style seems to be getting more popular - Haskell does it and to a certain extent so does Scala.

This is great. When dealing with bash stupid-isms I've always thought it would be awesome to have a shell with Python syntax. The tough part is figuring out how to combine Python with "normal" shell commands in a clean, intuitive manner but it looks like Xonsh has gone a long way towards solving that!

Just installed on my personal Macbook. Will see how it goes.

I agree, this is awesome. I hate dealing with bash but it's pretty common when you're DevOps.

You could already do some minimal shell-like scripting with ipython[1]. Just put your ipython-flavored script in a file with .ipy extension an call it with ipython, or just use #!/usr/bin/ipython as a shebang.

  #!/usr/bin/ipython
  files = !ls
  for f in files:
    print(f)

That being said, it is a bit clunky to use since it's not a proper shell, so xonsh is definitely a very welcome improvement.

[1]: http://ipython.org/ipython-doc/stable/interactive/shell.html

I genuinely wish there was an alternative that could gain enough traction to reach critical mass and hopefully start shipping with operating systems. I've never bothered to learn bash because it was always so damned confusing, and I just couldn't get over the idiosyncrasies when playing around with it. I wouldn't mind putting in time to learn something new and actually user-friendly.

Have you looked at fish? It is in the package repositories everywhere, I think.

Even so, the single most attractive trait of bash is that it ships with almost everything. If I have to write a script on a machine with no internet, then bash is the only option. This its strongest source of staying power.

You would likely have a regular sh, be it bash in "sh-mode" or a real sh, plus likely csh and maybe ksh. You'd probably also have awk, and maybe even perl.

perl is everywhere, but like python, it's power is in it's modules.

and those are not everywhere.

I actually gave fish a try after you recommended it. I can't use it in my workflow :(. Virtualenv has a bash script to initiate the environment and fish's incompatibility with bash scripts makes it impossible for me to work with it.

virtualenv deployments come with a fish specific script, it is at `venv/bin/activate.fish`.

You can source it with `. venv/bin/activate.fish`

Awesome! thanks

Well, if you do ever want to learn the "finer points" of bash, http://wiki.bash-hackers.org/doku.php and http://mywiki.wooledge.org/BashGuide have tons and tons of great info.

Having played with it for about an hour, I think Xonsh might actually be that alternative.

This is quite apropos, as just last week I released a little language called 'bish' designed to overcome exactly this issue. It compiles to bash for portability, but has more modern and familiar syntax.

See https://github.com/tdenniston/bish for the repo and https://news.ycombinator.com/item?id=9166386 for some previous discussion.

The language is still brand new and under active development. It's not stable yet.

.. you say a shell command line should be user friendly? What kind of user would use command line? We are all bofhs here.

Also some of us are being paid to write bash scripts. I'm for example forced into bash because of AIX server. Learing bash is fun and profitable...

> I'm for example forced into bash because of AIX server.

What? Consider yourself lucky. All of my AIX was ksh. If I wanted to be especially rebellious, I would use ksh93.

the demo gif could use some delays before each clear...

Yup, transitions way too fast to know whats going on.

I was able to read the gif as the information was displayed.

Read it, yes. Appreciate it, no.

I'd readily accept any new demo videos :) Just use a white background so it matches the website theme

Is there a reason xonsh doesn't honor the PS1 variable from bash to keep your terminal prefix? Or is there a way to customize xonsh's?

Example:

  10:54:01 foo@bar:~/scripts/xonsh$ xonsh
  foo@bar ~/scripts/xonsh $ # no time prefix

If you go to [http://xonsh.org/tutorial.html#environment-variables] and scroll down to the table, you can see how to change the prompt :)

Thanks, doesn't look like you can use anything that isn't in the supported keywords though (like date/time)

It looks to me like you can.

> PROMPT: The prompt text, may be str or function which returns a str.

If you need non-built-in values in your prompt, just set PROMPT to a function, and call something like `date` within the function. Use `%` to format `date`’s output into a template string, and then pass that template string to `xonsh.environ.format_prompt` to get the final value to return from the function.

Pretty cool, but I feel unsure what output something will produce. Look:

    a = $(ls)
    a  # '1\n2\n3\n4\n'
    b = `ls`
    b  # []  <--  What?!

Because of these little surprises I cannot really feel confident using it.

Is there a way to switch explicitly between Python and Bash mode? The "ls -l" example makes me pretty worried that this could mess up basic bash scripts that already work just by defining a variable like "ls" and forgetting about it, or forgetting where it's defined. It's a great idea, but there's a lot of room for stuff breaking that worries me.

You can use an uncaptured subprocess to run a command explicitly in bash mode. I'm unsure how you think xonsh would break a bash script, though. When you run a script in xonsh it's still going to be interpreted by whatever is declared in the script's shebang. Just like you can run a perl script from bash, you can run a bash script from xonsh. E.G.

  $ ls='foo'
  $ cat test.sh 
  #!/bin/bash
  ls ~
  $ print(ls)
  foo
  $ ./test.sh 
  bin    C:\nppdf32Log\debuglog.txt  foo	       initrd.img.old  lost+found  opt	 run   sys  var
  boot   dev			   home        lib	       media	   proc  sbin  tmp  vmlinuz
  cdrom  etc			   initrd.img  lib64	       mnt	   root  srv   usr  vmlinuz.old

The stuff that gets broken are bash scripts that are designed to be sourced directly: virtualenv, rbenv, various prompts, etc.

Presumably the shebang at the start of your script would cause it to be executed by bash directly.

I don't think you would replace /bin/bash with this, but rather switch to it as your interactive shell, using "chsh". On FreeBSD, for example, the default shell is /bin/tcsh but scripts are usually written for /bin/sh (with a shebang line).

Even if this thing can execute unmodified bash scripts (can it?), it seems to me that would be missing the point, which is to avoid the horror of writing complex bash scripts. When I start writing something in bash, because of those few things it makes so easy (e.g. running subprocesses), I usually regret it as the script grows. If xonsh can make those things just as easy, every script can be written in the right language – which is of course Python :) – from the start.

So, for anyone like me who wanted to install it with pip, but has Python 2.7 as their default (and doesn't want to change for fear of breaking things), do the following (on a Debian based system):

curl -O https://raw.githubusercontent.com/pypa/pip/master/contrib/ge...

sudo python3.4 get-pip.py

sudo mv /usr/local/bin/pip /usr/local/bin/pip3 # optional, adjust line below if ommited

sudo pip3 install xonsh

------

then:

chsh -s /usr/local/bin/xonsh <yourusername>

EDIT: Just realized I forgot a step. You have to edit /etc/shells to add /usr/local/bin/xonsh to change your shell to it using chsh.

For Debian/jessie it worked nicely for me to just install python3-ply and python3-pip with apt-get and then just do "pip3 install xonsh". Python3 seems to coexist just fine beside the default 2.7.

Somehow, python3-pip escaped me. I feel silly.

On OSX I just installed it in a virtualenv with python3:

  brew install python3 # will install python3 but not make it default
  mkdir xonsh && cd xonsh
  virtualenv -p /usr/local/bin/python3 venv
  source venv/bin/activate
  pip install xonsh
  xonsh

I see a lot of discussion on the awkwardness and idiosyncrasies of shells like bash.... So this is a great time for a shameless plug! Last week I released a language called 'bish' (https://github.com/tdenniston/bish) which compiles to bash, but has none of the same syntactical weirdness. So the idea is: write and maintain your script in bish, but compile to bash and ship the bash version.

Here (https://news.ycombinator.com/item?id=9166386) is the previous discussion from last week.

Wow.. this only needs python. Which means.. it works on windows?

Sadly no support for python 2.7 so I'm out of possibility to even test this.

Forgive me for my lack of knowledge, but where would one use this vs. just using Python?

You can use it instead of Bash. It basically combines the two... it looks glorious.

I think another question is when would one use Python over Bash and vice versa. Personally, I write bash (and shell, reminder bash script not always == shell script) if I need handy command lines. I could have run in subprocess, but would be simpler to just invoke commands directly in my bash script.

Is && and || supposed to supported? When I try "command && someothercommand", which is a common bash idiom, xonsh seems to just freeze up until a ctrl+c. It also jumps up to around 100% CPU.

Additionally, redirection of stderr with 2> doesn't seem to work. Can you describe (or point out in the docs) how to accomplish this? Thank you.

Is there oh-my-xonsh ready or should we wait for it?

hehe, you mean zsh ?

I mean xonsh. Oh'my exists for zsh, fish, git...

Kinda like it. Not too big a step up from Bash.

Interesting thread. For some alternative ways of doing "pipe-like" stuff in Python (with more than one meaning of the term), see these links:

Swapping pipe components at runtime with pipe_controller:

http://jugad2.blogspot.in/2012/10/swapping-pipe-components-a...

That above post was the last in a series that I wrote over a period. The series describes both a) a small experimental tool I wrote, with examples of a few ways of using it, and b) some other Python-shell tool alternatives that I came across. pipe_controller was inspired by reading about and trying out some of those other tools.

The post above links recursively to all the other posts in the series, but for convenience, I'm pasting selected other links below:

Using PipeController to run a pipe incrementally:

http://jugad2.blogspot.in/2012/09/using-pipecontroller-to-ru...

Some ways of doing UNIX-style pipes in Python:

http://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-st...

Plumbum, UNIX shell-like library and tool in Python:

http://jugad2.blogspot.in/2012/08/plumbum-unix-shell-like-li...

Is there a ruby-ish shell out there?

There was one called rush, by Adam Wiggins, co-founder of Heroku, IIRC. I had tried it out a bit after it first came out. Just now did a Google search for "ruby shell rush" - without the quotes, and it found it - first result, though the others about it look interesting too. Not sure if the project is active though.

Reply on news.ycombinator.com