Things you didn't know about GNU readline (2019)
twobithistory.org
twobithistory.org
Package: rlwrap
[...]
Homepage: https://github.com/hanslub42/rlwrap
Description-en: readline feature command line wrapper
This package provides a small utility that uses the GNU
readline library to allow the editing of keyboard input
for any other command. Input history is remembered
across invocations, separately for each command; history
completion and search work as in bash and completion
word lists can be specified on the command line. #!/bin/sh
# Simple chat system frontend, wrapping gyap in rlwrap and maybe ssh.
: ${GYAP_LOGFILE=$HOME/gyap.log} ${GYAP_COMMAND=gyap}
if [ "$#" -eq 0 ]; then
rlwrap -C gyap -l "$GYAP_LOGFILE" "$GYAP_COMMAND" "$@"
else
host=$1; shift
rlwrap -C gyap -l "$GYAP_LOGFILE" ssh "$host" "$GYAP_COMMAND" "$@"
fi
#!/bin/sh
: Simple chat system. cf. rgyap. ${nick=${1-$USER}} ${chan=${2-/var/tmp/chat}}
echo "Chatting on $chan as <$nick> (override with $0 \$nick [\$chan])"
touch "$chan" && chmod 666 "$chan" 2>/dev/null
sign() {
echo "$(date +%T) * $nick signed $1 ($(date +%05Y-%m-%d))" >> "$chan"
}
echo "Press ^R if a chat line appears as you are typing. Recently:"
tail -Fn 16 "$chan" & pid=$?; trap "sign off; sleep .1; kill $pid" 0
sign on; while read t; do echo "$(date +%T) <$nick> $t"; done >> "$chan"
it's not secure of course; anyone can spoof anyone else or fill up your /var/tmp disk, or read the log without signing onto the channel
it would be nice to have rlwrap options to customize tab-completion, for example for expanding to recently-mentioned nicks or slapping someone around a bit with a large trout
For years, I had problems pasting many lines of SQL in a psql session. It would slow down as it worked through the characters. One day the same happened in another shell, and it dawned on my that this was probably a problem with readline.
So I fired up a profiler and sure enough, something inside readline was recalculating something upon each character input with the work done proportional to all the characters already input.
I sent my findings to the mailing list, and Chet Ramey fixed it.
In the process, I discovered that wonky editing of long lines when resizing a terminal window was also fixable. When you resize, you change the line width, so readline needs to take that into account when it wraps the long lines. I think it did have code to do that at the time, but it just didn't kick in for some reason. Can't remember the details, but it does seem to work fine in psql today. That was another multi-year annoyance.
To remind yourself how much we take for granted, play with `psql --no-readline` some time and see how awful it is to lose the ability to up-arrow get the last query back, edit it, and send it again.
GPLv3 means that you can't use it with the code you write for your job. And there are cases where readline can sneak into the codebase through unintended routes.
(emphasis mine)
I know you get the distinction, but I hear too often that "we can't use GPLv3 software at work" from people who aren't clear on it.
It is. E.g. if you have a contractor working with your company, then giving software can be considered "distribution". And the contractor can gain the right to re-distribute your internal software.
It's OK to use GPL for software that runs in separate processes (e.g. GDB), but readline is really sneaky because it's a library.
I've never been tempted to think I had any right whatsoever to my employers' internal materials. That would be a highly unusual, maybe unprecedentedly so, situation.
Technically, a litigious-happy contractor can use that to cause a lot of damage to the company. So that's why many companies (e.g. Apple) just ban as much GPLv3 software as they can.
> To “convey” a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
in their faq about gplv2, the fsf explicitly takes the side of your hypothetical 'litigious-happy contractor' https://www.gnu.org/licenses/gpl-faq.en.html#InternalDistrib...
> Is making and using multiple copies within one organization or company “distribution”? (#InternalDistribution)
> No, in that case the organization is just making the copies for itself. As a consequence, a company or other organization can develop a modified version and install that version through its own facilities, without giving the staff permission to release that modified version to outsiders.
> However, when the organization transfers copies to other organizations or individuals, that is distribution. In particular, providing copies to contractors for use off-site is distribution.
note that this is talking about gplv2 ('distribution'), so presumably contractors can already sue apple for this if apple uses gplv2 code in internal tools; they might not win, but the fsf thinks they should
My current bane is BlueZ, which used to allow for a compile-time removal of readline but no more.
He was basically saying that seeing readline pulled as one of the dependencies gives him confidence that the tool won't suck. He said nothing about building tools with readline support (implicit or otherwise).
... you did write and release a replacement in the license you like, right? right?
What's this friction between readline and other licenses in other Free Software code? Unless I'm writing some software with a non-GPL license and want to include GPLed software in my own that I plan to distribute, it doesn't affect me.
On the bright side, that experience did cause me to do a little research and learn about readline, which I hadn't realized was a library I could make use of for my own stuff before then!
https://www.oracle.com/database/sqldeveloper/technologies/sq...
The sqlplus client continues to have all the problems that you have outlined. When sqlplus runs on the Windows command line, it does inherit command history.
in retrospect it's kind of stupid that ms-dos didn't have fantastic command-line editing and history facilities, since it had guaranteed control over the whole screen, a known keyboard layout, guaranteed at least 64k of ram, and was running milliseconds away from your fingertips without having to timeshare compute resources with other people or background compilation tasks (etc.). from today's perspective, it seems like a no-brainer in this context to build in a rich command-line history feature with search, cut, paste, and movement by words for command-line editing, plus filename completion. readline is solving a much harder problem given that it had to do everything through termcap and degrade gracefully on shitty terminals! maybe it's a harder sell when you have to write it all in 8088 assembly, or (more likely) when you've never used a system like that before
(for batch files, it also seems like a no-brainer to implement something like the bourne shell's control structures, and there was less excuse for not doing that, given that they were already aping unix's hierarchical filesystem)
it makes me wonder what kind of obvious improvements on today's computer systems we're missing
When it isn't just use 'rlwrap.'
> some time and see how awful it is to lose the ability to up-arrow get the last query back, edit it, and send it again.
[A
Apparently, a pretty incurious one.
when you want a program to get everything the keyboard sends (which means the program will have to handle everything the keyboard sends), your program needs to put your tty into "raw-mode". If you want the operating system to handle things like backspace or left-arrow to correct typos, instead of putting it into raw mode, your program just leaves it with the default in what-came-to-be-called "cooked mode"; cooked mode generally means a line will be sent at a time after ENTER, instead of char by char as keys are pressed (note, this is not the same as half and full duplex, though it is related).
Programs can be linked to use readline to read, while the rest of the program can be written as if the tty is in cooked mode; readline will handle the raw mode. Upon receiving characters typed and an ENT, readline will send cooked lines
If your program is getting char by char in raw mode, you need to realize that some keys don't send single chars, they send a series of chars, generally an "escape sequence", both because the sequence needs to be escaped, and also because the sequences escape with the ESC key-char.
you can get into weird modes where the ESC chars get swallowed but the rest of the key-chars pass through. That's what you are seeing when you see letters get typed but not interpreted as arrow key movements. if your keyboard is persistently screwed up, you can use stty to reset it, or if you are really fancy, set it the way your non-functioning program expects it to be set.
So if you aren't using readline, ^U deletes everything back to the beginning of the line. Readline preserves that because us pre-readline old farts have it wired into our fingers.
Before display terminals there were just printing terminals: IIRC # deleted a character and @ everything to the beginning of the line. This goes back to Multics and was carried forward in early releases of Unix at Bell Labs.
This is obviously pretty inconvenient when you have a display terminal but I am pretty sure that code remains in the bowels of the BSD TTY IO system at least and can probably be enabled. Maybe it even survived into linux for some compatibility reasons.
Also I WUZ THERE and I think Chet has it backwards on the POSIX controls issue: bfox talked about the TWENEX line input JSYS that had that stuff built in (I believe it was also, perhaps even earlier, in VMS, and maybe TENEX, and surely he had talked to some of those folks too)
http://gondor.apana.org.au/~herbert/dash/
The dash shell doesn't use GNU Readline to implement "set -o vi" but instead uses "libedit":
Debian does not appear to link dash with libedit, so command line editing is disabled.
They should consider doing so, and offering dash as an interactive shell. This would give people fewer bashisms to unlearn when they need to write portable scripts for the POSIX shell.
The dash shell is actually reasonably pleasant for interactive use when compiled with libedit.
For me it's the preferred interactive shell. When compiling dash with libedit, I edit dash source to enable tabcomplete. Then it feels more like NetBSD.
I've been running an experiment using busybox bash instead of dash as both interactive and scripting shell; have discovered numerous busybox idiosyncracies as a result. One thing I like about busybox bash is it's command history search: Ctrl-r. A bit faster than libedit.
NB. "set -o vi" can be abbreviated to "set -V"
busybox includes a config option to include a "bash" applet name that points to "ash"; typing "busybox" one will then see "bash" listed as an applet, but is the same shell
Sometimes when compiling software, authors insist on using bash scripts at compile-time rather than sh scripts. Trying to use busybox ash to run these scripts will fail because there is no applet called "bash". There might be another workaround but I find it useful to compile busybox to include a "bash" applet name.
Normally I would do this in NetBSD sh using the builtin "fc"; dash excludes the fc builtin.
Much like in the article I'm also a Vim user who excited to try Vim mode at the command line and I almost instantly hated it. I suppose it's almost required to mention tpope's rsi.vim[0] which gives you Readline bindings in insert and command modes. I've adopted Readline everywhere in my OS except in Vim normal mode which I believe is not too uncommon for many folks.
Other good ones with more double meanings:
- PROJECTionist - unimPAIRed - COMMENTary
Of course it's often in jest and somewhat famously the plural of anecdote is data, but it would be nice to know if there actually is something to the RSI. Ergonomics is complicated enough as-is.
That is why I have this in inputrc:
set editing-mode vi
set show-mode-in-prompt on
set vi-ins-mode-string +
set vi-cmd-mode-string :
set show-mode-in-prompt on
set emacs-mode-string \1\e[1 q\2
set vi-cmd-mode-string \1\e[3 q\2
set vi-ins-mode-string \1\e[5 q\2
bind -m vi-command 'Control-e: emacs-editing-mode'
bind -m emacs 'M-C-j: vi-editing-mode'
bind -m emacs-meta '"\C-j": vi-editing-mode'
bind -m vi-insert '"\C-e": emacs-editing-mode'Probably tens of millions at this point.
With `bash`, you can show which mode you're in by putting this in your `~/.inputrc`:
show-mode-in-prompt on
[foo@bar ~]$ # before turning it on
@[foo@bar ~]$ # after
@[foo@bar ~]$ set -o vi
(ins)[foo@bar ~]$ # and after I press esc:
(cmd)[foo@bar ~]$
> Tens of thousands of people probably use it every day without thinking about it.
seems like an understatement.
I would recommend against this. I delete from the cursor to the end of the line all the time. Maybe more in Emacs than readline, but still
You can already delete the whole line by issuing C-a then C-k. I also do this all the time and that key combo is as easy for me to type as any two letter word.
Remapping C-k means you now have two ways to kill the whole line and no way to kill from the cursor.
But worse: you're teaching yourself a keyboard setup that only you use. You'll be lost on somebody else's computer, or any time you don't have your config installed. The standard readline bindings turn up in sometimes surprising places and it's a joy when you realise you're already trained to use some system or software you've never used before.
I'm surprised the article doesn't also mention C-y. After you've C-k killed something it goes on to the "kill ring". Press C-y to bring it back! (You can bring more than the most recent thing back, but you now have the key words to look this up yourself).
> I sometimes think of my computer as a very large house. I visit this house every day and know most of the rooms on the ground floor, but there are bedrooms I’ve never been in, closets I haven’t opened, nooks and crannies that I’ve never explored. I feel compelled to learn more about my computer the same way anyone would feel compelled to see a room they had never visited in their own home.
- Implement readline(3)
- If you have any questions, see the man page.
- Extra credit: implement all GNU extensions.
#line-edit
^A home
^E end
^F right
^B left
^P up
^N down
^D delete
^W word-backspace
\ed word-delete
\ef word-right
THANK YOU FOR SHARING THIS
Ctrl-y yanks stuff back in from your kill ring. Esc-y will then replace the stuff with the previous item in the kill ring.
Ctrl-arrows moves a word at a time.
Ctrl-r starts a reverse interactive search through the history. Use any editing key, like ctrl-e to start working on the current search result.
Ctrl-g cancels Ctrl-r searches. So if you want to search and then search again you can clear the history position with Ctrl-g.
Ctrl-s moves forward if you went too far with Ctrl-r.
Ctrl-r twice in a row retrieves the previous search.
Alt-u upper cases. Alt-l lower cases. Alt-c capitalizes. From current position to end of word.
I use Ctrl-R a lot to find previous commands from my shell history. Often I have to press Ctrl-R several or even many times to find the right one. And then it happens that I overshoot and would like to change the search direction. Unfortunately forward-search-history is on Ctrl-S, which is overloaded by flow control. Occasionally I use flow control, so I cannot remove that. What solution are others using to get forward-search-history?
...basically "readline in a can" that you can use to improve text input of other programs.
I forget if it was RMS or ESR that really wanted to hold "readline" away from corporations, so it's licensed as GPLv3 (not v2), and he explicitly wanted it to _not_ be LGPL (ie: embeddable).
My crazy readline/bash tip is `ESC, .` (or maybe `alt+.`, I'm not 100% sure, random search result here: https://gist.github.com/tuxfight3r/60051ac67c5f0445efee ). Basically it up-arrows through the last "word" of previous commands, adding it to the current command.
Example:
mkdir foo
cd `esc, .` => cd foo
vim bar.txt
cp `esc, .` baz.txt => cp bar.txt baz.txt
I'm also in the same boat as the author in that `set -o vi` doesn't quite do it for me (although I consider myself an expert in vi). What I tend to do instead is:
C-a # <cr> => go to the beginning of the line and insert a comment char
fc => "fix command", see "help fc" (it's a built-in)
This lets me intentionally "damage" a command, but then get into `vim` to "fix it up" (eg: `dw, cw, df/, etc...`). Remove the comment from the beginning of the line to execute it, or keep it there to take the command back "out" of `vim` so you can keep building it up.
mkdir -p foo/bar
cd !$ # => cd foo/bar
I don't recall what the bug was, I also don't recall why the video I sent showcasing the bug had my national anthem in the background. But it sparked a nice bug report exchange.
That he is doing this for no pay and for the passion of providing software to users all around the world fits.
> Ramey has now worked on Bash and Readline for well over a decade. He has never once been compensated for his work
These people are generally unpaid and that has big ramifications. Sure, it makes them susceptible to bribes and more prone to hostile takeovers (however you view xz), but there's more. It means these works are simply a work of passion, and they are working on them in their spare time. We all know here that that time is less and has lower priority than our work time. So you're more likely to be rushed and make mistakes, or not be able to fix things fast enough, and so on. Is this really the way we want this infrastructure to be maintained? Startups have more stability than this and are more likely to pass the "CEO hit by a bus" rule[1]. Not to mention the difficulties of transitioning when the dev wants to retire or move onto something else. This does lead to issues, and while we've solved them in the past, that doesn't mean we can't do better.
Surely they deserve some compensation. We have 8 companies which are worth over a trillion dollars and certainly all 8 use readline. Are you telling me that these companies can't collectively kick back $200k/yr. That's literally what Elon makes in 10 minutes if he gets 5% on his wealth. I find it hard to believe we don't have enough money to fund possibly thousands of these projects (if there are that many, but there are likely that many people). It only takes Ballmer 16 minutes and there are 20 people who can do it in 30 minutes or less (~50 for 1hr). Collectively the top 10 make 230 million per day for doing nothing. I think we have the money. (remember, we're talking about products that these companies are using. This isn't just a "pure donation," as these things have clear market value and the loss of the maintenance will result in these companies also losing money)
At the least, I'd like to remind everyone that the company you work for likely offers some matching donation. If you're at one of the big tech, they do.
[1] If your CEO, or any single specific employee, is hit by a bus and can no longer work, does the entire company also collapse?
socat readline prog
rlwrap prog let mut line = String::new();
std::io::stdin().read_line(&mut line).unwrap();
You basically shouldn't use `tokio::io::stdin()` for what you want, as is documented at https://docs.rs/tokio/latest/tokio/io/fn.stdin.html
> For interactive uses, it is recommended to spawn a thread dedicated to user input and use blocking IO directly in that thread.
So the code under tokio would be
let line = rt.spawn_blocking(|| {
use std::io::stdin;
let mut buffer = String::new();
stdin().read_line(&mut buffer).unwrap()
}).await.unwrap();