Bash Completion from JSON Fields | Hacker News Reader

Bash Completion from JSON Fields

github.com

71 pointsmedv20 comments

Tangentially related: PowerShell offers something similar for XML and JSON.

    # XML:
    [Xml]$Data = Get-Content Data.xml
    $Data. <tab> # browse the file with <tab> and <.>
    
    # JSON:
    $Data = (Get-Content Data.json | ConvertFrom-Json)
    $Data. <tab> # browse the file with <tab> and <.>

Similar in concept and certainly useful. I'm happy to see this project!

The new rule on HN is that anybody who talks about Powershell has to share their profile ;)

-- Happy pwsh user on arch.

One thing that strikes me as a person who grew up with mouse pointer interfaces (starting with Workbench on Amiga) is how strange it is that the terminal allows for a time-lag between a mistake in typing and the user's becoming aware of it by pressing Return. Intense tab spamming removes this alienating phenomenon though, and I imagine that with a perfectly smart terminal (and commands), virtual tab-spamming is automatic so that a user is immediately informed and even prevented from typing something that could not possibly render a valid command.

I suppose this is how voice command will evolve too. Cortona will stop you short when you're not making sense, and even learn your idiosyncrasies in expressing yourself much like a spouse does.

By the way, I for sure thought this link was about a the complete J.S. Bach collection.

That's an interesting invariant that I've been thinking about: hitting TAB should never generate something that would be an invalid command (or more pedantically, generate a command prefix that would have a syntax error once the user fills in the rest of the command).

You would THINK that shell tab completion already works that way. But it doesn't in bash, and not even in fish or zsh.

Not only does the completion logic duplicates the tool's own parsing logic, it doesn't attempt to do it faithfully.

(1) First, there's version skew in all the repos for bash, fish, and zsh.

(2) Leaving that aside, the parsing models they use aren't rich enough to express some command lines.

"find" is a good example. They will let you do something like

find -type f -dept<TAB> -- completes -depth, but it's not valid here

This is because find is split up into options and the expression language. There are probably some simpler and more compelling examples I can come up with.

Another one is completing flags in this situation:

grep -- --c<TAB>

After --, there are only args, not flags, so it's misleading to complete.

----

Being able to spam TAB is what I'm going for with Oil. That's not true of bash. One annoyance the (y/n) prompt that bash gives you when there are thousands of completions (e.g. hitting TAB on an empty line).

I have some upcoming blog posts along these lines, which I mention here: http://www.oilshell.org/blog/2019/02/05.html

Glad to hear someone is thinking about this. Right, hitting tab should never generate something invalid. That's the shell being sound. Conversely, the system should also be able to determine whether what's being typed could possibly become a valid command or not. That's the shell being complete.

Yes, I think it basically reduces to the problem of using the tool's ACTUAL parser. There are enough variations in flag syntax that it's not possible to be 100% correct without running arbitrary code. And if you're going to do that, you might as well run the code that's already written (with some caveats).

That could be solved by my proposal for a "Shellac protocol", akin to Microsoft's Language Server Protocol (e.g. for VSCode):

http://www.oilshell.org/blog/2018/12/05.html

Summary:

1. The protocol was mainly motivated by the problem of bootstrapping a corpus of shell completions. Every shell has to "boil the ocean" itself, i.e. provide completions for many popular commands. git's completion is thousands of lines in each of bash, zsh, and fish. Upstream only maintains bash.

2. What I noticed in implementing completion for Oil is that bash fundamentally CONFLATES two problems:

a) The problem of completing the shell language itself, e.g. ${H -> ${HOSTNAME, or ~a -> ~andy

b) The problem of completing command line flag syntax.

This is a huge mistake from the POV of implementation complexity. And obviously if you want shell-agnostic auto-completions, (b) needs to be separate from (a), which is shell-specific.

Luckily the authors of bash-completion tried to separate these two problems a bit, and I was able to salvage most code.

Since I wrote that post, and implemented a compatible completion API in OSH, it's become a lot clearer to me how the protocol should work.

3) The invariant you mention would be a third benefit of having a more principled completion API that can run arbitrary code (in arbitrary languages).

Now, I'm not saying that EVERY command needs to use this. There are some engineering problems with this approach too. But I think you should have the POSSIBILITY of doing something correct if you want. Heuristics may be OK in the simple cases.

But overall what I discovered is that the system is a huge mess. The bash-completion project tries to parse bash in bash, and unsurprisingly does a poor job. My upcoming blog posts will explain this.

Bash/ZSH can have arbitary completion functions/scripts that receive the full input.

A lot of the completions are auto-generated from man files, which don't contain information like "argument x is only available if argument y is present", etc. (at least in a structured way)

It's "just" a matter of implementing completions manually to enable what you are asking for.

Nice!

The results seems somewhat muddied by the long lists. Have you considered omitting all but the first and last entries?

There is a very useful readline option to help with this:

  set completion-prefix-display-length 2

in ~/.inputrc replaces any common prefix longer than two characters with an ellipsis.

Or even just `allFilms.edges[6]` - just represent the max or size of the list.

General Developer Experience (DX) speaking, IMO, File and Folder are kind of better at CRUD-ing json. One file per attribute, and foldable attributes are folders (object and array types). It scales quite well rather than loading a big file into memory.

Tools around file system are battle tested. Basic operations; read=ls, create=touch/mkdir, update=mv, delete=rm. Linking data = symbolic link. File explorer/manager helps managing attributes much nicer than in-file operations. It can even use extension as types, structural folders as schema.

jq is nice but you've already got: find, grep, awk, sed programs.

The only downside I can think of is size might be larger depending on what's minimum block size (e.g. 4096 bytes) hmm

Just bunch of nonsense.

Could you elaborate which part is nonsense?

After reading the four-word-long README text and viewing the gif, I have no idea what this is or does.

Yes. Command line completion for fx is pretty obvious, if you know what fx does. It's missing a link to fx.

https://github.com/antonmedv/fx

I wasn't asking, but thanks. Was more making a point about the shitty readme.

I was going to suggest that somebody try this in Oil shell, which can now run thousands of lines of bash completion scripts [1], but I can already see it won't work because of the VERSION DETECTION anti-pattern.

https://github.com/antonmedv/fx-completion/blob/master/compl...

    if [[ -n "${BASH_VERSION}" ]]; then
      # ...
    else
     echo "Unsupported shell."
     exit 1
    fi

The better thing to do is FEATURE DETECTION, e.g. with 'eval'.

Does anyone know of a concise resource for explaining feature vs. version detection? I don't find that many great resources on Google.

Here's a page, which is OK but leaves a lot to be desired.

https://en.wikipedia.org/wiki/Feature_detection_(web_develop...

It feels like I need to write a blog post to explain this, since this knowledge seems to be waning. I recall reading about it 10+ years ago, but not recently.

Richard Stallman understood it very well and IIRC strongly advocated for feature detection in autoconf. (Despite all the bad things about autoconf, it does actually work on tons of platforms, and it's solving a hard problem. It wouldn't "scale" if it were using version detection.)

Here's an outline of a blog post:

1. Version/browser detection is why the User-Agent string from Internet Explorer, Chrome, Safari, Edge, etc. all start with "Mozilla". Fun fact: if everyone used feature detection, we could have saved exabytes of data transfer over the last 20 years by trimming 10-50 bytes from every single HTTP request ever made.

2. jQuery switched from version detection to feature detection early in its life. I recall some threads where Resig was criticized for this, and then he saw the light.

3. Likewise, the LLVM toolchain has to pretend its GNU in some cases because applications are using version detection: https://news.ycombinator.com/item?id=15195370

4. Autoconf is based on feature detection.

5. My hand-written configure script also uses feature detection, e.g. to detect the presence of the readline library. It basically tries to compile a small program with the C compiler. There are various tricks in this domain.

6. eval is useful for feature detection in dynamic programming languages. In fact, this might be the ONLY thing eval is good for! (exaggerating a bit)

[1] http://www.oilshell.org/blog/2019/02/05.html

Feel free to send a PR with feature detection. =)

Does it work with json over http?

See examples.

Reply on news.ycombinator.com