leonim

I'll say it again, Visidata is one my favorite new CLI/TUI tools over the last few years. It makes it extremely easy to explore/modify structured data in the terminal.

In the 2.9.1 release, they added support for: - window functions listing values from around the current row. - XDG support for config files. - new loaders for Apache Arrow (IPC and streaming) and parquet formats - Ability to quickly browse zip files on the web. - And even some support for Windows users. Though I usually think of this as mostly a *nix terminal based tool.

It can still load and view many different types of file formats like csv, usv, lsv, json, Excel.

Check out the tutorial and youtube videos.

If you are interested in this subject, Carl Zimmer wrote a great book that has all sorts of examples of parasites that control their host: "Parasite Rex: Inside the Bizarre World of Nature's Most Dangerous Creatures"

https://www.amazon.com/Parasite-Rex-Bizarre-Dangerous-Creatu...

I would add VisiData to that list. It is more suited for tabular data, but it can be used to explore large nested data. I find it is a good starting place to explore json data I'm not familiar with. I will also sometimes flatten the data with some general jq scripts or gron.

If I'm trying to understand some new json data, I start with visidata, and then I poke around to find the data I want. If it is data I want to extract again, I use jq to get the interesting bits. Sometimes I use that in combination with sqlite-utils to store that data so I can query it. (I haven't tried some of the other tools that will create a sqlite database for you.)

- https://github.com/saulpw/visidata - https://github.com/simonw/sqlite-utils

Small data sources like this I find are fun to explore from the terminal with Visidata [1]. Just run:

vd https://www.icann.org/resources/pages/listing-2012-02-25-en

Makes it easy to find TLDs owned by Amazon, Microsoft, and Google. What years had the biggest expansion (2014).

Now I'm wondering why there were no new ones added in 2020? (I guess Covid-19 was so bad.)

[1] https://www.visidata.org/

Looks like it is a country code: https://en.wikipedia.org/wiki/.io

I would suggest using the Jeremy Singer-Vine Tutorial: https://jsvine.github.io/intro-to-visidata/the-big-picture/v...

I had always wanted to use vim to do some CSV viewing and editing, but I never found a good solution. This plugin looks pretty amazing from the demo, but I would still recommend VisiData. It does tables in a more natural way. Plus VisiData can handle many different file types. Menus were recently added to VisiData which helps make it more approachable.

Thanks Will for creating Rich. I'm also a big fan of Rich.

Are there any plans to make it easy to generate the Rich tables from the command line? Such as from CSV, TSV, JSON/L, or even SQLite databases?

Second this recommendation for the terminal. For my CLI toolbox, VisiData is my favorite.

I find VisiData is great for quickly exploring and querying data from the CLI. It can handle many types of files (SQLite, CSV, TSV, Excel, JSON, YAML, etc). Visidata loads all the data into memory, and so is very responsive when exploring the data. It allows you to quickly do all sorts of of adhoc queries interactively, without having to write a valid SQL query.

I haven't used Q. When I first heard of it, I liked the idea that Q allowed you to run random queries on CSV and TSV files. However, it seemed like it would be slow if you wanted to do follow up queries, since it had to repopulate the in memory SQLite file for each query. Though it looks like the latest version has a way to cache the generated sqlite file. So that seems like it could help.

Also, if I have some CSV, TSV, JSONL data sqlite-utils is useful for converting them to SQLite, and then exploring with Visidata or SQL queries.

Q: https://github.com/harelba/q sqlite-utils: https://github.com/simonw/sqlite-utils

Textual is developed by the author of Rich, the Python library for fancy formatting in the terminal. Is is a framework based on Rich for creating TUI apps. It is inspired by Vue.

The article points to a YouTube video https://youtu.be/bXgIj2cXaZ4, where Will shows how to style a 4-pane TUI with CSS-like configuration file. The styling also includes the ability to style based on whether a panel is active or not.

This looks really powerful. I expect to start seeing a lot more TUIs written in Python using Textual. It looks like it will be easy to quickly build a very nice terminal app with this.

Visidata is my favorite new CLI/TUI tools over the last few years. It makes it extremely easy to explore/modify structured data in the terminal.

This release adds a couple new loaders for odt and the lsv. LSV (or Line separated values) is based on a request for an "awk" like key/value loader.

In the 2.6 release, Visidata added menus which makes it much easier to find commands.

I think the best way to learn about what it can do is see Jeremy Singer-Vine's tutorial: VisiData in 60 Seconds: https://jsvine.github.io/intro-to-visidata/the-big-picture/v...

Or the developer's YouTube videos: https://www.youtube.com/playlist?list=PLxu7QdBkC7drrAGfYzatP...

By analogy csvkit is to visdata as sed is to vim.

Visdata is an interactive tool and you can quickly explore/edit data directly. While csvkit is batch oriented. Visidata can be used in a batch way, but that is not its strength.

I prefer to use a similar tool mlr. I find mlr is complementary to Visdata. Either to process the data before going into Visdata, or to figure out what mlr commands I want to use for a shell script.

If you aren't familiar with Visidata, this video gives a pretty good idea of what it can do: https://youtu.be/N1CBDTgGtOU

It seems like making a living on shell tools can be difficult to impossible. The author Saul is using Patreon to help fund his work on this tool. I love this tool, and hope others can help support him too. Maybe this "duo-repo" approach will allow him to do this full time.

You can watch talks from past conferences: https://www.youtube.com/c/csvconf

Simon Wilson will be a keynote speaker this year, talking about "Datasette and Dogsheep: Liberating your personal data". Simon, Datasette and Dohgsheep show up on Hacker News regularly.

Here's my take, most of which is in other comments:

If you like GNU Screen, continue using that, they are very similar tools. I’m a long term user of both, using both for a while, and now only using Tmux exclusively. I need a terminal multiplexer, since I often use my main computer via ssh.

The main advantage of GNU Screen for me was it was slightly more robust, it crashed less for me. While both are pretty stable. I can run both for weeks at a time, and not have any issues. Though, I’ve heard others report the opposite about stability between the two tools.

Tmux has some advantages for me:

   - Tmux is scriptable.  You can run tmux commands to query and set  the state
    of Tmux. It is possible to pipe and process those results. There is some
    thought behind the commands, that make them consistent and orthogonal.
    Screen never really could not do much beyond listing the  available
    sessions.
     - This has made it easy for 3rd party developers to create Tmux plugins, adding some useful features.
  - I think GNU Screen can now do split screens, but I feel it is easier to do
    in Tmux.  Many times I have a monitor with full a window terminal, and Tmux makes
    it easy to split that up into smaller independent terminal panes.

  - GNU Screen development had pretty much stalled for years.  Tmux is being
    actively developed.  Outside developers are welcome to make contributions. 
     - The lead Tmux developer, Nicholas Marriot,  is actively engaged with
       users. He responds to bug reports, helping track down bugs, and
       collaborating with contributors.
     - I believe GNU Screen development has started again, but to me it just
       doesn’t have the same level of engagement from the developers.
     - Tmux has regular releases, compared to Screen
  - Tmux is more current with the latest ANSI/SGR/OSC codes, and has robust
    unicode support.  This hurt me most when I use  a wide terminal (> 256
    columns), and wanted to use the mouse, going past a certain column was
    broken.  There is a new encoding SGR 1006, and it didn’t work with GNU
    Screen, but did with Tmux.   GNU Screen picked this up in 2019, but Tmux
    had it for years before.  There are other codes for 24bit color, italics,
    fancy underlines, etc that GNU Screen doesn’t support, but many modern
    terminal emulators do.  Even ITerm2 has Tmux support builtin. While I
    generally don’t need these but if/when I do, Tmux just works, while GNU
    Screen just ignores these codes.

While both are capable and usable, there are a lot of details that make Tmux better to me.

The tutorial is well written and gives a great tour of the tool: https://jsvine.github.io/intro-to-visidata/

For interactive slicing and dicing of csv, xml, json, etc. I like VisiData. For scripting I like miller and jq. Not sure how that compares to your PS setup.

There are some recent Unix shells, like nushell for working with structured data, but I haven't used them.

  - https://github.com/saulpw/visidata
  - https://github.com/johnkerl/miller
  - https://github.com/nushell/nushell

I'll add to that list: https://github.com/johnkerl/miller

Also, I like using VisiData already mentioned.

That looks amazing. I've used NetApp NFS filesystems that provided user level snapshots, where it automatically created snapshots of files. It wasn't after every command, but at regular intervals. You could access the files under `.snapshot` directory.

Searching now, it looks like this is part of NetApp ONTAP, and managed by a snapshot policy.

My favorite terminal based client to explore SQLite databases and many other data formats is the TUI VisiData[1].

It has a spreadsheet-like interface and makes it very easy to explore a SQLite database in the terminal

[1] https://www.visidata.org/

[2] tutorial: VisiData in 60 Seconds(https://jsvine.github.io/intro-to-visidata/the-big-picture/v...)

I'm not familiar with the library, but looking at the documentation, it looks like it provides a programmatic way to get information about the schema of a table just from the create statement, which seems useful if you want to do something programmatic with any SQLite database.

It does look like SQLite provides some PRAGMA commands that can do some of the same things, like:

  PRAGMA foreign_key_list(table-name);
  PRAGMA schema.index_xinfo(index-name);

https://www.sqlite.org/pragma.html

If the terminal is your home, I would suggest looking at something already mentioned by someone else VisiData (http://visidata.org), it is great for exploring all sorts of data files like CSV, Excel, Sqlite3, etc in the terminal.

I find the Datasette author's related tools sqlite-utils (https://github.com/simonw/sqlite-utils) and the Dogsheep tools (https://github.com/dogsheep) and VisiData are nicely complementary.

I prefer the interface of VisiData, but I don't usually need to share links.

Yes, vd does a nice job on JSON data. To open nested data you can use the expand-col (keyboard shortcut "(") command, and the unfurl-col "zM" command.

You can also use the pyobj-cell "z^Y" command to explore single cell as a table. A single cell can be a whole nested part of your JSON data.

Also, the transpose "t" command is useful when working with JSON that is an object with a lot of keys.

Totally agree. I think mlr is a wonderful CLI tool. It is very robust, and can handle/convert multiple tabular formats including csv, tsv, json, fixed-format, etc. It has a pretty decent text output formatting, with the --opprint flag.

I use to be very comfortable using awk/sed/perl/sort/uniq/tr/tail/head from the CLI for the sort of data cleaning this article is talking about. However, over the past year I've found I use VisiData https://github.com/saulpw/visidata for interactive work.

If I need to clean up the data first, I'll use mlr or jq as input to Visidata. If my data is too dirty for mlr, then I'll use Unix toolbox tools mentioned as input to mlr, jq or VisiData.

VisiData provides some ability to script, but when possible I prefer to have the shell do the scripting with all the tools mentioned as input to Visidata.

I recommend Jeremy Singer-Vine's tutorial for getting started: https://jsvine.github.io/intro-to-visidata/index.html

The tutorial gives you a good taste of what it can do and how to use it.