sounds like you are looking for xsv. I like that tool a lot for selecting specific columns.
359 karma · joined September 27, 2021
A stream does not work for tv because a stream does not know how many rows are in the file a priori. Displaying the dimensions of the file is a priority for `tv`. I am very happy with that trade-off. I would rather know the dimensions of a file than have a file stream of unknown dimensions.
From the Unix philosophy:
> Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
1. As a rule-of-thumb, I have been working on functionality before optimization. That said, `tv` is really fast. It is completely false that `tv` only works for relatively small files. I just pushed a 624MB file to `tv`. It ran in 2.8 seconds. With `column` it takes 5.0 seconds. Now, I would love help from programmers smarter than me. I am sure there are a lot of optimization gains to be had in `tv`. I just wanted to make sure potential users are not misled. `tv` is performant.
> Some (most?) tools that output data in columns and fit each one to the largest value in that column need to scan the whole file as a first pass just to start displaying data.
> Not only is it the case with this tool, but from what I'm reading in main.rs it looks like it's also loading the whole file in memory.
2. `tv` reads once, but parse partly. This means that it reads the full file only to grab the number of rows. It only parses(take) the first n rows.
https://github.com/alexhallam/tv/blob/b548f0d19f64438d53f732...
https://github.com/alexhallam/tv/blob/b548f0d19f64438d53f732...
1. As you noted NA comprehension 2. Column overflow logic for different sized terminals 3. Summary meta data in the header 4. Significant digits logic. This allows users to view more columns than they would otherwise view due to decimal dust shifting the columns over. 5. This is the most import! It looks really pretty!
> How does tidy-viewer compare with csvlook?
The most important issue to me is that csvlook is a much less pleasant viewing experience, but there is also this ...csvlook reads and parses all of the data. Try pushing diamonds.csv to csvlook. When I do it on my machine it takes 15.228 seconds while tv takes 0.0042 seconds. For this reason tv is much faster, but speed is not the goal of the package. tv's purpose is to maximize viewer enjoyment.
2. Looking at the demo video, there seems to be an odd fixation with "N/A". The CSV spec, AFAIK, doesn't recognize this phrase. I don't understand why someone would expect a quoted string field whose raw characters are "n/a" should be rendered as anything other than n/a (i.e. lowercase and without the quotes). I'm guessing maybe in your workflow you want to use that phrase a lot, but for a tool for the general public I'd not do this kind of interpretation; and I would leave an empty field as empty.
I could not say it better than this:
> The norm of treating missing data as NA exists in R (which the developer of this is clearly inspired by based on the GitHub readme.). Pandas in Python is stuck with NaN for numeric types (not quite correct) and "" or None for string types. Personally I like the choice to both explicitly render missing data in colour and to apply NA as a placeholder text to display that colour.
3. tidy-viewer seems to require "unstable library features", or at least ones which were unstable as of Rust 1.48.0 . It would be nice if you could be compatible with older rust distributions/versions.
That is a good point. I also release binaries which I think makes this requirement less needed. What are your thoughts.
4. Many systems, especially older ones, especially ones which you access remotely and don't have root privileges on, won't have a rust installation. It would be even more convenient if you could provide binaries with little or no extra dynamic library dependencies, which could be used on older / rustless systems. I realize this is a tall order, however.
With github actions I auto-build binaries for many OSes. See https://github.com/alexhallam/tv/releases/tag/0.0.13
5. What about scrolling? The worst part of viewing CSVs is having to handle wide ones which exceed the terminal width, and having decent horizontal as well as vertical scrolling ability. less doesn't cut it, because it doesn't keep the header row, plus it doesn't recognize field widths.
Scrolling is nice. To offer scrolling the only option I am aware of is turning this cli into a tui. I made the choice early on to stay chose the more minimal path and stick to a cli. The goal is to be a `column` replacement not a spreadsheet replacement.
6. tidy-viewer does not seem to support wrapping longer fields onto multiple terminal lines.
The goal is to glance at the data as a whole not a cell or fields. If there are cells with long text they get cut at 20 characters. I like this a lot. I would prefer to know that there is a lot of text that I can dig into latter, but when I am glancing at the csv I just want an overall picture. In my view tables of data are data visualizations meaning that I don't have to show everything to understand enough of it.
7. When the user doesn't specify the color scheme, are you choosing one based on the terminal colors, or are you using absolute color values? I suggest the former.
Great question. I want to eventually add the ability for users to make a config file will their own colors. At this time I just have absolute presets. If you are interested I would happily take a contribution that allows users the option to configure tv with some dotfile.
8. tidy-viewer loads and parses the entire CSV immediately; and, in fact, seems to keep two copies of it in memory at once. This means it cannot be used with large files without thrashing; and even if your CSV does fit in global memory, it will still be kind of unusable, trying to dump gigabytes onto the terminal.
That is almost true. tidy-viewer reads the entire csv, but only parses the head. If I knew of a way to get the number of rows and columns of a csv without reading the whole file then I would. I know there is a good deal more room for memory optimization. This is not my strength and I am still learning.
9. Bottom line: A nice initial effort, but the more serious challenges are yet to be tackled, plus needs to be more robustly cross-platform.
Thanks for the compliment. It is still a work in progress.
Columns that are exceptionally long but cutting and appending an ellipsis if the line is over 20 characters.
> a built-in grepping feature would make this very useful.
see the following for csv data manipulation:
xsv - Command line csv data manipulation. Rust
csvtk - Command line csv data manipulation. Go
tsv-utils - Command line csv data manipulation toolkit. D
q - Command line csv data manipulation query-like. Python
miller - Command line data manipulation, statistics, and more. C
2. I need to update the README the binaries are here https://github.com/alexhallam/tv/releases/tag/0.0.13
I recently moved away from manually building binaries to automated building for many architectures. I am still learning how to use github actions to build for a matrix of architectures. I am still learning.
1. tv gives a quick summary of the count of rows and columns
2. tv shows all columns at the bottom that don't fit in the terminal. With vd I have to scroll on wide data.
3. tv guides the eye to missing data better with NA highlights
4. tv has sigfig logic that is better. I work with files where the decimal dust can become long. Those unnecessary characters pushes remaining columns off the screen. This means the user would need scroll over to see additional columns. I generally think it is better to avoid additional key presses if possible.
5. tv is fast for large files. It does not have to read and format all of the data like vd. tv is focused more on looking at the file and not operating on file. It does not have to do as much as vd. That helps tv with what it is uniquely good at. "Do one thing and do it well"
It does not matter if your file is really wide (lots of columns) or really long -- tv will give the user a compact useful pretty print of the data. Why not use vd as a TUI spreadsheet and tv for glancing at csv files. They are both great tools in my eyes with different purposes.
tv modifies raw files in the following ways:
1. NA detection and highlighting 2. Printing only significant digits 3. Header and footer meta data
I have been using this a lot at work. There is a lot more work to do, but it is in a usable state.
Give it a try! If you like it then star on Github!