Hacker News Reader

Comment by fwip | Hacker News Reader

I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these, so, I guess that's an accurate title.

View on HN

I didn't know about shuf because it's a GNU specific utility, and not installed on my Mac by default.

Homebrew installs the GNU Core Utilities with the g- prefix, and checking now I see it's available as gshuf.

I see a couple of problems with how it's used in this essay:

  cat big_csv.csv | shuf | head -n 50 > sample_from_big_csv.csv

Since shuf's -n option is "output at most COUNT lines", there should be no need for the head:

  cat big_csv.csv | shuf -n 50 > sample_from_big_csv.csv

In principle this should be faster because there's less I/O, and it should take less memory if shuf implements the -n option using something like reservoir sampling. (EDIT: it does - https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shu... )

Also, since shuf takes a filename, it could be:

  shuf -n 50 < big_csv.csv> sample_from_big_csv.csv

However, in terms of data science, don't most CSV files contain a header? Most of mine do, and the example 'data.csv' has the header "var_x".

Using "shuf" means there's a very high chance that the sampled CSV file either won't include the header line, or will have it somewhere other than the first line.

Doesn't that mean that most data scientists will rarely use 'shuf' for sampling from CSV files?

There is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines.

The GNU utils are usually the way to go; the Mac/Darwin/BSD variants have some weird quirks and usually aren't worth the bother to fight with. awk and sort are notably deficient.

Ease off the "data science" a tad--this is just scripting, a perfectly honorable pastime! Why else would GNU coreutils exist in the first place?

There is a CSV standard, RFC 4180. There are also many variants. I used products which specifically say that follow that RFC.

It's awkward to preserve a header using shuf, even with head/tail. Here's what I came up with:

  head -1 x.csv && (awk 'NR>1' x.csv | shuf -n 5)

My point is that "shuf" is GNU specific, while the author said it was a Unix tool like the other ones. Linux != Unix. Also, there are no Mac/Darwin/BSD variants of shuf.

"Ease off"? Why? The term "data scientist" appear in the title and each of the first three paragraphs, and the comment I replied to was "I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these". I want to know why data scientists who work with CSV files should be using shuf and not some more appropriate tool like csvkit.

That is, the entire topic is about scripting as applied to data science, not "just scripting".

I do something like:

    < foo.csv (read -r head; echo “$head”; shuf -n 50)

If you do this a lot, you could combine the read/echo into a single shell function.

Or write a Python program. But a quick test shows that CPython (using a pure Python reservoir sampling) is 1/4th the speed of shuf.

RFC 4180 is really a well-intentioned attempt to document popular ways CSV is used in the wild, but it does not set a standard. There is no such thing as CSV well-formedness in the way that exists for XML. Can a CSV field value span lines or not? How are linebreaks encoded within a field? Can backslash escape quote (or other) characters, or is the double-doublequote approach contemplated in the RFC proper? How is one to know definitively if the first line is a header or a data row? What about comments?

My beef with "data science" is that this is not science--it's even less science than political science--it's munging.

shrug Sure, yes, I pointed out there are many variants. But you said there wasn't a standard when there is. You meant to say that there is no widely used consistent CSV format, I think, which is a bit different.

RFC 4180 says a CSV field can span multiple lines. Section 2.6:

> Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.

You asked "Can backslash escape quote (or other) characters". 4180 says "no", and that double-double quoting is used. The grammar in 2.7 is:

> escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE

The RFC says that the MIME type should use header=present or header=absent to indicate if the first line is a header. "Implementors choosing not to use this parameter must make their own decisions as to whether the header line is present or absent."

I have never seen a CSV file with comments.

Note also that the Python csv module, which supports a large number of CSV dialects, also does not support comments.

While you have a beef, that doesn't change that you wanted to drag me off topic (from "data science" to "just scripting"), when my point was that that example wasn't a good data science example in the first place. It also isn't a good scripting example.

FWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV):

    Randomly samples CSV data uniformly using memory proportional to the size of
    the sample.

    When an index is present, this command will use random indexing if the sample
    size is less than 10% of the total number of records. This allows for efficient
    sampling such that the entire CSV file is not parsed.

    This command is intended to provide a means to sample from a CSV data set that
    is too big to fit into memory (for example, for use with commands like 'xsv
    frequency' or 'xsv stats'). It will however visit every CSV record exactly
    once, which is necessary to provide a uniform random sample. If you wish to
    limit the number of records visited, use the 'xsv slice' command to pipe into
    'xsv sample'.

    Usage:
        xsv sample [options] <sample-size> [<input>]
        xsv sample --help

    Common options:
        -h, --help             Display this message
        -o, --output <file>    Write output to <file> instead of stdout.
        -n, --no-headers       When set, the first row will be consider as part of
                               the population to sample from. (When not set, the
                               first row is the header row and will always appear
                               in the output.)
        -d, --delimiter <arg>  The field delimiter for reading CSV data.
                               Must be a single character. (default: ,)

> don't most CSV files contain a header?

  tail +2

D'oh! I forgot that one, and gave an awk description instead.

Personally, I'd use another tool to remove the header before sending it to shuf. As an example:

    cat myfile | tail -n +2 | shuf -n 50

I know most of those, but I've never used awk.

One command I didn't see mentioned that was surprising was cut[1], especially the -d flag. (Maybe it's less necessary if you know awk?)

It's really great for cleaning up the output of poorly structured, wordy scripts. It's also a quick and dirty way to generate csvs with the –output-delimiter flag.

[1]https://www.geeksforgeeks.org/cut-command-linux-examples/

I’m a windows .net dev and I know 6 of them. It’s quite useful to run these on windows using Cygwin. Grep I use a lot the others not so much.

I assumed the bc command would be listed.

Reply on news.ycombinator.com