Hacker News Reader

FWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV):

    Randomly samples CSV data uniformly using memory proportional to the size of
    the sample.

    When an index is present, this command will use random indexing if the sample
    size is less than 10% of the total number of records. This allows for efficient
    sampling such that the entire CSV file is not parsed.

    This command is intended to provide a means to sample from a CSV data set that
    is too big to fit into memory (for example, for use with commands like 'xsv
    frequency' or 'xsv stats'). It will however visit every CSV record exactly
    once, which is necessary to provide a uniform random sample. If you wish to
    limit the number of records visited, use the 'xsv slice' command to pipe into
    'xsv sample'.

    Usage:
        xsv sample [options] <sample-size> [<input>]
        xsv sample --help

    Common options:
        -h, --help             Display this message
        -o, --output <file>    Write output to <file> instead of stdout.
        -n, --no-headers       When set, the first row will be consider as part of
                               the population to sample from. (When not set, the
                               first row is the header row and will always appear
                               in the output.)
        -d, --delimiter <arg>  The field delimiter for reading CSV data.
                               Must be a single character. (default: ,)

There is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines.

The GNU utils are usually the way to go; the Mac/Darwin/BSD variants have some weird quirks and usually aren't worth the bother to fight with. awk and sort are notably deficient.

Ease off the "data science" a tad--this is just scripting, a perfectly honorable pastime! Why else would GNU coreutils exist in the first place?

There is a CSV standard, RFC 4180. There are also many variants. I used products which specifically say that follow that RFC.

It's awkward to preserve a header using shuf, even with head/tail. Here's what I came up with:

  head -1 x.csv && (awk 'NR>1' x.csv | shuf -n 5)

My point is that "shuf" is GNU specific, while the author said it was a Unix tool like the other ones. Linux != Unix. Also, there are no Mac/Darwin/BSD variants of shuf.

"Ease off"? Why? The term "data scientist" appear in the title and each of the first three paragraphs, and the comment I replied to was "I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these". I want to know why data scientists who work with CSV files should be using shuf and not some more appropriate tool like csvkit.

That is, the entire topic is about scripting as applied to data science, not "just scripting".

I do something like:

    < foo.csv (read -r head; echo “$head”; shuf -n 50)

If you do this a lot, you could combine the read/echo into a single shell function.

Or write a Python program. But a quick test shows that CPython (using a pure Python reservoir sampling) is 1/4th the speed of shuf.

RFC 4180 is really a well-intentioned attempt to document popular ways CSV is used in the wild, but it does not set a standard. There is no such thing as CSV well-formedness in the way that exists for XML. Can a CSV field value span lines or not? How are linebreaks encoded within a field? Can backslash escape quote (or other) characters, or is the double-doublequote approach contemplated in the RFC proper? How is one to know definitively if the first line is a header or a data row? What about comments?

My beef with "data science" is that this is not science--it's even less science than political science--it's munging.

shrug Sure, yes, I pointed out there are many variants. But you said there wasn't a standard when there is. You meant to say that there is no widely used consistent CSV format, I think, which is a bit different.

RFC 4180 says a CSV field can span multiple lines. Section 2.6:

> Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.

You asked "Can backslash escape quote (or other) characters". 4180 says "no", and that double-double quoting is used. The grammar in 2.7 is:

> escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE

The RFC says that the MIME type should use header=present or header=absent to indicate if the first line is a header. "Implementors choosing not to use this parameter must make their own decisions as to whether the header line is present or absent."

I have never seen a CSV file with comments.

Note also that the Python csv module, which supports a large number of CSV dialects, also does not support comments.

While you have a beef, that doesn't change that you wanted to drag me off topic (from "data science" to "just scripting"), when my point was that that example wasn't a good data science example in the first place. It also isn't a good scripting example.

> don't most CSV files contain a header?

  tail +2

D'oh! I forgot that one, and gave an awk description instead.

Personally, I'd use another tool to remove the header before sending it to shuf. As an example:

    cat myfile | tail -n +2 | shuf -n 50