TL/DR: xsv is probably what you want, or maybe zsv and/or awk

awk can do this super easily. Here's an example snippet that not only shards, but compresses your shards. ``` (NR - 1) % shard_size == 0 { # ready to start a new shard current_n = current_n + 1 output_file = sprintf("%s%04d.%s.bz2", target_prefix, current_n, file_type) print "writing to " output_file > "/dev/stderr"

    # close any prior-opened output_command (else will err on too many open files)
    if(output_command != "")
        close(output_command)
    output_command = "bzip2 > " output_file

    # print header
    print headrow | output_command

}

NR != 1 { print $0 | output_command }

```

This of course assumes that each line is a single record, so you'll need some preprocessing if your CSV might contain embedded line-ends. For the preprocessing, you can use something like the `2tsv` or `select -e ...` command of https://github.com/liquidaty/zsv (disclaimer: I'm its author) to ensure each record is a single line.

You can also use something like `xsv split` (see https://lib.rs/crates/xsv) which frankly is probably your best option as of today (though zsv will be getting its own shard command soon)