Show HN: An app to split CSV into multiple files to avoid Excel's 1M row limit

Why is this a yearly cost? I don’t understand.

There’s already open source utilities [0] for users who aren’t proficient in Unix commands.

If anything this should be the definition of a one time fee.

You’re free to charge whatever you like, but it seems odd that anyone would pay you year after year to use your app.

It’s not the price, as $40 isn’t that much, but the value and principle of the thing.

[0] https://github.com/philoushka/LargeFileSplitter

The video (why?) doesn’t load for me, so I can’t check features, but what’s wrong with split and csplit? (https://www.gnu.org/software/coreutils/manual/html_node/spli..., https://man.openbsd.org/split.1)

TL/DR: xsv is probably what you want, or maybe zsv and/or awk

awk can do this super easily. Here's an example snippet that not only shards, but compresses your shards. ``` (NR - 1) % shard_size == 0 { # ready to start a new shard current_n = current_n + 1 output_file = sprintf("%s%04d.%s.bz2", target_prefix, current_n, file_type) print "writing to " output_file > "/dev/stderr"

    # close any prior-opened output_command (else will err on too many open files)
    if(output_command != "")
        close(output_command)
    output_command = "bzip2 > " output_file

    # print header
    print headrow | output_command

}

NR != 1 { print $0 | output_command }

```

This of course assumes that each line is a single record, so you'll need some preprocessing if your CSV might contain embedded line-ends. For the preprocessing, you can use something like the `2tsv` or `select -e ...` command of https://github.com/liquidaty/zsv (disclaimer: I'm its author) to ensure each record is a single line.

You can also use something like `xsv split` (see https://lib.rs/crates/xsv) which frankly is probably your best option as of today (though zsv will be getting its own shard command soon)

This site probably isn't your target audience....

I think $40 a year just for the split feature is a little too much. However, adding more useful features for manipulating CSV files would probably change my mind about it. For example, doing some reorder or preprocess of the files as if people split into X files they'll have to do that action X times if they do it in Excel directly.

I got smacked with this just last week.

My answer was just import into MS Access.

Note: if you're pulling in a lot of S3 list_objects_v2() data, and have some honking big object sizes, the Long Integer type craps out at representing a 2Gb file. You need to use Double.

It's interesting how people use and abuse Excel. How many people try to use Excel to process millions of rows? Time to use tools that can directly query csv files to aggregate the data.

The answer is actually PowerPivot. Access you can do a few GB, but with PowerPivot, you can do a billion or more rows. Don’t expect any insane performance though.

Do people really have CSV files with more than 1M rows? How do you search and update such files? Why not use some database?

so this will be shared again with no updates in a few months? https://news.ycombinator.com/from?site=superintendent.app