Ask HN: What do you use to open large CSV files?
As a data analyst in a large organization, we still deal with large Excel files - wondering what the HN analytics community uses to open these large files?
Or you could load into the Excel data model instead of the grid, look in the Get & Transform feature. Then do your initial exploration there. Data model (formerly Power Pivot ) is very underrated IMHO.
What I do is dump it as CSV again after manipulation and then bulk load it into the DB using the DB's native tool. It's messy but 10x faster.
If you can't do whatever you'd like one line at a time, read the file as a stream and write it to disk with some database(e.g sqlite, MySQL, Mongo). This will be slower but might be easier to play with the data.