Hacker News Reader

Comment by moonshinefe | Hacker News Reader

The killer feature for me with this app is the stupidly easy CSV loading. I deal with huge CSVs at work often. SQLite Browser is the easiest, fastest and most lightweight UI tool to analyze them I've tried. It can load millions of records into a table in like <3 seconds. Then obviously you can query the CSV data via SQL.

View on HN

Thanks for letting me know about this! I do the same thing... DuckDB loads CSVs very quick and easy. Malloy uses duckDB and can work in VScode or vscode.dev or the vscode built into github. So, you can analyze large .csv files in any of those places.

For example, go here [1]. Press "." (this loads vscode in browser) and you will be analyzing large .csv or .parquet files directly in your browser, after you click on the .malloynb file

Something I made to analyze every form submitted to the Securities and Exchange Commission (SEC) (10-K etc.) since 2014. [2] Go there, press ".", install the malloy extension, and click on the .malloynb file! Do it all within your browser, or download it and use it in VScode.

[1] https://github.dev/malloydata/quickstart/blob/main/README [2] https://github.dev/mrtimo/isa5

How do you feel about Malloy vs PRQL?

I think I would miss the the semantic data modeling of malloy, and Malloy is definitely easier to read and write IMHO, in part because of the semantic data modeling.

I would also miss the nest feature of Malloy, and the way it handles time / dates / timestamps.

One of the senior devs (Martin) spent a loooot of time optimising the CSV import process. Many different approaches using sample data sets (from rough memory) of between 800MB and a few GB I think.

I've noticed throughout the years SQLite has lots of handy features built in for dealing with CSV and other input and output formats. Sometimes it's easier to get data into it and use SQL to slice and dice that data than it is to pipe a bunch of unix shell commands together that really only understand unstructured text. Discoverability and subsequently trying to remember all shell incantations is a bit of an issue though, so many people don't see SQLite as the Swiss army knife it can be.

You might want to give VisiData a look: https://visidata.org

It's a TUI program, so if you want a full-featured GUI, it might not be up your alley, but it's the best tool I've ever used for doing ad-hoc work with large CSVs.

I think this feature should be included in more structured database tools. Being able to load a CSV and query against it, either to an in-memory sqlite or a temp directory (depending on size) is a great option for this kind of work.

I tried the non-installer exe version (32 &64bits) because that csv loading sounded very interesting, but cant make it work without installation because sqlite3.dll, qt5network.dll, qt5xml.dll nor qt5printsupport.dll can't be found.

I don't see the point of providing a standalone exe version that does not work?

Well, back to excel for csv :/

> I don't see the point of providing a standalone exe version that does not work?

Which download did you use?

For windows there's really only an .msi which installs everything, and a .zip which needs to be unzipped and have the executable file run from inside the uncompressed directory.

The files it's complaining about missing should next to the .exe inside the unpacked directory. If you move the .exe elsewhere, or just extract it by itself, it's not going to work.

All of those DLLs are included in the same zip file as the portable exe. Sounds like you may be trying to run it without extracting first.

I can see them but somehow it does not find it being next to it.

Don't know if it is a windows policy I am not aware.

How are you unzipping the .zip before running the .exe?

It sort of sounds like you're double clicking on the zip? I'm not personally sure if that unzips the whole folder in the right way or not.

If that's what you're doing, maybe right click on the .zip and extract everything using that first?

Note that I don't personally use windows much any more, so my windows knowledge is getting rusty for some stuff. ;)

Reply on news.ycombinator.com