Hacker News Reader

Comment by teej | Hacker News Reader

I imagine the author used this as an excuse to play around with Spark. If it were me doing this for work, yeah I'd drop this in Postgres. Most of these analyses would be short SQL queries.

View on HN

I believe that was the intention of the author, write a DIY post to demonstrate usefulness of Spark, not to gain insights out of the Amazon reviews.

A bit of both. I wanted an excuse to test out Spark to find the kinks which were ommited from the documentation (and boy did I find kinks), and also provide a practical demo.

Thanks for writing this! I'm thinking about using Spark for a little 2M-data-point project that I'm working on, just for the learning experience.

Out of curiosity, what kinks did you find?

Essentially the same Spark caveats of lazy evaluation and immutability of caches: neither are a big deal on small datasets, but making a mistake on either on a large dataset can result in a lot of lost time or confusion.

Then there are the massive shuffle read/writes that result in 50GB i/o which are not great for SSDs.

lets say i tried to load up postgres and a data set on my ..fairly powerful.. laptop to run queries. how many records could i get up to? 100m? 1b? say 16gb ram

One would need to know the size of the record. This is an exercise you'll often do if you're doing capacity analysis / growth analysis for planning (or in conjunction with FP&A).

1,000,000 4kb records takes up, as you'd guess, 4GB of RAM. You can obviously go well beyond your allocation of RAM and still have the database perform, but you'll find you're now bottlenecked on the speed of IO from your SSD / HDD. Throughput will quickly decrease, queries will run slower, etc.

This is why you'll often find that DB benchmarks that never exceed RAM can be "false" comparisons if the expected workload will always exceed available memory, and data not resident to memory will need to be loaded.

So, to truly answer your question, it's actually less a question of RAM, and more a question of HDD. The 2015 ACS (American Community Survey) plus some geographic data is around 100GB, and I comfortably run analysis against it on my wimpy 2015 Macbook (8GB RAM, 1.3Ghz Core M).

you can load up a billion records with half that RAM but the most important part is the type of queries you want to run. in most cases even the most complex select queries are okay (as longh as you're not running 100+ in parallel). if you are running a lot of inserts/updates, you'll probably run into issues. but selects are fairly trivial as long as you have the space.

Reply on news.ycombinator.com