Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by buzzkillington | Hacker News Reader

Parent Full thread

buzzkillington·

Spark is worse because you need Scala as well as regular Java.

I've tried building it for my day job, I would rather have a colonoscopy without sedation.

It's more pleasant and dignified.

View on HN

And then add PySpark on top of that. Couldn't leave my last job fast enough when they decided to use Hadoop/PySpark when the largest incoming files we received were at most a few GBs.

I once had a consulting gig where the customer desperately wanted to build a Spark/Scala ML pipeline, for a dataset that was 10 MB. We spent 3 months hammering it together for a flat Python process that would've taken us 2 weeks.

If you'd sent it off to mechanical Turk it would have been done in an afternoon.

> This find xargs mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

I build it in a container for work and didn't find it that difficult to be honest. And Google has plenty of example Dockerfiles that show the steps needed.

The only real system dependencies are Java8, maven and texlive (and Python/R if you build for that). Then it's `make-distribution.sh` with the appropriate flags. Scala and everything else that is needed is downloaded by maven. The resulting directory is self-contained assuming you have java8 runtime on your target machine.

FWIW, the below linked Dockerfile will download, build and install Spark in a single step.

https://gist.github.com/Mister-Meeseeks/1ebf875b6e1262449cbc...

Reply on news.ycombinator.com