Hacker News Reader

We recently ran a large clustering job over billions of records (and a few TBs of data on spinning disks) on a single machine with minimal command line tooling in a few hours. Not really optimized yet. People forget how fast modern hardware is and overestimate how much useful data they have (or need).

I think I should start a company around minimalistic data tooling or the like - the amount of waste seems large across the industry.

I saw the de-skilling a few year ago, where a guy stiched together a compete application from a couple of SaaS APIs. Cool, but it somehow does not impress me.

Bare metal can be incredibly fast, if you can get access for a reasonable price. Virtualization is becoming a continuum but the overhead is always there.

Great point, r5.metal instances have 96 CPUs and 768 GB of RAM. Lots of "big data problems" can actually be solved with a single big EC2 instance. Cluster computing should always be avoided when a single node will do.

We had something like 24G of RAM, but we have 500GB RAM machines as well (we own the hardware) and the job would have been even more of a breeze there.

> People forget how fast modern hardware is and overestimate how much useful data they have (or need).

Where your model breaks down is when you have folks throughout your engineering org who have data needs but don't have folks to spin up bespoke pipelines like this and spend time optimizing them. You have data, you want to write something SQL-ish or have some nice APIs, and you want to get results dumped into a predictable place. When you have mixed workloads, mixed data sources, and those jobs are being tweaked and changed frequently, the actual underlying compute cost is hardly the issue. Getting the data, chewing on it [fast enough] without having to spend much time optimizing, getting it to its destination, and making that happen regularly and reliably is where the value is.

> * AWS has a managed Spark offering called EMR

There is also my rinky-dink open source project, Flintrock [0], that will launch open source Spark clusters on AWS for you.

It's probably not the right tool for production use (and you would be right to wonder why Flintrock exists when we have EMR [1]), but I know of several companies that have used Flintrock at one point or other in production at large scale (like, 400+ node clusters).

[0]: https://github.com/nchammas/flintrock

[1]: https://github.com/nchammas/flintrock#why-build-flintrock-wh...

The last point isn’t so dramatic. There is version control for notebooks, and better version is coming (right now it’s in preview - code name - Projects, it’s in official docs). Tests are possible - either Nutter from Microsoft, or home grown (about 20-30 lines on top of built in unittest). Deployment is also not so complicated - there are tools (provider for terraform, cicd-templates project, Databricks-cli, etc), you can refer MS Learn for short course about CI/CD for Databricks and Azure DevOps, etc...

The last point was for teams that only rely on notebooks, sorry if I didn't make that clear.

You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files.

Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6

I’m sorry for delay, will fix ASAP...

My point is that you can do that even without jars/wheels - you can do VC and tests of notebooks. For example, https://github.com/alexott/databricks-nutter-projects-demo

As alexott says, you can link notebooks to a git repository and commit and rollback them from the Databricks UI (https://docs.databricks.com/notebooks/github-version-control...)

Problem is that it’s limited. You can’t, for example, commit multiple files in one go (that improves a little bit in https://docs.databricks.com/projects.html), or merge changes a colleague made with your changes. You also have to use the UI Databricks provides. You can’t use a git CLI or whatever GUI you prefer. (all AFAIK, but I’m fairly certain about it)

> Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management)

The last three of those things might be valid issues, but since when are notebooks not just as subject as any other source code format to version control (I get that the difference between the UI and the on-site structure may make typical diff tools less-than-ideal, but VC itself is unaffected.)

In my experience the use of notebooks (exclusively) goes hand in hand with not knowing things such as version control, tests, deployment process or dependency management exist.

I don't mean to sound harsh about other Data Scientists from a non software engineering background but the standard workflow is to fiddle around with a notebook until you can get a result. That's as far as it goes, no real robustness to it.

That's a pretty big generalisation but in organisations where they "home grow" their Data Science capability many of the online courses don't cover production level Data Science.

Your experience aligns with what I've seen.

All the notebooks are in one place. Some are for important production jobs, other are for data exploration.

It's easy to make a little edit in a notebook and accidentally break production jobs.

Comparatively harder to make an edit in a git repo and do a deploy that'll break production jobs (e.g. if the JAR doesn't compile or the CI errors out cause the tests don't pass).

Notebook based production jobs get even more dangerous when NotebookA depends on NotebookB and so on.

Treat notebooks like other code - separate onto staging and production, with defined promotions between them - it’s possible. You can run tests in CI/CD pipelines, etc. You can set permissions so nobody can update production notebooks manually, ...

> The difference between the PySpark & Scala APIs confuse the hell out of people

I personally had this experience when first wanting to learn spark, and it really turned me off to the whole spark ecosystem. Curious if you have any suggested resources that do a good job on this?

Spark offers Scala, Python, Java, and R APIs. Scala & Python are the most viable options (R lacks a lot of features and Java is only good for ppl that love Java).

Scala & PySpark are both great options. Lots of devs are terrified of Scala, so PySpark is more popular now. I'd say Scala has a slight technical advantage, see here for more details: https://mungingdata.com/apache-spark/python-pyspark-scala-wh.... Both are great overall.

I wrote a book that's a practical introduction to Spark: https://leanpub.com/beautiful-spark/

Most of the training materials are theoretical, which makes Spark seem really intimidating. You can learn some basic Spark principles and get up-and-running with production workflows quickly.