Mistakes Developers Make When Using Python for Big Data Analytics | Hacker News Reader

Mistakes Developers Make When Using Python for Big Data Analytics

airpair.com

87 pointsyarapavan45 comments

In my experience, developers' top 3 mistakes in this area are:

1. Convincing themselves they have "big data" instead of just "data". If it fits in RAM on your laptop, it's definitely not "big".

2. Thinking an example of "sophisticated analytics" is "Average over time with no standard deviation".

3. Resume Driven Development over pragmatic solutions that get stuff done.

This is so true.

The problem I have though is that looking around for jobs recently, people want "experience in MongoDB", rather than "ability to recognize when MongoDB is clearly not the best tool for the job".

You could always take some Art of War inspiration and learn MongoDB really well before your interview, then once you get in there become the in-house Mongo go-to expert, then use your clout to recommend a replacement. :)

I've often wondered why people don't use databases and SQL for more often data analysis. SQL is relatively easy to learn, and there is so much you can do, especially if you use analytical functions. I almost always load the data into a DB, and do as much as I can with straight SQL. And, if I can't do it in SQL, I write a query to the transform the data to the format I need, and then access it via python to do the rest.

They do. In real companies (those with you know, revenue) millions of ordinary people called business analysts, accountants, statisticians use SQL or tools like Business Objects to do real work on multi-terabyte databases called Data Warehouses every day. Have done for years before the first hipster called himself a "data scientist".

There's more you can do with straight SQL than you'd think. Say, multiplication of sparse matrices.

I'm beginning to think that "big data" is the most unfortunate possible term.

By any reasonable definition of 'big' even a gigabyte or two counts as big. And since 32bit isn't quite one for the history books just yet, many of us still have 2-4GB fresh in our heads as a threshold where you might have to start thinking about the data needing special treatment.

There have been conversations in my job where I refer to a 40GB file as "big" and someone else will go "that's nothing." Well, obviously there are bigger files somewhere, but that doesn't mean 40GB is nothing.

"Big" is always a relative term. Anything is big in the right perspective, so to say that something "isn't big" is almost always wrong. We should really try to find a term with a more absolute meaning.

I think a rough definition might be something like "datasets big enough to cause problems with standard tools on standard hardware" - i.e. where you need to change your tools to get things done.

Still pretty vague but it captures what I think many people are getting at.

Big data does not refer only to the size of the dataset. It's variance and velocity, in addition to volume.

I was going to say something similar except for me it's a function of size and multidimensionality.

We're sitting on several PB but it's generally in two dimensions - standard analysis techniques work, we just have problem moving the data around efficiently and processing it.

I like your inclusion of velocity, I'm including that in my definition.

Not only that, most users of "big data" make assumptions about what the underlying dataset is. Not all large datasets are unstructured text you are trying to run information extraction on.

Indeed. It's amazing how impressive you can look rescuing someone from a "big data" quagmire by just loading that CSV file into a halfway decent RDBMS and letting a simple SELECT statement spin away from a couple tens of μs.

Not that I've got anything against container ships, mind you, it's just that I find they're a rather unwieldy vehicle to use for getting groceries home from the market.

5 years ago at least, I was dealing with 50Tb databases on hundreds of Tb of physical storage. We didn't call it big data, it was just data, we didn't call ourselves data scientists, we were just database guys, DBAs and devs.

Nowadays a million rows is "big data" and only a "data scientist" can handle it. These guys are a joke.

re 1 - but it sounds so sexy! I recently had a recruiter talking up a local shop that was on the "cutting edge of big data techniques". He also talked up that they had a dataset which was "several terabytes".

We churn out many times that a day and I don't consider us to be "big data".

Yep. For things that aren't "big data" (nearly everything), I've started to use in-memory SQLite databases for analysis. Just loading your data into a table makes doing all the selections/aggregations pretty simple if you're familiar with SQL and it fits in RAM.

Any particular reason you pick SQLite over more featured DB's such as MySQL or Postgres? I am assuming that they would be faster, with the only downside being more difficulty in the initial install/setup.

For a lot of "one off" tasks (like analyzing a big log file), just configuring a full blown DBMS like MySQL or Postgres would take longer than it would to write a script using an in memory SQLite DB and running the necessary queries.

I'm not sure MySQL and Postgres are faster enough to justify the extra setup work when everything fits in memory. My intuition is that they may even be slower, because they still have to write everything to disk, where SQLite can be memory only.

It's very easy to make a throwaway SQLite in-memory database for processing... not so much for a traditional RDBMS.

Mistake #0: not looking at the data.

Whether you're designing a processing pipeline or simply keeping track of what one is doing, decent visualization of inputs and outputs is critical to understanding what's going on and ensuring that nothing surprising is happening.

I'm always surprised when people come to me with data analysis problems (big or otherwise) and have never bothered to do any kind of visualization. Even if you're just visualizing sparse samples it's remarkable how easy it can be to spot issues.

Good visualization won't solve all your problems but there is a significant sub-set that go from hard to easy when you do it.

Yep. Even looking at quartiles, min, mean, and max for each of your (numeric) variables can yield a bit of insight for prettymuch no effort.

Right. Using pandas may allow you to process more data than an excel spreadsheet, but if the file is on your laptop then it's inherently not very big. It's a great tool, but it is a data exploration tool, not a data processing workflow tool.

IMHO, there are too many 'data scientists' nowadays that are taking averages and calling it 'analytics'. If the call you are making exists in a library, more than likely it's not "sophisticated".

While the engineer in my agrees, the business person says it really doesn't matter what techniques are used if he/she is providing value. If all it takes is some averaging to save a company a good chunk of money, then that person may be earning their pay.

Averaging the right things in the right way is important too--such as averages within clusters.

> If the call you are making exists in a library, more than likely it's not "sophisticated".

What a disparaging comment. The whole point of good software design is many of the algorithms you may need to use are packaged up for ease of use. The best ones are highly specified by the parameters you supply. "Sophisticated" should play no role in a data scientists workday - results should be verifiable and understandable, and any data analysis pipelines should be extensible and repeatable. Writing their own deep-learning implementation does not a good data scientist make.

I see an awful lot of #3. Hard not to laugh when I see it. Sometimes I think that describes 3/4 of the things I see Data Scientists working on.

The challenge in the field is very few folks know programming, math/stats, and the domain they're working on. It's rare to even get 2 of 3.

Regarding #1, it took a phone call and two emails (the last of which was CCed to my boss) to convince someone in another silo of the company that yes, I do need all 3.5 million rows of a particular table and that no, he wouldn't crash the network drive if he dumped it there.

If all that you use is Excel, then there are a lot of huge datasets out there.

I kind of feel like pandas and ipython are anti patterns here.

They seem super convenient but I've found that writing a lower level analysis with independent scripts linked via make or similar saves massive amounts of time in the long run.

IE the first few steps should retrieve and process the data until it is just arrays of numbers (or whatever your actual analysis needs) that can be handled with efficient numpy code.

You can use pandas for this but a real database works too. After this step pandas becomes irrelevant and just leads to lots of unnecessary allocations from recasting data on the fly.

The problem with ipython is that it leaks like a sieve and you end up with all sorts of copies of the data, worker processes and no longer used variables which will eat all of your ram even on a big cloud instance.

It is much nicer to have each analysis script start with a clean stack and release its memory when done. Plus you can use non python utilities like grep, vowaple rabbit etc as intermediate steps.

I've found practices like these significantly lower the memory requirements of analysis and allow one to tackle bigger datasets on single machines.

Each to their own though.

I find that IPython NB is a great tool for initial exploration -- to sketch out the ETL steps that are needed, what types of computations I'm going to be doing, etc. As soon as those ideas are roughed out, I switch to writing Python scripts that I can run through completely, write tests for, git commit and get reasonable diffs, and more. I'll generally keep an ipynb open in a tab for experimentation, but I'm doing all of the work in PyCharm or Spyder.

This also forces me to think about the engineering implications of the analysis a bit sooner. Inevitably these projects are not one-offs, and will need to be at least repeated regularly, if not outright productionalized -- and ipynb files do not lead to production-ready code.

I also agree that using a real database is often a better option than pandas. Many folks avoid it because it's less comfortable to spin up postgres + a new DB instance than it is to sit in the comfort of IPython, but it's totally worth it. Exploring and previewing the data via SQL is just so much faster and more intuitive than spinning it around with pandas.

And note seccess's comment about using in memory SQLite db to eliminate the need to configure postgres:

https://news.ycombinator.com/item?id=8930488

Why anti-patterns?

Pandas makes a great companion to a database. Aggregate / reduce large datasets (>RAM) efficiently in a DB, then pivot in Pandas, and perform non-SQL-support analyses in Python. Same can be said for R.

iPython is just a shell, a very good one, with options to share work for presentation and reproducibility.

Because they drive memory usage up a ridiculous amount making people think they need big data solutions when they don't.

They are great for some things but I think the idea they should always be used is unfortunately prevalent making them anti patterns.

> the idea they should always be used is unfortunately prevalent making them anti patterns

Yes, this is often too true - also to include Hadoop, NoSQL, ...

Even when not using pandas, do yourself a favour and use csv.DictReader/DictWriter, as csv format has quite some quirks.

https://docs.python.org/3/library/csv.html#csv.DictReader

Yes. Never use split(",") or equivalents to read or write a CSV, always use a library. Otherwise you'll get a nasty surprise next time your code encounters a line or row separator in a value.

The csv module has it's fair share of quirks to be aware of as well, like not supporting Unicode.

Even in Python 3?

In Python3 it's completely fine, if you open the files with the correct encoding. For Python2, you've got to use UnicodeReader/Writer (and to open in binary mode): https://docs.python.org/2/library/csv.html?highlight=csv#exa...

...which is why I usually turn to this: https://pypi.python.org/pypi/unicodecsv/0.9.4

It's a "drop-in replacement for Python 2's csv module which supports unicode strings without a hassle."

Great points about data provenance and sanity checking at the right level. This is all valid for any kind of data and not just python stuff.

> Doing the task in vanilla Python does have the advantage of not needing to load the whole file in memory - however, pandas does things behind the scenes to optimize I/O and performance.

There's a neat way around this, just set iterator=True or chunksize and pandas'll return you an iterable TextFileReader object: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunk...

Another tip is that even before you get into cythonizing stuff, see how much of the computation you can push down into high performance libraries like numpy - use a map() with a vectorized numpy function, and store your stuff in a numpy data structure instead of a manual for loop on a regular array-of-arrays, etc.

Sidenote, is there a decorator version of cythonmagic? Basically I want to annotate my functions with it since sometimes the non-typed basic cython is still much faster than the pure python version, and I don't have to manage the compilation step.

I would agree most strongly with number 6. Having spent time in the biotech arena, there are so many fractured data format structures. A good example of this would be small variations in .fastq files. There are different formats generated from different equipment, and different software expects format A when you have format B, and I've seen people trying to shove the wrong formats into pipelines that blow up or create a tone of work as a result.

Honestly though, the biggest problem with people using python for bid data though is just the opposite of number 2, sometimes you need to use that framework, but when you rely too heavily on various frameworks, that's how you end up in dependency hell, largely due to the aforementioned format issues. eg. Framework 1 update fixes bug X but creats bug y in framework 2's parsing.

Your "bid data" typo seems like a nice little Freudian slip!

As far as your second point though: although it's not ideal, don't virtualenvs do the trick? If you really needed to, you could even set up a workflow of different sandboxes to pass data through. If the alternative to relying on frameworks is rolling your own, frankly I would probably choose the former, but that's just me.

Good point on virtualenvs, but it's a python specific fix (which does work). The problem is often you are tying in other kinds of tooling.

The other thing to remember about roll your own is you can get efficiency that frameworks can't, and that time difference adds up fast. For example, we designed a worker distribution system, rolled our own worker management code on top of a framework, and reduced compute times from ~1-2days to ~4 hours. That's a huge increase in productivity that no tool or framework could give us.

There is power in roll your own, I would just say whip your programmers into submission regarding good commenting/documentation though.

Nothing to do with the mythical concept of "Big Data" here.

Not necessarily a bad post, but the "Big Data" title misleading.

big data (noun) 1. another name for data warehousing.

Reply on news.ycombinator.com