Hacker News Reader

Comment by Bishonen88 | Hacker News Reader

A very gentle intro, indeed ;)

Pandas is such a vast monster, that even after going through the book from the original author of Pandas (https://www.safaribooksonline.com/library/view/python-for-da...), I was absolutely unprepared for doing real analysis.

Whilst I understood the basics, such as data loading, (simple) cleaning, selections, functions, groupby', indexes etc., I spent most of my time on stackoverflow looking for solution to actual problems I was facing. I reckon that many other users have made the same experience - there is lot's of general info out there when it comes to pandas, but every data is different and the devil lies in the details. Long Story Short: learning pandas is all about trial-and-error and will take months (years even), to be efficient in.

View on HN

As a daily user of pandas for a few years now, I really must suggest that anyone looking to use it for serious data analysis familiarize themselves with the Split/Apply/Combine paradigm [0].

Lots of data munging has been enabled or sped up by judicious application those concepts.

[0] https://pandas.pydata.org/pandas-docs/stable/groupby.html

I agree. Hadley Wickham (a very prolific author of important R libraries) wrote a great paper about this method using one of his libraries. I'm a Python + pandas user, but his paper really helped me understand the approach better: https://vita.had.co.nz/papers/plyr.pdf

It might take months to be be able to use its full power and be efficient (delivering at a high velocity), but as someone who was proficient in python but had never used pandas it took me just over a week to write a process to clean data and produce graphs (with seaborn) to compare sets with boxplots. This incldes anonymizing the data properly and play with different Turkey's fence values for the graphs to make the most sense. This is after people spent weeks, and failed, to try to get a similar process with excel.

It's good to warn people, but let's not scare them.

Serious question: I've tried to use Pandas for some data analysis for my small business. Data sets are on the order of 10,000 data points or less. After struggling for days with Pandas, I've begun to wonder if it wouldn't be easier to code the analyses in raw Python. I wouldn't mind taking longer to complete the task at hand if in the process I was acquiring skills that will pay off down the road, but I wonder if Pandas isn't so esoteric and difficult that I may never reach the point that I can cash in that investment of time and effort as long as I am only a casual user.

In contrast, while I'm not an expert in JS or Python, I find that time spent struggling with those technologies pays dividends since the lessons learned make everything I do in the future easier.

This is highly subjective of course, but in your opinion, should I keep fighting with Pandas? Is it worth it?

Make sure you learn what a Series is and how it relates to the things in the DataFrame and how selection works, specifically .loc and .iloc. Then your life will be much easier. Try starting with this article: https://medium.com/dunder-data/selecting-subsets-of-data-in-...

10,000 data points is well within the range of what Excel can handle without needing PowerQuery. What type of analyses are you attempting?

Linear optimization and generating some simple graphs. I would like to be able to at least generate the graphs automatically from my database.

> Linear optimization

Did you mean linear regression? Linear optimization isn't a use case that Pandas covers, but there are other tools that I can recommend.

Yes, that's what I meant.

I would recommend to "keep fighting with Pandas". Many of its feature seem confusing at first, but later on you see the value of them.

I think some of the problem is that people who use pandas don't necessarily know how to drop one level in programming. Like, if the file isn't a nicely formatted csv, they don't know how to read and parse the file directly. If they can't use basic filtering or a boolean mask easily with pandas, they don't know how to use lists, loops, and conditionals directly. It's great to use pandas rather than reinventing the wheel, pandas is an excellent library, but if you're going to deal with data at a very intricate level, you do need to know how and when to punt and just write the code yourself.

I think this comes up particularly in the context of pandas, because it's a common entry point into a programming language for people who don't think of themselves as programmers and may resist the notion that this is actually what they're doing.

I still see myself going to spark and Spark SQL for some tasks like stratified sampling which I haven't been able to properly do with pandas. Somehow the spark DF API feels more intuitive and I was able to figure out a lot by myself.

Can you use groupby for your stratified sampling work?

That's interesting as the Spark api was inspired by Pandas

Reply on news.ycombinator.com