Hacker News Reader

Comment by infecto | Hacker News Reader

While definitely doable, using a SQL database to do data discovery is obtuse.

It's obtuse, but it's effective for some people. I know someone who hires devs from the .NET and Python space for finance. When asked to suggest a solution to any kind of data problem like this the interviewees split down the middle with the .NET ones almost entirely using a relational database and the Python ones using either the library du jour or suggesting something on the command line (since they often have Linux experience.)

In both parts of their org, the .NET area tends to use SQL exclusively for analysis and the Python folks use Pandas and bunch of other stuff. These departments are also in significantly different parts of their organization with their own mandates and culture.

As someone with +20 years in finance (hedge funds/trading) and knows .Net, Java, Python, C++, shell, etc, the first question I'd ask is: where's the data?

If I'm being asked to do data analysis, it's because we need the answer yesterday. So, the tool I choose always going to be a matter of which gets me the answer fastest and with the least amount of friction. That's almost always dictated by where the data is _now_. Not where it'd ideally be.

It's in this email attachment right here. Now what?

To me that sounds like a csv file and it'll likely be a python script from me.

Depends, that is what OLAP is all about, naturally most of the good tools happen to be commercial for the enterprise space.

Not really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.

Totally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.

Reply on news.ycombinator.com