Ask HN: Recommended language to learn data manipulation | Hacker News Reader

Ask HN: Recommended language to learn data manipulation

10 pointsdigdugdirk9 comments

I have a few very large datasets (in csv format) that I need to work with that completely overwhelm Excel. I'm looking for recommendations on a single language (or minimal number of languages) that I could learn in order to take these datasets in and start running various calculations on.

Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria.

I'm not necessarily looking for a how-to book (though I wouldn't turn it down) I'm really just looking for language advice. There's a lot out there on the Internet, and much of the advice is conflicting.

View on HN

https://www.r-bloggers.com/handling-large-datasets-in-r/

>Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria.

Given this criteria, I think SQL would be a good choice: simple to get started with, useful in the future.

SQLite doesn't get in your way.. https://sqlite.org/

Python Pandas is absolutely amazing for this. Have a look at the cookbook and see how you feel about the language: http://pandas.pydata.org/pandas-docs/stable/cookbook.html

I also did a presentation/workshop on data analysis with Python you might enjoy here: https://github.com/jackgolding/FullStackDataAnalysis

Any language is suitable for data processing, but if you want something which is written especially for data processing and analysis, and especially data processing and analysis for large scala data sets, I personally would go for R: https://www.r-project.org/.

Working with CSVs in Python is pretty simple. It also has plenty of libraries for deeper processing depending on what exactly you want to accomplish.

https://docs.python.org/3/library/csv.html

What do you mean by large? Probably as your data is in csv format it is small enough that python or R are both good choices. Personally I'd recommend python as it has a large community using it so googling questions about python libraries such as pandas usually provides good answers.

Will it fit in ram? Python + pandas. Pandas is really amazing.

Otherwise, try postgresql or SQLite.

SQL. PostgreSQL has decent CSV support.

100% agree. I've loaded multi-gigabyte datasets into Postgres on my laptop and SQL is a great language for manipulating and aggregating data. Plus, when you use a database and SQL you can access that data from a wide range of applications and languages via drivers (JDBC and ODBC).

You can even access Postgres from Excel!

Reply on news.ycombinator.com