Hacker News Reader

Comment by fredley | Hacker News Reader

Django/Jupyter/pandas is an ecosystem that's hard to beat.

If you're not already using it, check out `django-extensions`. You can use `./manage.py shell_plus --notebook` to launch a Django-friendly Jupyter notebook.

From there you can prototype with extreme ease and confidence.

I have also built a trivial custom model Manager so I can do stuff like:

    User.objects.all().to_dataframe("first_name", "last_name", "email")

Honestly I can't see myself being even slightly tempted to move to something else unless I can replicate this workflow.

View on HN

Learning that I could just run pandas.DataFrame(User.objects.all().values()) in a jupyter notebook vastly improved my QoL in some projects.

My god... I can't tell you how happy that just made made me. I've been using runserver_plus for code reloading for a while, but hadn't realised this could do the same for the shell. Thanks!

Wow that looks powerful. I have a YOLO / openCV project I need a web app built for so users can log in and download preconfigurations... was thinking elixir/phoenix framework but i'm second guessing the productivity hit i'd take due to lack of SO questions, and mature libraries.

Not a data engineer / scientist here.

What am I missing with data-frames?

I've written products that utilized them before (years ago), but it's been long enough that I don't know what I would even use them for in daily programming.

They're for tables and doing table-ie things: an imperative programming method for the same tasks people use Excel or SQL for.

This dovetails nicely with most data science-ie tasks, like statistical analysis, engineering features, simple machine learning techniques (the tables can be easily transformed into numpy arrays), and plotting results.

You can use them any time you feel the need to do table manipulations on non-insanely-large data: aggregate, transform, combine, split, reshape, join...

Got it, I was aware of this and many of the performance benefits another user mentioned, but wasn't sure how it could be useful in daily operation or quick Django shell_plus style scripting (one-off commands) when inspecting something like a User's table, as the original comment above indicated.

From NumPy, you get fast (vectorized) operations on large multidimensional arrays of numbers. This bit not be so useful outside of a data-science context but it depends on what it is you _are_ doing. Most graphics stuff is also arrays of numbers, for example. Also lots of useful indexing/slicing/masking/filtering on those structures. You can also have non-numeric multidimensional arrays, but the performance part might not be there depending on the type.

Pandas adds utility methods like join, groupby, and fromsql. Also hierarchical indexing, and a bunch of other stuff that I don't know much about - I'm not in data-science either, but I work with some of them.

Overall, lots of useful stuff for processing tables locally once you've fetched them from SQL or similar. Also, the simple fact of having a tabular data structure without needing to define a custom Row class. Lists of dicts or tuples have a lot of overhead by comparison.

pandas.read_sql('select first_name, last_name, email from users', conn)

this approach avoids the full data copy calling '.all()' materializes.

.all() doesn't actually do anything, and could be omitted. The beauty of this setup is if you need to rewrite the implementation of .to_dataframe() to optimize it, you can (e.g. to remove the .values() you're worries about and use your above code)

You just discovered repl driven development that lisp users have been advocating for years. Check it out in Clojure: https://vvvvalvalval.github.io/posts/what-makes-a-good-repl....

Well, python has had repl driven development for years too, notebooks only change the tooling around that if I’m not wrong.

Agreed, the GUI repl may be more friendly to data scientists and analysts who know some coding but not emacs.

Furthermore, the marriage of application framework / application model wrappers with industry-leading data science tools sounds particularly special, and orthogonal to the presence of a repl.

Yes, Python does have a repl, but I've never seen anyone doing real "repl driven development" in Python in my 15+ years of using it (btw, just testing some code in the repl isn't anywhere near repl driven development). See the blog post linked above to understand what I'm talking about.

In Python, the repl is an accessory that helps testing small portions of code. In Clojure, the repl augments the editor, becoming an integral part of the development process.

My understanding of the parent argument was that the combo Django/Jupyter/pandas is "an ecosystem that's hard to beat" because of the tight feedback loop between writing the code and seeing what it does.

Ive never heard the term repl driven development, but I have been doing most of my coding in a jupyter/ipython notebook for around 7 years now and have been advocating it the whole time. The longer I do this the more I realize that I really need to try lisp

Awesome! That's exactly why I pointed out Lisp. Those who enjoy the tight feedback loop of Jupyter might also enjoy a language where this process is the natural way of doing things.

I think ipython/jupyter notebooks are more analogous to Clojure's REBL[0][1], which is very cool but not nearly as mature.

[0]: https://github.com/cognitect-labs/REBL-distro

[1]: https://www.youtube.com/watch?v=c52QhiXsmyI

Thanks for the link, I'd never heard of REBL. Very cool!

Reply on news.ycombinator.com