Comment by vgatherps | Hacker News Reader

Comment by vgatherps | Hacker News Reader

Parent Full thread

I have some experience working with such a team after coming from one of the more sophisticated shops, and have built an 'improved' tool along these lines that my own shop uses to great success. There's a few problems that occur:

* excel is a super powerful tool that lots of programmers underestimate - it's a legitimate dataflow programming tool, provides rapid feedback, has mediocre model visualization (as opposed to the none that many replacements offer), can be hooked into tones of data sources with live updating in the gui, supports all the expected copy/paste stuff, etc.

* Worse versions of what normally comes from changing workflows - 'too risky', 'too much effort', etc

* In my experience human traders go hand in hand with GIANT egos and are very unwilling to listen to anybody saying things can be made better

* A firm that finds success with human traders and excel won't reap the benefits of a proper programming model - easier simulations, better version control, reusable models, etc.

Try sending an executive some random long python model code. He will not understand it, not care about the "but it will pay off, just invest 1000 hours, trust me" argument, and finally demand an excel model.

How often did i sit with CFOs and discussed transaction models, tax models, p&ls and more. How often did they surprise me with their calculations?

Excel is the tool to go and there is a reason why EVERY SINGLE business is using it.. except some 30 FTE 100% dev startups, probably.

I'm not sure if you meant to respond to me, but the tools in my use case are use by 'quantitative' traders and modelers - not executives that have other things to do than worry about programming. Excel is absolutely NOT the tool to go here, it's usually just a sign that either the firm's edge isn't in being particularly quantitative or they are leaving most of the trade on the table. In my experience it's usually the former, since the latter are (and by now have mostly been) outcompeted by trading shops that have moved on to better tools.

I honestly can't think of a use case for excel that pandas doesn't handle better. The 1 million row limit is crazy limiting, it's just nicer to not have to worry about that kind of stuff.

There isn’t a 1M row limit. There is an entire backend data model that can hook into any data source for the most part and can be written in a functional language.

Pandas just extends that 1M row limit to ~10M row limit (whatever fits in your RAM)

And whenever I work with pandas I run into pesky bugs with not handling integers properly.

My answer to that, for both pandas and excel work is to do as much of the work in SQL/views as possible.

could you elaborate more on the pandas bugs?

I think it goes something like this like this (from the top of my head):

For historical reasons pandas did not handle NULL in integer columns. Which happens regularly in SQL or when you import csv:s. And that was always a pain, but a known pain. If you have NULL you will have to treat it as a float.

Up until recently when they introduced a new type called Int64 (distinguishable from the original int64 by the capital "I"). Unfortunately that uncovered new bugs, in particular for plotnine (ggplot for python) which is my favorite plotting library. Seems like they didn't want to handle Int64 types. Or at least not yet.

And your sibling post also gives a hint about the confusion. She also mentions seaborn which is another alternative plotting library which apparently also doesnt work.

I've been trying to use pd.NA to embrace the future, but it runs into a nightmare of other bugs - float vs Float64 mixing bugs and stuff like seaborn that doesn't handle Float64 anyway etc.

As an example of the sharp stuff that causes bugs.

If the job is to model financial statements to produce a forecast model, to answer questions like

* what cash flows can we expect from this business

* what will the profit be in 12 months

* what will that do to my balance sheet

... then Excel is still the best tool. Financial statements are usually less than 100 rows each. All the modelling is on a tab, out front where you can print it and sense check it with a highlighter. People on hn often think towards the big data world ( and like the other poster I would exhaust sql before anything else for ad hoc analysis), but there is a huge amount of work done that is more like my use case than yours.

Reply on news.ycombinator.com