Comment by RayVR | Hacker News Reader

Comment by RayVR | Hacker News Reader

this is a really weird competition.

I know of many desks still using excel to run/update models which are almost always consumed by human traders.

This is a very antiquated and generally fragile system that has a very steep learning curve with very expensive maintenance in the form of what are often called "Junior Portfolio Manager" or "Analyst" being mostly responsible for waking up before the rest of the team to log in, make sure the spreadsheets were/are updating, and fix any issues.

The "scheduling" for these systems is usually a manual start or in more sophisticated teams, a scheduled job on a windows desktop (which fails ~30% of the time for unknown reasons)

I personally know several people that started this way, survived for 3-5 years and then moved to larger/more successful teams or companies that had the resources and knowledge to actually solve these problems.

Anyone thinking about going this route would be better served by picking up some basic Python.

I have some experience working with such a team after coming from one of the more sophisticated shops, and have built an 'improved' tool along these lines that my own shop uses to great success. There's a few problems that occur:

* excel is a super powerful tool that lots of programmers underestimate - it's a legitimate dataflow programming tool, provides rapid feedback, has mediocre model visualization (as opposed to the none that many replacements offer), can be hooked into tones of data sources with live updating in the gui, supports all the expected copy/paste stuff, etc.

* Worse versions of what normally comes from changing workflows - 'too risky', 'too much effort', etc

* In my experience human traders go hand in hand with GIANT egos and are very unwilling to listen to anybody saying things can be made better

* A firm that finds success with human traders and excel won't reap the benefits of a proper programming model - easier simulations, better version control, reusable models, etc.

Try sending an executive some random long python model code. He will not understand it, not care about the "but it will pay off, just invest 1000 hours, trust me" argument, and finally demand an excel model.

How often did i sit with CFOs and discussed transaction models, tax models, p&ls and more. How often did they surprise me with their calculations?

Excel is the tool to go and there is a reason why EVERY SINGLE business is using it.. except some 30 FTE 100% dev startups, probably.

I'm not sure if you meant to respond to me, but the tools in my use case are use by 'quantitative' traders and modelers - not executives that have other things to do than worry about programming. Excel is absolutely NOT the tool to go here, it's usually just a sign that either the firm's edge isn't in being particularly quantitative or they are leaving most of the trade on the table. In my experience it's usually the former, since the latter are (and by now have mostly been) outcompeted by trading shops that have moved on to better tools.

I honestly can't think of a use case for excel that pandas doesn't handle better. The 1 million row limit is crazy limiting, it's just nicer to not have to worry about that kind of stuff.

There isn’t a 1M row limit. There is an entire backend data model that can hook into any data source for the most part and can be written in a functional language.

Pandas just extends that 1M row limit to ~10M row limit (whatever fits in your RAM)

And whenever I work with pandas I run into pesky bugs with not handling integers properly.

My answer to that, for both pandas and excel work is to do as much of the work in SQL/views as possible.

could you elaborate more on the pandas bugs?

I think it goes something like this like this (from the top of my head):

For historical reasons pandas did not handle NULL in integer columns. Which happens regularly in SQL or when you import csv:s. And that was always a pain, but a known pain. If you have NULL you will have to treat it as a float.

Up until recently when they introduced a new type called Int64 (distinguishable from the original int64 by the capital "I"). Unfortunately that uncovered new bugs, in particular for plotnine (ggplot for python) which is my favorite plotting library. Seems like they didn't want to handle Int64 types. Or at least not yet.

And your sibling post also gives a hint about the confusion. She also mentions seaborn which is another alternative plotting library which apparently also doesnt work.

I've been trying to use pd.NA to embrace the future, but it runs into a nightmare of other bugs - float vs Float64 mixing bugs and stuff like seaborn that doesn't handle Float64 anyway etc.

As an example of the sharp stuff that causes bugs.

If the job is to model financial statements to produce a forecast model, to answer questions like

* what cash flows can we expect from this business

* what will the profit be in 12 months

* what will that do to my balance sheet

... then Excel is still the best tool. Financial statements are usually less than 100 rows each. All the modelling is on a tab, out front where you can print it and sense check it with a highlighter. People on hn often think towards the big data world ( and like the other poster I would exhaust sql before anything else for ad hoc analysis), but there is a huge amount of work done that is more like my use case than yours.

If Excel is the Old Way, what’s the current state of the art for GUI software to do these kinds of operations and analysis? Is there an equivalent of the modern ELK stack, but for numerical analysis?

Not everyone can, will, or should have to learn Python.

Excel is superior to Python in so many ways for many use cases. It is visual (and this already kills the competition. Forget leading three executive through the final m&a python model), you can build small models in seconds or minutes, the learning curve is flatter and hence it reaches more people, auditing is much easier, tiny tweaks can be made much quicker (especially for people who did not setup the model) and it does not require years of studying coding.. i put our interns infront of excel and within 2-3 weeks of hard work they can do their calculations. Impossible for regular business and finance guys if they would have to so this all in python.

Have you used Power BI? I haven’t, and I’m not sure they’re comparable, but at least Microsoft isn’t resting on their laurels when it comes to productivity software for analysis use cases.

Yes, ofc i used power bi. It's great, but it cannot do those things that excel can and vice versa.

They complement each other rather than replace.

Do you think Excel will gain tighter integrations with scripting? I read about Excel getting better scripting support but I forgot what all that entailed and what languages were supported.

Thanks for your perspective, by the way. I don’t use Excel much for higher order stuff, but I want to progress past pivot tables.

Not sure what exactly you mean by scripting support. I used to know a girl who was working in big consultancy, and her team was making sales predictions for huge global brands, in Excel. She was writing some frightening amounts of code in Visual Basic, up to the point when I thought they used Excel just like a fancy IDE for their VB, which was the actual work. Who needs Excel formulas when you have VB, right? It was very philosophical for me to hang out with her sometimes, because we liked to discuss our jobs and all that time she denied when I told her she was a programmer.

Excel on the web has different scripting features and functionality than the desktop version, called Office Scripts, and Power Automate integrations. It also has something called Action Recorder, which seems like a native built-in AutoHotKey type deal.

https://docs.microsoft.com/en-us/office/dev/scripts/overview...

Reply on news.ycombinator.com