1,464 karma · joined August 19, 2017
Good thing Pyoneer generates test cases for the formulas it generates! No need to trust my spelling abilities -- your Excel file is the ultimate source of truth.
The devs we've worked with so far have the goal of replacing the Excel process - inheriting it from the team that runs it manually, and automating it fully in Python. From them on, changes to the process would run through a more traditional software-development lifecycle, as you would be editing code.
For these devs - this is a feature not a bug! In Excel, version control, testing, and review is pretty much non-existent...
Cool username btw...
We're still working on Mito - it's not a retired product by any means. Pyoneer is just another stab at the same problem for a different user group.
MMR-wise, we scaled to profitability. PMF-wise, we have not reached this. We have large customers who make up the bulk of our revenue who love the product, and use it quite effectively as the basis for their entire Python program, but, transparently, scaling is hard!
> Is it the trap of dogfooding (building the thing you wish you had) without sizing the market?
Very possibly. But I think this is a much bigger pain point at large orgs with legacy processes than you realize, though. Every large bank has an entire development teams that are tasked with transitioning legacy processes out of spreadsheets. We're aiming to improve the efficiency of these developers dramatically.
For some of the spreadsheets I've personally automated, I think this would take 300 hours of work and make it like 5...
I will say: we are considering some more approaches from traditional compilation / transpilation. I think these are very compatible!
Consider the following (pretty easy) translation of a simple table:
# This is not a useful or usable artifact; you're still trapped in Excel
# but things are just worse now, since it's not on a 2d grid
A1 = "Prices"
A2 = 1
A3 = 2
B1 = "With Tax"
B2 = SUM(A1, 10) * 1.3
B3 = SUM(A2, 10) * 1.3
To make it really concrete: you can't just transpile a SUMIF or VLOOKUP to a Python implementation of SUMIF or VLOOKUP, and you absolutely can't do this on a cell-by-cell basis.
Rather: we're trying to generate a Python script that appears to be written by an expert developer. To do this, you have to be willing to ditch the Excel formulas / execution engine, do more abstract reasoning over the file (like identifying tables / consistent formulas in columns and translating them as pandas dataframes), and translate them without just relying on matching the Excel exactly.
You want something closer to this:
df = pd.DataFrame({'Prices': [1, 2]})
df['With Tax'] = (df['Prices'] + 10) * 1.3
We considered using this, but to copy from our MVP spec:
The easiest thing to do is to replicate Excel’s execution engine — you can see someone who has done this [here](https://pypi.org/project/xlcalculator/). But this is just evaluating Excel formulas is not what users want when transitioning a process to Excel - they want to be able to ditch Excel (mostly).
The next easiest thing would be to transpile Excel formulas to the following format:
# This is not a useful or usable artifact; you're still trapped in Excel
# but things are just worse now, since it's not on a 2d grid
A1 = "Prices"
A2 = 1
A3 = 2
B1 = "With Tax"
B2 = SUM(A1, 10) * 1.3
B3 = SUM(A2, 10) * 1.3
1. It’s 100% impossible to read. A large Excel file often has 100k+ formulas (many of them with shared structures). This is 100k+ lines of code...
2. It’s impossible to maintain. Yeah, since it lacks all semantic structure, there’s no f** way you’re going to test it or modify it.
In general: *we're trying to generate a Python script that appears to be written by an expert developer. To do this, you have to be willing to ditch the Excel formulas / execution engine.
You're welcome to: 1. Edit it to fix up places where it can't translate fully. 2. Send it to your colleagues and tell them you wrote it (although... for your own personal morals... maybe don't :) ) 3. Upload it to your companies Github 4. Whatever the hell else you want
To be very clear: this is your code!
The code we generate currently has no dependencies on anything other than pandas, numpy, the Python core library, and the Excel file you uploaded in the first place. This might change as we try and support more Excel features (and so need to do some pivot table mocking), but we're not aiming to lock folks in!
P.S. If you do anything particularly interesting with the code you generate... tell me about it. nate @ sagacollab.com. I'd love to hear P.P.S. I'll update the language on this ASAP.
The problem is data gets into these mega-excels through all sorts of funky routes... and I really do mean funky :)
1. PowerQuery: this is defined statically in the notebook so is detectable by Pyoneer. But I don't know a ton about the integration in Python here. I imagine this is doable.
2. Manual data entry: Pyoneer can't detect this from a static Excel file, really - what's the difference between the static Excel sheet and data updates ever time? Oftentimes, users with a lot of manual data entry to "automate this in Python" by turning an Excel file into like a form. Generating a proper web app out of the Excel file would be pretty sweet!
3. Database output copied - aka, copy in a table. This one is sometimes pretty crazy - I've seen Excel workbooks that have SQL queries just copied and pasted into a random cell in the notebook, so you can copy that and run it on some archaic SQL server. And then copy the output back in...
4. Macros: runs an API call, or an SQL query, or pulls (and then formats) data from another Excel sheet. Then put it in the right place. This then requires translating Macros - which are a whole programming language of their own. This is actually pretty high-priority for us right now, based on early feedback from developers who are in the thick of it with big Excel files.
6. Custom plugins. Big finance shops build/buy plugins that pull in data all the time! We haven't really started investigating how to handle these.
5. Other workbooks: at large banks, there's an additional dependency graph of workbooks that rely on eachother across the org. It's epic. There's a single workbook that defines all market holidays, that's used for all excel files that do performance reporting. And then these performance reports feed into other Excel's (by way of direct references, but also by way of copy and pasting, but also by way of uploading/downloading through a database). Support multiple Excel files at once is something we'll have to tackle eventually!
So... there's a lot to do here. We're really early - so we're focused on two primary things right now:
1. Solving the most pressing pain points first. Hence the early launch so we can talk to more folks and prioritize better. I've got a reasonable idea since I've done so much of this work myself, but every finance shop does things different...
2. Leaving good TODOs when we can't translate something. Currently, we can't translate pivot tables or complex formulas -- but we generate TODOs for these so you can go back and fill them in with the Python skills you have (and maybe the help of ChatGPT).
We're aiming to just give you a Python script. So if we don't translate the data pull how you want... you can just edit the notebook :)
1. Data remains stored in the excel file. The generated script pulls the raw data directly from the notebook - but it's a single read_xlsx call. So if you want to switch it out for an API call, db read, whatever - it's easy to do so.
2. We model data as primitive Python data types, or, if it's a table, as a pandas dataframe.
Currently, we detect at most one table per sheet, and it's gotta be contiguous. These are pretty huge limitations we'll be relaxing soon -- but we wanted to get something out as soon as it would have been useful to one person -- and in it's current state, this would have helped me with some of my larger Excel automation projects :)
Me and my two cofounders spend the past 4 years working on Mito (https://trymito.io) -- where our customers are primarily large finance shops (including some bulge bracket banks you've heard of) that has a really concrete goal of getting users out of Excel and into Python. It's not every finance shop, but a quite a few are trying to make this transition. This usually means: multi-day Python trainings, a Python support team, a few developers who semi-full-time job is helping transition existing Excel processes to Python.
We built Mito to be a tool for the Excel-first users - we tried to make it easier for them to use their existing spreadsheet skills to write Python. But in working with the developers that support these new Python users, it became clear to us that there's a big pain point around:
1. I'm a dev who was given a big, old Excel file
2. It has a lot of business logic in it, understood by the person who made it, but not by me - who is tasked with turning it into real software
3. I have to spend 100s of hours: trying to understand the file, faithfully replicating the logic, and testing for consistency - to convert this to an Excel process.
I personally have been this developer in quite a few cases - just as support for Mito and helping these Excel users trying to transition to Python. Some Excel files literally take 300+ hours to "rebuild from scratch" in Python. It's often very engaging work, but brutally slow - so we're trying to automate as much as we can with Pyoneer.
> There's a vanishing window for stuff like this, if you're a Microsoft shop like 99% of the corporate world I think you are turning those excel files into power apps and powerBI dashboards, before you are hiring python devs.
I think this is a really fair point! We're not sure exactly what a reasonable business model really looks like, long-term. Right now, we're really focused on finding the developers for-which this is a big pain point, and seeing what we need to prioritize to make their lives better. I'm one of those developers...
> confronted with an all-too-familiar dilemma: copy your data into a tool like Excel to make the table, or, display an otherwise unpolished table.
One add-on (coming from the past 4 years of working on a tabular-data from Pythons startup [1]) is that users aren't just copying data into Excel because if it's good formatting capability: very often, there are organizational constraints that mean that Excel _needs_ to be where this data ends up.
The most common reasons I've seen for data ending up in Excel: 1. Other parts of the report rely on Excel features - you want to build pivot tables or graphs in Excel (often, these are much easier to build in Excel than in Python for anyone who isn't a real Pythonista) 2. The report you're sending out for display is _expected_ in an Excel format. The two main reasons for this are just organizational momentum, or that you want to let the receiver conduct additional ad-hoc analysis (Excel is best for this in almost every org).
The way we've sliced this problem space is by improving the interfaces that users can use to export formatting to Excel. You can see some of our (open-core) code here [2]. TL;DR: Mito gives you an interface in Jupyter that looks like a spreadsheet, where you can apply formatting like Excel (number formatting, conditional formatting, color formatting) - and then Mito automatically generates code that exports this formatting to an Excel. This is one of our more compelling enterprise features, for decision makers that work with non-expert Python programmers - getting formatting into Excel is a big hassle.
Of course, for folks who can ditch Excel entirely, this is entirely unnecessary. Great Tables seems excellent in this case (and anyone writing blog posts this good is probably writing good code too... :) )
[2] https://github.com/mito-ds/mito/blob/dev/mitosheet/mitosheet...
- `f_p` - Fraction of all human-level technological civilizations that survive to reach a posthuman stage
- `f_I` - Fraction of posthuman civilizations that are interested in running ancestor-simulations
- `N_I` - Average number of ancestor-simulations run by a posthuman civilization
- `H` - Average number of individuals that have lived in a civilization before it reaches a posthuman stage
And then the formula for the fraction of observers with human-type experiences (after simplifying) is just: f_sim = (f_p * f_I * N_I) / (f_p * f_I * N_I + 1).
By first arguing that `N_I` is likely to be very large (because, pretty much, why not) -- you can thus conclude that one of these three conditions are met:
1. `f_p ~= 0` -- or the human species is very likely to go extinct before reaching a “posthuman” stage;
2. `f_I ~= 0` -- or any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof);
3. `f_sim ~= 1` -- or we are almost certainly living in a computer simulation.
It's all feels pretty similar to the Fermi paradox to me -- which I'm also suspicious of for reasons I can't justify properly. Something about point estimates for variables like `f_I` is... weird? Idk. I'm honestly not good enough at math to disagree - but it also feels like folks who are too good at math might be using `f_I` in equations in a way that isn't legitimate.
Like, assuming the "existence" of `f_I` as a concept to reason with -- doesn't it feel like more might be sneaking in with this assumption?
In the hundreds of data-science-in-Excel files I've seen, I can't think of a single one that doesn't make use of a pivot table in some way.
(https://trymito.io, if you're interested)
Who remembers what they had to eat yesterday? What about last week? What about over the course of 8 months, *literally years ago*? What, you don't remember how much soda you drank in 2020? That's weird. All 300 study participants here probably have better memories than you, I guess.
---------
Many very different things fall under the label of science, but not all of these things are the same. The confidence we can have of our models and understandings in particle physics is very different than the confidence and understanding we can have with vaccine trials is very different than the confidence and understanding we can have with nutritional epidemiology. Personally, I think the last falls outside the realm of science [1], and closer to "here's something I thought of. There's some data attached too. Misled yet?"
Aspartame has been exhaustively studied since the 1980s! The odds you _don't find_ correlations between consumption of a novel compound (that has the most explicit healthy-user bias you could imagine) and medical conditions is 0.
(A reminder that is _not_ a pro-aspartame comment. I don't drink diet soda (unless I get snipped by Diet Coke), or much regular soda for that matter. But a single, *non-randomized, retrospective, recall-based (!!!!) study* making it to the front page is really surprising to me.)
[1] https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
Edit: even a cursory skim of the paper should totally disqualify this from the front page, IMO, *even from nutritional epidemiology standards.* The paper does not even attempt to account for "for maternal overweight/obesity and diabetes, maternal mental health, and other potential confounders in our study." The classic: let's imply a casual variable, ignore the rest of the other possible ones, simply b/c "no covariate data were available."
1. Why is it necessary to generate code in the first place? Can you just skip to the "solution?" 2. Why is just writing the code by the hand not the best solution? 3. So you do want to do code-gen, does it make sense to do it in a chat interface, or can we do better?
As a Figma user, I'd answer these in the following way:
> Why is it necessary to generate code in the first place?
Because mockups aren't your production website, and your production website is written in code. But maybe this is just for now?
I'm sure some high-up PM at Figma has this as their goal - mockup the website in Figma, it generates the code for a website (you don't see this code!), and then you can click deploy _so easily_. Who wants to bet that hosting services like Vercel etc reach out to Figma once a week to try and pitch them...
In the meantime, while we have websites that don't fit neatly inside Figma constraints, while developers are easier to hire than good designers (in my experience), while no-code tools are continually thought of as limiting and a bad long-term solution -- Figma code export is good.
> Why is just writing the code by the hand not the best solution?
For the majority of us full-stack devs who have written >0 CSS but are less than masters, I'll leave this as self-evident.
> So you do want to do code-gen, does it make sense to do it in a chat interface, or can we do better?
In the case of Figma, if they were a new startup with no existing product and they were trying to "automation UI creation" -- v1 of their interface probably would be a "describe your website" and then we'll generate the code for it.
This would probably suck. What if you wanted to easily tweak the output? What if you had trouble describing what you wanted, but you could draw it (ok, OpenAI vision might help on this one)? What if you had experience with existing design tools you could use to augment the AI. A chat interface is not the best interface for design work.
ChatGPT-style code-generation is like v0.1. Github Copilot is an example of next step - it's not just a chat interface, it's something a bit more integrated into an environment that make sense in the context of the work you're doing. For design work, a canvas (literally! [2]) like Figma is well-suited as an environment for code-gen that can augment (and maybe one day replace) the programmers working on frontend. For tabular data work, we think a spreadsheet is the interface where users want to be, and the interface it makes sense to bring code-gen to.
Any thoughts appreciated!
[1] https://trymito.io, https://github.com/mito-ds/mito [2] https://www.figma.com/blog/building-a-professional-design-to...
Similarly to this library, it gives you a `@component` decorator that allows you to create components out of functions. But it also: 1. Includes all existing React hooks (use_state, use_memo, etc) -- so you don't have to learn new patterns. I believe this results in a bit less magic (and so easier debugging) than just using raw variables. 2. Works with ipywidgets, so many existing data apps can be ported over very easily -- Jupyter users celebrate.
I'm not associated with the project, but I know the maintainers (creators of Volia [2]) and they are honestly excellent. I haven't use the project in production, but the getting starting guide is pretty compelling.
[1] https://github.com/widgetti/reacton [2] https://github.com/voila-dashboards/voila
In general, it feels it's halting problem hard to determine if any general purpose programming language program is executable or safe... but maybe if we reduce the size of the spec under consideration?
> AI will be the single most significant driving factor of change in the world. If we solve AGI (or achieve intelligence close to AGI), we'll likely solve most of the world's problems.
I literally don't get the confidence in this statement. I'm not an AI-doomer by any means, but AGI (if possible) will likely be the most powerful technology humankind has ever invented. Just in terms of possible impact, why would we assume it will solve more problems than it will create (or the opposite)?
Think of recent super-powerful technologies we've invented. Sure, there's the potential for fantastic fixes to many problems that come out of nuclear tech. But there's also... the threat of nuclear annihilation? Is that all net positive? Do we even really have a way to evaluate on the timescale of 100 years? How can we know the net impact of nuclear tech in the next century, or millennium?
How can we call this sort of rhetoric anything other than blind optimism? Why would we have any priors about how AI will go? Why do we say things that make us blindly rush forward?
I'm not being sarcastic, or trying to argue one way or another. I'm genuinely asking. How does anyone have confidence in "AI is good" or "AI is bad" claims? Is confidence even good in this case?
For me, these questions lead into such deep and treacherous waters it's probably best to stop the comment there. There are limits to what even interested HN addicts can ask of each other.
Google sheets is a bit harder to figure out -- it doesn't have an explicit row limit, but does have a cell limit of 10M cells - so depending on your number of columns you can back out to the max rows.
If you want to see some practical effects of these row limits, check out the time that England misreported thousands of covid cases - because their Excel file ran out of space silently [1]. Oops.
This looks like a pretty sweet engineering effort! Anything cool you learned while building this that you can share here that helped you achieve this performance? What language did you use to build this application?
I'm asking for interest and for selfish reasons -- fellow spreadsheet builder of Mito [2] - which is pretty much just a UI wrapper for Pandas dataframes (and so can support 10M+ rows as well).
[1] https://www.bbc.com/news/technology-54423988 [1] https://trymito.io
I'm a co-founder of a non-AI data code-gen tool for data analysis -- but we also have a basic version of an LLM integration. The problem we see with tooling like Pandas AI (in practice! with real users at enterprises!) is that users make an edit like "remove NaN values" and then get a new dataframe -- but they have no way of checking if the edited dataframe is actually what they want. Maybe the LLM removed NaN values. Maybe it just deleted some random rows!
The key here: how can users build an understanding of how their data changed, and confirm that the changes made by the LLM are the changes they wanted. In other words, recon!
We've been experimenting more with this recon step in the AI flow (you can see the final PR here: https://github.com/mito-ds/monorepo/pull/751). It takes a similar approach to the top comment (passing a subset of the data to the LLM), and then really focuses in the UI around "what changes were made." There's a lot of opportunity for growth here, I think!
Any/all feedback appreciated :)
EDIT: Also, shout out to JupyterAI (https://github.com/jupyterlab/jupyter-ai) -- it's an official Jupyter project with some really awesome LLM support directly in JupyterLab. I saw it debuted at JupyterCon last week :)
I mean, maybe. Sure, LLMs can currently code shockingly well, and LLMs are just programs, so an LLM writing an LLM is def much more likely than a nuke building a nuke. But we don't _know_ if there are limits to what AIs can do. Or, to pick on the word "exponential," we don't know if there are limits to growth rates of AIs improving AIs!
FWIW, I personally think that AIs will be able to invent more powerful/intelligent AIs. But we should be stating this as as conjecture, not a fact - mostly b/c my experience in saying things like this to people just causes them to be _extremely scared_. And for most people (myself included) extreme fear is a really bad place to try to make decisions from.
> We need an equivalent of the Food and Drug Administration for new technology, and we need it yesterday.
I wish there was more detail on this. Does Yuval Noah Harari really understand how the FDA works and is it really an appropriate model for the AI Safety Administration (AISA)? I mean - the FDA is not really batting 100% - and can we afford to miss swings when it comes to AI safety?
Maybe we can. Maybe we can't. Anyone saying we can afford to miss swings is probably selling you something (or their livelihood depends on it). Anyone saying we can't afford to miss swings is probably terrified. I honestly don't know how to possibly feel other than ???!!??
UX suggestion: let me select a new style without having to re-upload the photo. We wanted to check out a bunch of styles on the same photo, but doing so took 10 clicks instead of just 1!
Tldr: Mito is a spreadsheet that you can use to edit pandas dataframes, from directly within a Jupyter notebook. For every edit you make, Mito generates the equivalent python code, allowing you to perform some tricky pandas operations with the ease of a spreadsheet.
In practice, most of our most active users _are_ in the financial industry. They turn to Mito because they’re transitioning from spreadsheets to a programming language (for one of many reasonsg, but doing so while not knowing how to program is challenging. For most people, the hard part of writing code is the actually process of writing code - so we give these folks a spreadsheet interface they already know.
Happy to answer any questions / hear any feedback!
P.S. We’re open core + source available. Check it out if that’s your thing [2]
[1] https://trymito.io [2] https://GitHub.com/mito-ds/monorepo
(Off topic but very related) I’ve been wondering recently more spreadsheets don’t support a mode of operation where everyone column just has a single formula. Think about e.g. a pandas dataframe - most of the code you write only operates at the column level. For certain data science applications, I think it might be a more sensible and convenient default than every cell being a different formula.
Of course, you can’t do an LBO model with that limitation… but not every spreadsheet need support every use case…
He stopped playing dice baseball when his older brother got his first computer. It allowed you to make super basic spreadsheets (or something that felt like spreadsheets, I don't trust his memory a ton) -- and the first spreadsheets he made were about dice baseball. From there he made spreadsheets about his highschool baseball team, and from there he decided he loved computers and went back to school to study them.
30 years later, here I am on HN :)
For anyone interested in understanding the dynamics behind bank runs, the Diamond–Dybvig model [1] is worth reading. It describes a very simplified situation where due to banks short-term liabilities and long-term assets, a bank run is a valid Nash equilibrium. I think they won a Nobel prize for this model.
There’s also some very interesting discussion at the end about preventing runs: first if banks can suspend withdrawals, and second through central bank backing. I’ll avoid summarizing it because I’m too dumb — but to quote: “Deposit insurance provided by the government allows bank contracts that can dominate the best that can be offered without insurance and never do worse.”