narush

Your videos are awesome. I've watched every one of them multiple times. I love when you go deep into the technical details of the problems / solutions you run into (especially with your code) -- especially in _integration hell_.

I'm sure not-your-entire audience is interested in these things. Have you ever considered making a second channel where you go into more detail about the details of the technical problems / solutions you run into? The videos could be a lot less well produced, I'm just desperate for the info.

Thanks for all you do!

This is a cool overview, and I certainly learned new things about the Python language from it. Thanks for posting!

We do lots of Python metaprogramming at Mito [1], but generally avoid all of this fancy Python fluff to get it done. Specifically, we avoid metaclasses, invisible decorators, etc. Instead, we take a much simpler approach of having Python code that literally populates as template .py file, and then writes it to the correct location in our codebase.

As a concrete example: we’re a spreadsheet, so we let our users transform data in a bunch of different ways - adding a column, writing a formula, deleting some rows. Anytime I want to add a new transform (say, encoding a column), I tell the metaprogramming package “python -m metaprogramming step —name “Encoding A Column”. It will ask me some questions about the parameters and types of those parameters, and then write most of the 4-6 boilerplate Python and Typescript files I need automatically! You can see it here [2].

This is still metaprogramming (it’s certainly code that writes code). But the code you end up with at the end of the day is very simple Python code that is extremely easy to understand / maintain long-term.

I’ll pass on the fancy stuff for now. Thanks though!

[1] https://trymito.io

[2] https://github.com/mito-ds/monorepo/blob/dev/mitosheet/dev/c...

I like that framing a lot! Aligns with some other comments about focusing on the positives, certainly!

Very cool! This second part (running Python in the sheet) is coming soon - both allowing you to call custom Python functions in the sheet - but also allowing Enterprises to extend the spreadsheet interface as well!

I could have been more clear with my language -- "not ideal" doesn't mean that it was unpleasant (actually not awful hot-reload loop).

It was more-so that it's technical infeasible, given with Excel exposes to you as an extension developer. Aka, the gates the add to their ecosystem make it pretty much impossible, which is another one of the reasons we're open source!

Will clarify my language around this going forward!

Super fair and honestly great feedback. I think the phrasing you say is probably much more appealing to users when they think about what they want/need!

This sounds sweet. Pretty much like Excel's auto-fill functionality, but instead it works on entire tables (and generates Pandas code).

I wonder if there are good heuristic-based approaches for determining this. We could always to some ML code-gen, but I prefer deterministic approaches for their reliability (and how it's easy to grok what/why failure occurred) -- at least for now!

Very cool, thanks for linking! I’ve never heard of this before - do you know if it generated Python or rather allowed you to edit the spreadsheet with Python?

One thing we’d like to add soon is Python spreadsheet formulas. Often requested - I think it’d be super sweet as well!

Thank you! It's recorded using QuickTime screen cap, and edited in Final Cut Pro. I also made some assets in Figma (e.g. the little spreadsheet grid background).

It took longer than I'd like to admit... but feel validated in spending that time now that you've asked :)

I think your description is a pretty accurate description of most of our users: they are struggling to write Python in a Jupyter Notebook, and can install some basic packages (albeit it with some struggles -- see our Discord install help channel). The ML code part, you're right, def more rare :)

Python code helps these users do a variety of tasks that aren't possible in other analytics tools like PowerQuery/M. Many of these tasks are specific to the company/existing infrastructures, as I mentioned above.

A super concrete example: the head of data strategy at a life-sciences company made the transition to Python primarily because the rest of his (2 person) team uses Python. They primarily communicate about new datasets using Mito generated code (e.g. here are the steps to clean this data) - but he's not great at Python - so in practice he uses Mito for 9/10 analyses he does to generate this code he sends to his colleagues!

Can give a few more if you'd like -- let me know!

It's a good question why our users prefer Mito+Python over something like PowerQuery+M! One might similarly ask what's wrong with Excel+VBA - although I'll note I haven't heard anyone champion VBA recently... :)

In practice, most of our users are have started with Python by the time they use Mito. For now, we're not positioning ourselves as an alternative to PowerQuery, but rather a tool for someone who is coming from spreadsheets, has chosen Python, and is struggling to write code.

The next obvious question is why our users are choosing Python in the first place -- what I'll say here is that like any programming language, there are a huge number of reasons: some of our users prefer Python because that's what their colleagues work; some choose Python because they think it's trendy/cool; others choose python because that's where the libraries they want to use are; others are starting down the path of getting into ML (which is primarily in Python); others want to integrate with existing Python infrastructure within their company. We've also seen massive enterprises with top down edicts to move to Python "within the next 5 years", as well.

In practice, Python is the most popular general purpose programming language for data science - and so we're doing our best to meet our users where they are: writing Python code, in Jupyter Notebooks!

Thanks! Lots of investment in meta-programming in the past 3 months - in our case, code that writes code that writes code :)

Takes from spreadsheet wizards greatly appreciated.

We've thought a lot about this one. It's a good idea for usability - agree with you there - but there are some development complexities that make it hard for other reasons.

We spent a considerable amount of time two years ago developing Excel extensions for our spreadsheet-version-control product. It was... not ideal from a development perspective.

The benefits of being in Excel (it has all the features!) is also the cost of being in Excel (you have to support all the features!). This means v1 of the extension you describe with either have to be non-functional on most of these power tools you mention, or we'd need to spent years building in stealth mode before launching something fully working (and I'm not even sure we ever could get there... Excel is... literally so big).

Also, the actual extension points for Excel are not as fully-featured as you might think! In practice, we'd likely have to gate much of Excel's functionality to get an extension that actually works -- there are some hard limits to what you can extend, further making it really hard to actually support these power tools in practice.

Also, for the sake of our users, we love being in a Python development environment! In practice, many of our users move really fluidly back and forth between writing Python and editing a Mito spreadsheet. Effectively - bring a spreadsheet in for what it's good at, when you want it.

We'll keep considering this one, though -- I have a _feeling_ Microsoft might make some Python moves in Excel the next few years... :)

Streets themselves have also gotten much worse! The vast difference in car deaths in the US and other countries is more a question of our infrastructure and what it encourages than anything else.

I read a really great book about this recently -- no vested interest beyond just really liking it. Check it here [1]

[1] https://www.strongtowns.org/book

Totally agree that audibility is an important part of the story here!

Having spent the past 2 years building a spreadsheet [1], it's really interesting how often we run into design problems that pit "audibility" against "what you expect from a spreadsheet."

A simple example: imagine you add a filter to column to remove null values, you then go and create a new formula that is dependent on this filtered column, before finally going back to edit the filter.

On one hand, effective audibility usually implies a nice, linear story you can follow and understand. On the other hand, users expect filters to update based on the most recent values - but the filter is applied in a way-old step. In practice, there's a bunch of extra state you have to store to make sure things work properly, and even then, there are lots of foot guns!

[1] https://trymito.io

Excel is an extremely cool piece of software. As someone who spent the past two years of his life implementing a spreadsheet [1] (built to solve many of The issues with spreadsheets mentioned in this thread: data size limits, speed limits, and repeatability problems), it’s unbelievable how many different use cases are supported by just Excel.

One of the funnier Excel use cases I’ve seen in the wild, when talking to users, is someone who had implemented the game 2048 entirely within a spreadsheet (ofc VBA). And it wasn’t just for fun - they weren’t allowed to play games on their work computer, but they needed their daily gaming fix. If you can dream it, Excel can probably get it done…

[1] https://trymito.io

If you're running a node, and it finalizes a block, that means your node will refuse to ever revert that block.

A long-range attack is when a validator withdrawals their stake, waits the withdrawal period (e.g. the 6 month delay delay mentions above), and then creates a fake chain starting from before they withdrew their staked eth.

Because in the "real" history (e.g. the ones that most nodes have seen over the past 6 months) the validator doesn't have Eth locked up still, there's no way to punish them. Thus, these long range attacks get very cheap (you could even imagine someone who pays validators for old keys -- aka, you don't even need to be a validator yourself).

These two facts together mean that PoS blockchains require some "weak subjectivity" - which pretty much means when you download and start syncing your node, you need to know a "finalized" block hash from the past 6 months (or within the withdrawal delay). This ensures you won't get tricked by a cheap long-range attack.

In practice, I don't think this will be much of a problem - clients can just do a new release with a new block has every few months for new users!

Check out the book! You'll get a lot from it I think -- even if it's just a better understand of a different viewpoint :)

I'd highly recommend reading Confessions of a Recovering Engineer before debating my communication of it. There's a lot more depth to the arguments than the one-sentence summary a novice (me) can give :)

Namely, Strong Towns isn't anti-car! In fact, they explicitly arguing _against_ removing cars from city centers in most cases, for reasons related to those that you mention. If I remember correctly, many towns tried to do this in the 70's or something, and most of them failed; cars are still required in most cases.

That being said, _requiring_ cars and _allowing_ them are very different things. Being anti-bulldozing city blocks to put in 6-lane highways isn't anti-car, it's pro people!

Again, highly recommend engaging with the source material rather than taking my word for it. I've been consuming and parroting (sometimes badly) their content for months - they have been thinking about these questions for literally decades!

I _highly_ recommend consuming more of the content put out by Strong Towns [1]. If I were to summarize their message, it's that in the past 100 years, we've engaged in a large-scale experiment to design our places around cars. This is bad for safety, bad for the economy, and (most importantly!) bad for us people that live in these places! Content and education-wise, they are my favorite non-profit :)

If you're looking for audio: their podcast [2] is great (this article came out yesterday as a podcast, and I listened to it while I was biking... and then got angry at all the cars driving around me, lol).

If you're looking for something longer-form, I'd recommend reading Confessions of a Recovering Engineer [3]. It's literally fantastic, and changed the entire way I think about the suburbs and cities. I grew up in a suburb and recently moved to the city, and had this deep, under-thought feeling that the walking/biking (and the livability that came from it) was why I loved cities so much. This book helped me understand why I felt that way, what specifically is wrong with how we build place currently - and also practical things I could do about it!

Not to be dramatic, but that book took me from pretty much _never_ thinking about place to: volunteering for a local bike advocacy group, biking twice as much, and literally being a more friendly neighbor. Effects not guaranteed, but I can't recommend this book highly enough if you're interested in this sort of thing.

P.S. I am totally unaffiliated with Strong Towns. I just really like them :)

P.P.S. If you're looking for some interesting drama, check out the lawsuit they are currently engaged in [4] with the Minnesota Board of Licensure. IANAL, but it kinda just seems like the engineering licenses boards are trying to shut down his speech because he points out that maybe letting only-technical engineers totally design our spaces isn't really the best idea!

P.P.P.S. Donate here [5]!

[1] https://www.strongtowns.org/

[2] https://www.strongtowns.org/podcast

[3] https://www.strongtowns.org/book

[4] https://www.strongtowns.org/journal/2022/2/21/the-latest-upd...

[5] https://www.strongtowns.org/membership

Glad to hear. Jim Thorpe seems like a really amazing dude.

Slightly-off topic: if you're ever passing through Pennsylvania, check out the town Jim Thorpe. The waterfall hike there is top-notch.

I’ve done dumber things, no judge. That being said, the easier we make language models to use, the more likely we’ll accidentally end up releasing things like this (and worse) into the wild.

Anyone here work on language models (duh, yes)? Can you give me a not-off-the-shelf-and-sorry-for-the-snark-but-actually-convincing argument for why this isn’t going to be a huge problem? I don’t know much about much, but sounds scary 2 me!

Sounds like a bunch of very cool gobbledy-gook to me. My (extremely) layman's understanding is that this this is effectively better thermometers that could be useful for (among other things) bio-calorimetry.

My previous understanding of the word calorimetry was "indirect vs. direct calorimetry" - which (I think) are two techniques for measuring the energy used by some person either at rest or during. It's useful to understand how much energy someone is burning for obvious reasons related to health, nutrition, energy balance, etc. It seems like calorimetry has a much more general usage though, and it's pretty much the science of measuring heat transfers. Cool :}

There is so-much extremely-high-context science out there, and it would would be so fun and thought-provoking to keep up with some it it. It feels like my only options are a) spending a ton of time trying to parse extremely high-context articles (and probably misinterpreting them, like I do above), or relying on pop-science communication. The latter, to me, seems to miss most of the time - it usually simplifies things to just conclusions like "new X does Y," which feels like it misses most of the scientific processes while also misunderstanding the scientific spirit of this sort of knowledge...

Thanks a ton!

Mito is open source, but using Pro features does actually require a Pro or enterprise license. You can check out this callout in the license [1], as well as the restrictions on Mito Pro features here [2]. We're in the process of fixing up the upgrade to Pro process a bit... as you can tell... :)

You can of course fork Mito and turn off telemetry as long as you open source your changes! Go for it - happy to hop on a call and help you get set up with the codebase, if you want. Yay open source!

[1] https://github.com/mito-ds/monorepo/blob/974091b455950c6c50e... [2] https://github.com/mito-ds/monorepo/blob/dev/mitosheet/mitos...

Good points! I also think that this is an area that Mito could do better in. While we do provide pretty cool summary stats [1] and graphing capabilities [2], there isn't a great view for the summary stats of the entire dataframe. It's def on the roadmap -- but this comment makes me think we should move on it quick.

Thanks for the feedback!

[1] https://docs.trymito.io/how-to/summary-statistics

[2] https://docs.trymito.io/how-to/graphing

Yeah, we've thought a bit about a plugin API - for the reasons you say, I think it would be an awesome feature to open up to teams!

Any tips on going about it? No need to share the secret sauce, unless you want :P

To be totally honest, we're not architected super well to support plugins currently. The big challenge would be allowing users to specify this plugin in pure Python (seems like we want this) - but we think that hand-coded UIs outperform autogenerated ones for now. We've been thinking about how to do better though... maybe soon.

Of course, if Mito is missing features, we're open source [1] -- all contributions are welcome! Also feel free to open an issue and we can discuss :)

[1] https://github.com/mito-ds/monorepo

Super fair, lol. We'll work on optimizing it - just a tiny team and lots of things on our plate rn. The main issue is our images / video, which I have tried compressing but can't do so while maintaining the quality. Any tips are greatly appreciated!

Believe it or not, the last version of this website was even heavier...

This is on the roadmap! Would love to hear a bit more about how you would use this component...

1. Would you want the component to generate code? Or would it just be the editing of a dataframe that is useful to you?

2. What other components would be used in this dashboard? Would love to hear a bit more about the workflow around Mito here.

The more detail you can provide - the more helpful in prioritizing this! I think Mito in streamlit would be ... awesome!

Oh, sorry I wasn't clear! We totally expect that users will use Mito in notebooks on the cloud cloud, and we are in support of this usage!

Ideally, we will continue to extend our support to these environments over time, as currently there are lots of environments where users want Mito but we don't support it yet (notebooks api differences, etc) - a good example being AWS Sagemaker.

I'll edit my answer above to be more clear about this as well. Thanks for the ask for clarification!