closed

Hey y'all, creator of siuba here--happy to answer any questions!

One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].

I'm a big fan of pandas though. It's what siuba is built on!

[1]: https://m.youtube.com/c/chowthedog

[2]: https://mchow.com/posts/2020-02-11-dplyr-in-python/

[3]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

Hey, author of siuba here, I totally agree that subclassing would be a natural choice in python. One challenge there is that users will often get a DataFrame back (e.g. from pd.read_csv), so it requires a lot of casting to the child class.

Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do [1]! One obstacle here is that the DataFrame has 300+ methods, which can be overwhelming to learners.

I've spent a lot of time wondering whether the piping syntax feels like too much vs chaining. It's still an open question in my mind, so it's really helpful to hear what feels most natural!

https://pandas.pydata.org/pandas-docs/stable/development/ext...

I volunteer as co-director for an organization called Code for Philly. We help set up collaborations between tech volunteers and other non-profits.

One thing I've noticed is that bootcamp grads consistently putting in time on projects often get a ton of feedback / mentorship.

CfP is part of a network called Code for America, and with covid a lot of projects are meeting virtually, so I'd look for the nearest "Code for" group! (Or similar communities).

Best advice I got in a similar situation:

Don't plan to do any work in the first month.

I ended up doing a ton of cleaning / organizing my apartment. Straightened out finances. Started journaling. Tried things like taking freakishly long walks.

After a month I not only felt ready to work, but also that I had prepped a lot of useful habits!

> What I've since discovered works much better for me is to separate studying and memorizing

Worth noting that the studying vs testing is exactly what's tested in the paper that repopularized the test effect:

https://psycnet.apa.org/record/2007-08020-001

One complicating factor though, that might hit on what you mentioned is how complex the thing you're learning is. There's one group that claims (re)studying is better for complex material than testing.

https://psychology.wikia.org/wiki/Worked-example_effect

Interesting approach! I've been working on it from the other angle: having pandas code generate SQL. If you're interested in checking it out, happy to try and show how it would generate the query in your readme!

https://github.com/machow/siuba

One thing I was wondering about the gnarly ast stuff you mention, what about operator overloading? E.g. Q("Select a from" + subquery + "where a < 1")

> The argument here is scoped to people who have some degree of mastery over the language but are being sloppy with their written communication, which is to say "careless and unsystematic; excessively casual".

I think there are some good points here for technical writing. But it's easy to forget how memorable playful communication can be.

E.g. a responding via haiku or something no-caps and full of enjambment, might be extremely memorable. (especially in a chat full of technical writing style).

Agreed! It seems like there's a lot of power in having "one to beat". As in, let's get some model up before we worry about one that needs daily updates.

AFAIK plotnine is the only one with substantial adoption (2.3k stars on github) and active development for several years.

Would be curious to hear which ones you're thinking of (since I wouldn't be surprised if the are others, but also am guessing no others hit both points)

> Using controlled vocabularies for column names is a low-tech, low-friction approach to building a shared understanding of how each field in a data set is intended to work.

Can't stress how useful this has been working on data science teams. There is really an art for getting a group of people to standardize around a vocabulary of column name pre/suffixes without being overly prescriptive.

Being able to unlock nifty tools / behaviors by naming your columns certain ways seems crazy powerful. (Like a form of function dispatching)

I feel like a lot of these concerns are resolved in tools like R's dplyr. You use mostly the same R code, whether your data is a data.frame, or living in a SQL source. dplyr generates the SQL query for you.

By reading the queries it generates it's quick to pick up how the SQL works! Another big advantage is that you can always pull the data into R and have a ton of general purpose tools available.

dplyr is aimed at data analysis though, so may be other use-cases for edge db?

I am pretty sure the author is describing a well understood limitation of Elo, they just need a tiny bit of connecting to models.

Elo can be thought of as an approximation to item response theory models [1]. These describe skill as normally distributed, and whether one person will win using a logistic function (not exponetial).

I think what the author has keyed in on is that afaik in simple Elo there is no slope coefficient for the logistic, but in general IRT models there is (called item discrimination). So in Elo you can't learn that flatter curve they show.

[1]: http://hvandermaas.socsci.uva.nl/Homepage_Han_van_der_Maas/P...

Working on siuba, a data analysis tool for python. It's a port of the R library dplyr, and can produces SQL queries!

I've programmed in python for much longer than R, and really want to be able to move at the same speed when using python for data analysis :o.

It's a weird problem though because the two languages have basically opposite approaches to DataFrames. pandas has a very fat DataFrame implementation, R an extremely minimal one. (Pros and cons to both approaches).

https://github.com/machow/siuba

Have you seen the R library dplyr? It is a weirdly powerful data analysis tool, that can also produce SQL queries!

Articles like these are interesting, but what surprises me is that they rarely set up a holistic use case, so most debates imagine how long it would take an expert user to use each tool. But time constraints (eg spent coding) separates expert from novice performance in many domains.

FWIW I have 2020 set aside to implement siuba, a python port of the popular R library dplyr (siuba runs on top of pandas, but also can generate SQL). A huge source of inspiration has been screencasts by Dave Robinson using R to analyze data he's never seen before lightning fast.

Has anyone seen similar screencasts with pandas? I suspect it's not possible (given some constraints on its interface), but would love to be wrong here, because I'd like to keep all my work in python :o.

Expert R screencasts: https://youtu.be/NY0-IFet5AM

Siuba: https://github.com/machow/siuba

> ... ability to perform complicated abstract reasoning

This seems like it's kicking the can down the road. It is like saying that people need the ability to perform memory. It's tautologically true that these are both necessary for complex learning.

A lot of the challenge for learning systems is figuring out how to help people develop the mental schemas they'll be pumping abstract reasoning through. (Similar to the design of procedural pieces, which most coding lessons focus on!)

This might sound nuts, but I have settled on a three step process that mostly involves keeping most notes in my head.

Step 1 is using the method of loci (MoL) to memorize up to 30 things throughout the day. For context, MoL is a strategy where you visualize things you want to remember in a room. I have one "room" for each day of the month, with 10 spots in a room, and will often visualize up to 3 things in a spot (so 10 x 3 = 30 a day). I have 4 paths that each lead between 7 rooms, one path per week.

Step 2 is just keeping a page of notes per day in a planner. This is really helpful for jotting miscellaneous things down (like future things to read, possible connections).

Step 3 is that often as I'm thinking back over notes from step 1, or reading new things, I'll want to sort of "put it all together", or write up something that cuts across multiple days. There, I'll often refer to my planner, and might start a spreadsheet to keep references on.

I think the problem before was that I'd wanted to write useful longterm notes when first learning / studying something. But that's probably when I'm least qualified to take really insightful notes. It's been nice to let things marinade, and focus on keeping the "building blocks" around!

https://en.wikipedia.org/wiki/Method_of_loci

Yeah--we're basically second-order black formatters now. Not writing code that looks nice, but writing code so a tool that writes code will make it look nice.

I would be a bit surprised if this is true, because it implies that when you're writing code that will be formatted you do not think of the reader's experience.

This could be the case if you have extreme confidence in the formatter, but I have def seen some very funky black outputs. It seems like when writing people will likely do so with a model of how black will yield the final code in mind. (Eg "if I add more characters to this one line list statement, it will get exploded across 5 lines", and affect readability, etc..)

This is something bail funds in Philly track, and I'm guessing bails funds in other cities do too! Many cities committed to some degree at the beginning of the pandemic to reduce arrests / setting bail for minor crimes. (how well they've done that is often a matter of debate).

The situation in local jails became a big issue at the beginning of the pandemic. I would reach out to a bail fund in your city (or read more on their web page) to see if there are useful ways to help their broader mission, which will include protestors :).

Some useful context in this slate article:

https://slate.com/news-and-politics/2020/06/bail-funds-donat...

> But this again looks not doing enough homework.

It seems naive to read a tweet with a little information about someone completing suicide, and then assume that if they had known (more) about bankruptcy they would not have done it.

There are other things that we can't read out of the tweet that are really important in these situations, like access to social support.

Thanks for the pointers! Not original commenter, but familiar with the literature a bit. One thing that confused me reading your article is that you are criticizing one, fairly narrow way Ericsson has defined deliberate practice.

But in Hambrick, Campitelli, and Macnamara's volume on expertise, they do a good job showing that he has not been consistent with definitions (e.g. 3 definitions: must have, sometimes must have, doesn't require a teacher).

Not a problem for your post, but I was pretty confused about what you were calling deliberate practice until reading this response.

There's a cool python library for machine embroidery called inkstitch, but I love the idea of using a gameboy..

It seems like there could be some interesting ways to teach programming using machine embroidery. So, eg, kids program a pattern, and then get to stitch it onto a hat. Feels like an area with a big pay off right away, but also incredible depth (eg turning an image into a stitched pattern).

https://inkstitch.org/

https://www.turtlestitch.org/

Depending on where you are in the project I would encourage something between...

* Rule of 3 - try applying whatever you're thinking of doing to 3 cases before giving in to the temptation to architect / consolidate.

* Architect w/ UML class diagrams + sequence diagrams of core interactions.

Someone mentioned architecture decision records, which are a great way to connect the first point to the second with a narrative!

https://en.m.wikipedia.org/wiki/Rule_of_three_(computer_prog...

porting the R library dplyr to python. :)

(I set aside 2020 to work on this, but quarantine definitely created much more time for it)

https://github.com/machow/siuba

Excited to see! I had to pick up inkscape over the last month for--of all things--machine embroidery. It has a python extension called inkstitch for creating patterns.

At first it was very confusing, but I'm slowly realizing that where tools like Sketch give a very clean, simple UX, inkscape often has the extra functionality I need!

By python, are you also including pandas? Because it is definitely doing a lot of that!

This is not a bad thing, and maybe I'm alone in this, but I spent my first minute thinking over whether "Shishua" was some kind of romanized mandarin or mandarin-esque word!

(I speak cantonese, though, so maybe mandarin speakers wouldn't fall in this trap)

I agree there are ways to organize this that would benefit outside readers, but am guessing the simple, consistent structure probably helped the author get to 5 years of TILs!

It's interesting they chose to alphabetize and start many entries w/ verbs.

I think everyone feels like they're being hit with too much information at some point. But wanting to hide, and knowing it won't be over sounds really frustrating.

Have you thought about running it past a therapist? They can help you size up the situation (eg are you anxious, and finding yourself avoiding work?), and think about options, or just help track how things are going.