AlexeyMK

That is mostly right, with the caveat that customer.io is trying to go upmarket too now.

https://playbooks.hypergrowthpartners.com/p/picking-your-lif...

I wrote this up about a year ago for a more comprehensive perspective for companies series A+

If you'd rather go through some of this live, we have a section on Stats for Growth Engineers in the Growth Engineering Course on Reforge (course.alexeymk.com). We talk through stat sig, power analysis, common experimentation footguns and alternate methodologies such as Bayesian, Sequential, and Bandits (which are typically Bayesian). Running next in October.

Other than that, Evan's stuff is great, and the Ron Kohavi book gets a +1, though it is definitely dense.

There's a pretty good book dedicated to this exact question: https://unscalablebook.com/

Yes, but it is _rough_. What actually hurts is "browse on mobile, buy on desktop" type behavior.

Still worth doing, but you end up needing more black magic than you'd like (IP-based assignment, Ad Network-sourced assignment, CDN proxies for Analytics tools, etc).

Working on a separate post about that.

Super helpful - looked it up, will aim to apply next time!

Curious how the bonferroni correction applies in cases where the overlap is partial - IE, experiment A ran from Day 1 to 14, and experiment B ran (on the same group) from days 8 to 21. Do you just apply the correction as if there was full overlap?

Thanks, that is a well reasoned argument!

My take is for small n (say 5 experiments at once) with lots of subjects (>10k participants per branch) and a decent hashing algorithm, the risk of uneven bucketing remains negligible. Is my intuition off?

False positives for experiments is definitely something to keep an eye on. The question to ask is what is our comfort level for trading-off between false positives and velocity. This feels similar to the IRB debate to me, where being too restrictive hurts progress more than it prevents harm.

Thanks for factorial design! I'll update the post to the proper nomenclature.

The frequentist/bayesian debate is not one I understand well enough to opine - do you have any reading you'd recommend for this topic?

> Nooo! First, if one actually works, you’ve massively increased the “noise” for the other experiments

I get that a bunch at some of my clients. It's a common misconception. Let's say experiment B is 10% better than control but we're also running experiment C at the same time. Since C's participants are evenly distributed across B's branches, by default they should have no impact on the other experiment.

If you do a pre/post comparison, you'll notice that for whatever reason, both branches of C are doing 5% better than prior time periods, and this is because half of them are in the winner branch of B.

NOW - imagine that the C variant is only an improvement _if_ you also include the B variant. That's where you need to be careful about monitoring experiment interactions, I called out in the guide. But better so spend a half day writing an "experiment interaction" query than two weeks waiting for the experiments to run in sequence.

> Second, xkcd 882 (https://xkcd.com/882/) I think you're referencing P-hacking, right?

That is a valid concern to be vigilant for. In this case, XKCD is calling out the "find a subgroup that happens to be positive" hack (also here, https://xkcd.com/1478/). However, here we're testing (a) 3 different ideas and (b) only testing each of them once on the entire population. No p-hacking here (far as I can tell, happy to learn otherwise), but good that you're keeping an eye out for it.

[Author here] Heh - yes but don't, though...

Yes: you could use bayesian priors and a custom model to give yourself more confidence from less data. But...

Don't: for most businesses that are so early they can't get enough users to hit stat-sig, you're likely to be better off leveraging your engineering efforts towards making the product better instead of building custom statistical models. This is nerd-sniping-adjacent, (https://xkcd.com/356/) a common trap engineers can fall into: it's more fun to solve the novel technical problem than the actual business problem.

Though: there are a small set of companies with large scale but small data, for whom the custom stats approaches _do_ make sense. When I was at Opendoor, even though we had billions of dollars of GMV, we only bought a few thousand homes a month, so the Data Science folks used fun statistical approaches like Pair Matching (https://www.rockstepsolutions.com/blog/pair-matching/) and CUPED (now available off the shelf - https://www.geteppo.com/features/cuped) to squeeze a bit more signal from less data.

Another option is monetizing via a niche job board for your readers. Pallet.com is good for that - will find companies, etc. Disclosure, tiny investor.

Sure, but you need an "undeleted" view per table now - I haven't seen an ORM offer maintaining those views capability out of the box - any to look at?

You can get around the "check in every transaction" problem with an ORM, but now you're (more) coupled to your ORM which you will occasionally inevitable need to circumvent for something or other. And now you've made it (more of a) leaky abstraction.

It's an article on a personal blog targeting a specific readership familiar with the term.

More broadly, I think the level of granularity at which it makes sense to define terms (Mesos, Kubernetes, even Uber) has to roughly match the level of familiarity of the reader that will get something meaningful from the piece.

This is super well done & polished. Will try out for future blog posts.

Thank you!

> The truth is, had tech workers actually assumed a significant measure of political influence, and led in local politics, San Francisco would today be one of the greatest cities in the world. But not only was such political influence not achieved, it was never attempted.

Is that right? I thought Ron Conway was decently involved in SF politics (https://www.sfexaminer.com/news-columnists/hes-back-ron-conw...) and Ed Lee was backed by tech (https://www.theverge.com/2017/12/13/16771912/mayor-ed-lee-sa...).

My mental model lately has been, if I'm enjoying the content, I should try to figure out how to pay ~$1/hour.

If I'm not enjoying it, I should find a better way to spend m time.

Thanks! Updated on both fronts.

author here

> We explicitly only changed the infra which served our landing pages, and kept the content - the HTML/CSS/JS - identical. Once the new infra was shown to work, we would begin to experiment with the website itself.

- http://alexeymk.com/2020/07/14/lies-damn-list-and-front-end-...

We did that for about a month in 2015; made it for Songkran. It was pretty great. Frustrating with time zones and car pollution, but otherwise solid. Ristr8to remains a favorite.

Both? Github, originally, Gitlab, more recently.

Agreed that tech companies will make the switch a lot sooner. What about a GE-types, though?

Also, I'm generally bearish on remote-friendly or half-remote. Either the culture is set up for it, or it isn't. You can't be just a little bit pregnant.

That made sense. I was following along, and then all of a sudden, it just kind of ended.

As a layman who doesn't clearly remember B Trees, it would be awesome to have even a sentence at the end, like

...and that's B Trees! Commonly used for storing fields in relational databases, filesystems, and more!

For fellow laymen, https://en.wikipedia.org/wiki/B-tree isn't bad, but is there more?

Opendoor Eng here - Cloudflare workers with Optimizely Full-Stack A/B testing have been awesome for improving experiment velocity without ugly URLs or any client-side experimentation code.

Consider https://slatestarcodex.com/2014/09/30/i-can-tolerate-anythin... for another take on political centrism/general tribalism. A bit (err, quite a lot) lengthier but it gets to some interesting ideas about the inevitability of tribes. Stephenson's Seveneves also comes to mind as a book-long meditation on the same topic.

> Stop trying to shorten code by 2 lines. It’s not that bad to write it out.

That brings me back to Blub.

"""

After a certain age, programmers rarely switch languages voluntarily. Whatever language people happen to be used to, they tend to consider just good enough.

[...] I don't want to hurt anyone's feelings, so to explain this point I'm going to use a hypothetical language called Blub. Blub falls right in the middle of the abstractness continuum. It is not the most powerful language, but it is more powerful than Cobol or machine language.

And in fact, our hypothetical Blub programmer wouldn't use either of them. Of course he wouldn't program in machine language. That's what compilers are for. And as for Cobol, he doesn't know how anyone can get anything done with it. It doesn't even have x (Blub feature of your choice).

As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.

- http://www.paulgraham.com/avg.html

"""

Being able to express an idea concisely (IE, "save two lines") is a big part of why we bother to innovate on programming languages. It's what it means for one language to be more powerful or expressive than another.

None-aware operators can help express an idea more concisely (IE, make python more powerful a language). I've used them to that effect in Ruby and CoffeeScript.

I agree there is potential for terseness. Every project needs to have a sense of where the team stands on terseness versus time - probably don't use `??` much if you're creating tutorials or teaching, but also avoid other pitfalls - picking unclear variable names, writing deeply nested logic (cyclomatic complexity), or clunky nested list comprehensions.

Avoiding needless terseness is the job of the programmer or the code reviewer, not the language. The job of the language is to provide power, or the ability to concisely describe intent. None-aware operators have helped me with that in other languages, and would be great to have in python.

Danny Crighton has a CS degree from Stanford. I'm not entirely convinced by the article either, but your criticism would be a bit more credible if you managed to avoid being factually wrong.

I'm most curious to see what the stats are deeper into the funnel, specifically:

- At what rate do bootcamp grads vs new grads get offers (intro --> offer at portfolio companies)?

- Is the above metric significantly different for different classes of companies (either segmented by company size, field, or "CRUD-eyness" of company?

As a former hiring manager at a "much harder than CRUD" company, I remember looking at some bootcampers and saying "I wish we could interview these people, but the knowledge gap is just too significant".

My biggest concern with coding bootcamps as an industry is that the people making the decision about which bootcamp to attend are dramatically under-qualified to make the decision and therefore relatively more prone to both scams and instructor incompetence.

Here's how I imagine this goes, from the author's questions:

> Do you want to do a bootcamp?

I guess so? I'd like a job, and I hear this pays well and folks are hiring.

> Do you want work as a professional developer? If so: front end or back end?

I... what does that even mean? How do I know?

> Any specific programming language you want to learn?

I like the, uhh, curly braces? So... ruby?

> How much money are you willing to spend on tuition?

As much as it takes to set me on the right career path?

> Do you need a lot of mentoring and teaching? Or are able to learn stuff by yourself?

This phrasing is patronizing. How about "In your past experience learning STEM-type material, did you benefit disproportionately from having structured learning time and dedicated instructors, or were your more the 'learn from the book' type?"

(As an aside, if you're not an autodidact, you'll probably not do well in the kinds of positions that people are hiring typical bootcamp grads into, C-grade start-ups with at most a seed round in funding)

There is probably an opportunity out there for an assessment that yields a decent prediction for "if I take one of the better bootcamps, is there a decent chance I will get into an engineering career in tech, and, will I want to stay there?" This blog post isn't it.

(Another aside, the better bootcamps (HackReactor, for one) have already gotten rather good at this sort of screening and are generating money based on their placements and the quality of their reputation).

This topic seems less about power and more about invariants. I could imagine a powerful, fully expressive language that does a better job of explicitly stating its invariants.

In the `urlpatterns` example,

  urlpatterns = [
      url([m('kitten/'), views.list_kittens, name='kittens_list_kittens'),
      url([m('kitten/'), c(int)], views.show_kitten, name="kittens_show_kitten"),
  ]

I could see `m` and `c` being used as a more-specific-than-regex DSL specifically for routes. Reversing routes would be even easier because of the inherent limitations of using the DSL - you have to pass a string to `m`, you have to pass a type (and a variable, ideally, so it would be `m('kitten/'), c("id", int)` or just `m('kitten/'), id()`.

The lesson I gain is "figure out the implicit invariants from your design decisions, make them explicit and try to keep as many of them as possible, only letting go once you have no other choice."