Why we’re pausing our Pay Later program
blog.interviewing.io
blog.interviewing.io
Another example to add to the annals of A/B horror stories. Sample size too small, test length time too short, extremely relevant KPIs like chargebacks or fraud complaints not included.
A/B the background color of the home page all you want, but A/B testing legal disclaimers is going to be, uh, risky.
And this is a very tired and boring thing to say, but startups are not AAAMM. If you're on a product team at Meta you can do ten A/B tests a week because tens of millions of people are using your product. If you're an early stage startup looking for market fit, a A/B test really will authentically take weeks or months to get enough data. If you're on a short runway I could imagine it would be painful and unpleasant to just sit on your hands the whole time, but otherwise you wander down blind alleys like the above.
Edit: A better way to put that might be - only use A/B testing to make tactical decisions, but never use it to decide strategy or logistics.
No. Definitely not. I worked at a company that lent money for car loans. We A/B tested the HELL out of our lending strategies. Testing, data collection, dashboards, monitoring was a HUGH part of our strategic advantage.
The right starting point for thinking about AB testing is every experience is something that feels safe to put into production. That's what you're doing, but something about it being a "test" makes it not feel real to people. Maybe that threshold is low for some call to action copy, maybe it's much higher for something more critical. Use whatever tools you otherwise would to get to this point. Then you can AB test to see if it actually works.
The second part is that in theory AB testing doesn't specifically drive to local optimizations (you can test and iterate on big changes) but in practice it sure does. It also tends to encourage short term thinking, maybe comically so, because those things are faster to test. It's local optimization on the time axis. "Changing the expected ship date for custom work from 90 days to 30 days increased sales 4% with no increase in support tickets!" and they ran the test for 4 whole weeks, so how could it be wrong? Maybe someone should tell manufacturing about this test and how great it went...
A/B testing is a tool. It gives you data, not answers.
Would I make a color change off of 100 data points and a p value of 0.05? Well, multiply the risk of being wrong against the cost of being wrong. A 1 in 20 chance of being wrong isn't terrible when the cost of being wrong is low.
Would I make sweeping changes to my sales funnel at n=100 and p=0.05? Maybe not. n=10000 and p=0.001? Slam dunk yes. (And you should have probably halted the test earlier... imagine all the potential customers who got the "bad" leg and bounced?)
If results are currently at n=5000 and p=0.06001, and it's going to take 6 more months to hit n=10000, and you've only got 5 months of runway... Well then, give up on that test and do something else. Opportunity cost is a cost too!
At the end of the day, it's up to you to make an informed business decision using your best judgement. "Bounded rationality." All decisions are made with imperfect information and under time pressure. Like in TFA, if your chargeback rate spikes after a change, is it because the change was bad, or because you're expanding out of early adopters and into choosier (more annoying, more normal) customers? Or it's just a random Thursday and your sales volume is so low that every day is spiky and weird?
As @joelrunyon notes above, in the very earliest days, when your total sales number in the double digits, no A/B test in the world is going to be powered enough, short of "normal landing page/404 error". You just have to be good at your job, and lucky.
> If you're an early stage startup looking for market fit, a A/B test really will authentically take weeks or months to get enough data.
Most of the times at startups you will never have enough data to parse out exactly why something is or isn't working. It's more like an artist where you need taste to see where things are going.
Once you have that - then you can start to optimize with data.
Seen so many super-smart data guys at FAANG companies switch to startups and get stuck in the mud because they're looking at dashboards everyday and wondering why they're having a data anonomoly instead of looking at the bigger picture.
Sometimes there's not a huge reason why traffic/conversions/sales are lower. Sometimes it's just Thursday.
The Stanford course, for the curious: https://web.stanford.edu/class/cs9/
The product we're working on: https://devscreen.io
Do you have any reference clients? Would love to interview at such companies.
I don't feel so pressured when I interview through pair programming which helps a lot to showcase my normal self.
It's easy to display your technical skills (and any gaps in it), soft skills (such as communication, perseverance, thought process, etc.), problem-solving skills and so on.
I'd much rather spend a 5-hour interview just pairing with different people than going through rounds and rounds of whiteboarding. Whiteboarding makes me feel like I'm a lab subject, being prodded in different directions while the experimenters take notes on their clipboards. It's extremely artificial to evaluate any real-life scenario.
>By choosing DevScren, you send a clear signal to candidates
Best of luck to you!
Is it "rent seeking" to film a TV commercial for your plumbing business or to work on your online marketing skills? It probably doesn't make you a better plumber. It probably makes you more money and may even bring in opportunities you wouldn't have had access to that could lead you to develop skills.
Could be, depends
He will gain market power in his local market though, which could lead to his expansion and the closing of his least successful competition, bringing down area plumbing wages and allowing him to scoop up bad and inexperienced plumbers at a discount.
After that, we don't have to worry about his skills because he isn't working as a plumber anymore. He'll instead be paying bad plumbers bad wages, and keeping all the profits for himself. His desire will be to lower prices while expanding advertising, so the only variable to be played with is plumbers' wages vs. profit margins. He could then make high wage offers to the more skilled plumbers working for his competition in order to kill them off, and each time another independent shop dies, more of the wages will be shifted to profits.
Eventually, the market is served by the same set of plumbers as before (or likely fewer because some will leave the profession or the area), working at lower wages, but charging higher prices, whose excess goes directly to the shrewd marketer plumber. That's when our plumber enters local politics.
Maybe it is rent-seeking.
I've certainly seen both kinds of businesses before.
I don't think so - it just means that your competency can be hidden, and it's possible to make it more visible to the interviewer.
For an extreme example if I go into an interview and remain mute, then I won't pass. If you teach me that I have to speak then I would do better in the interview. My technical competency hasn't changed between the two interviews.
The only interpretation of this statement I can’t think of is “interviews do not measure competency.” That’s not true. They do, but the bias for high precision instead of high recall. That is, it’s better to pass on a competent person than to hire an incompetent one.
Regarding the guidelines, I’m not sure what you mean. One of the guidelines is to assume good faith, however.
But you can study for the interview just like you study for the test.
I could have made the same mistake. Their customers are highly technical engineers applying to FAANGs and other top companies, I'd personally expect a greater understanding of terms and payment details from that cohort.
They don't, as far as I can tell, vet their customers. They deliberately cast as wide a net as possible, so it was entirely foreseeable they were going to hook some less sophisticated users.
This is a lower scale and lower cost version of the for profit bootcamps and IT schools. Yes, if you go in with the tools you need to succeed, you will probably gain something and get a good start at a FAANG or similar. The programs cannot work miracles and have a vested interest in casting as wide a net as possible even though that's a disservice to their customers. This service figured out they need to reign in that net to confirm people can actually pay their cost.
The vast majority of the people signing up are going to be wannabe engineers - what percentage are actually highly technical needs to be determined.
Tech has become popular because of the salaries, the amount of people interviewing has heavily increased, but the talent levels are quite diluted. Especially because any engineer worth their salt is off the market in under a month right now, if not shorter.
People can get so caught up in the numbers on a dashboard or AB test that they forget that there's a human on the other end that just got duped into a 4 figure expense. Gross indeed!
>Practically speaking, I’ve learned that creating some amount of friction is necessary when you’re asking people to promise to pay you ~$1000 in the future. Removing that friction can create short-term wins but may hurt you (and disappoint your users) in the long run.
But people are going to be confused from a "click this button and you owe us a grand" UI seems blindingly obvious. I suspect they got greedy and thought it wouldn't going to be as big of a deal as it turned out to be.
Because I have CC disputes as a backup
Because I'm lazy
Err, this doesn’t line up with any of my experience, nor that of any of my peers or the multiple employers I worked for throughout the pandemic. Maybe things slowed down for a month or so, but there was so much job mobility and choice it was bonkers!
Additionally, they wrap up the gray/small print with the final word, on what the reader is supposed to think, in boldface, and delimited by triple asterisks: "94% of our users find a job within 4 months of starting to practice, and of those, the majority get offers from FAANG".
There seems to be a conscious component of persuasion in that, so I think no surprise if some customers later felt they'd been misled or manipulated.
If you'd have asked me to guess, I would reckon only 30% will pay.
A chunk will have used credit cards with no balance, or cards about to expire. A chunk will dispute charges. A chunk will never see your emails because they land in spam.
Those types of users tend to be put off by PDF documents you have to print, sign and return, because then they feel it's more likely that they will get a court summons for not paying...