“Design me a highly resilient database”

I’m not usually one to complain about AI being used to generate content because I find this an especially interesting topic, but it really took me out of it.

More relevant to the topic at hand, I feel like I didn’t put that much thought into designing my interview questions, but I’m always astounded when I read these sorts of descriptions of other interview questions. There should perhaps be some formal training or certification or something for asking good engineering interview questions.

The best interview question to find great developers is this:

"What number am I thinking of?"

Smart developers - really smart developers are always right. Truly great developers have seen this sort of problem before and know how to attack it - they need no fancy questions or "deeper understanding". They simply hit the whiteboard, do the logic and output the answer.

Don't settle for anything except A players.

I think it’s a great interview question. Open-ended, rewlly leads to a good discussion that can give you a sense of how the applicant approaches a problem and thinks it through.

Disastrous in the hands of an interviewer who’s looking for a one word answer.

The interviewer question was adversarial... don't think it puts them in a good light.

More importantly I appreciate the OP for describing lessons learned through this hard experience. I worked in finance: Cassandra is no kind of answer for systems of record

Thanks

At a job interview in 2016, I was given a fairly typical "design Twitter" whiteboard problem, so I started doing a fairly typical thing involving load balancers and Nginx and the like.

The topic of the database came up, and I asked "Do we care more about consistency or availability?".

He responded back with "we want both".

I said "I mean, I would like both too but I'm pretty sure we can only get one"

He disagreed and we went back and forth, and eventually I pull out my phone and pull up the Wikipedia for CAP theorem. He eventually (and begrudgingly) agreed and said "umm, I guess availability", so I used Cassandra and moved on with it, though five minutes were wasted with this pointless argument.

To be fair, they did make me an offer, and it was even a decent enough offer (though I ended up taking a different job), but I remember being kind of flabbergasted that I had to explain CAP theorem to a person who had like a decade more experience than I did. It's not like CAP theorem is this super deep arcane topic; I'm pretty sure it's one of the first things you learn about when getting into distributed systems. It was certainly one of the first thing that I learned about.

I really appreciate that your response to being interviewed by an idiot is to write, not an angry rant, but an educational blog post. You are a good person and they didn't deserve you.

When I’m interviewing I never ask a question about something I know super well. I circle around what the candidate signals he has great passion and understanding at, and start deep diving into that.

If I know about it as well, then we can have a really deep discussion, if I don’t- then I can learn something new.

The aim when interviewing is to check how well / deeply the interviewee can think through a problem.

If we pick a topic that they don’t have deep knowledge - they can either stumble and freeze emotionally, or hallucinate something just to appear smart. At this point it is an examination not an interview. And sure some people are capable enough to get to an answer, but that’s more of a lottery than a real analysis.

It usually boils down to how often have they interviewed before and been in a similar situation. And “people who have interviewed a lot” is hardly a metric I want to optimise for.

Now picking something they know or have expressed interest or passion in, this means we are sure to have more signal than noise. If the interviewee’s description is more of a canned response - then I delve deeper or broader.

“I’ve managed to solve this issue by introducing caching” - “Great, are there other solutions? How do you handle cache invalidation, what are the limits? What will you do if the load increases 10 fold, would you need to rethink the solution?”

> The answer he was looking for was ‘Cassandra’. “You’d have a hard time having it NOT be resilient.”

A "database" that starts to fall if the ring in the cluster becomes choked? Draining a broken Cassandra ring and under high load is hell and ball clenching.

Especially hoping that another Cassandra node doesn't fail under high load. Pish, Erlang and mnesia.

Imagine you have a small modeling agency. You have like 10 girls, most of them hover around 6/10 in looks department, you send them to do ads for small companies, assist small events and so on. One day you see an application - 10/10 girl with tons of experience. Way above what you're doing here. What is the correct move? You should reject her application - having ten 6/10 girls and one 10/10 girl will create lots of drama, which will offset any gains from having her on board. If you want your agency to improve, you should hire a 7/10, maybe 8/10.

Exactly the same principle applies with engineers. If you have a team of idiots and you hire one of top programmers, you'll create friction in the team.

I'm glad I read the "Designing Data Intensive Applications" book because I get the full context of the blog.

And the lesson of the blog is why forcing LLM usage is stressful to some engineers. Most implementations are in a quantum state and the back and forth to have clear requirements is what can make the wave function collapse. We know all the possible answers , but all of them are equally valid.

We could go with the LLM route and just do a quick guess on the answer, but "Accountability" and "Professionalism" prevent us from doing so.

"Kubernetes backed by S3-compatible object storage for WAL archiving — giving you 11 nines of durability on your backups .. " - that might be the case with AWS S3 but maybe not with Minio or even Ceph when you need to operate it yourself.

Anyway that interview question is so stupid it hurts. I would never expect just a simple answer to an interview question and mark it off as "done". It's much more valuable to get into a dialogue and see how the candidate is approaching things.

You are better off not working there.

> At Apple Pay, we processed millions of daily transactions across 30,000 servers

Why does it require 30,000 servers to process millions of daily transactions? Is there a typo in the amount?

Hiring good engineers is not that hard. Why do we keep screwing this up?

For the sake of argument, a smart engineer can run Sqlite more resiliently than a dumb engineer can run Cassandra resiliently.

Good time to stand up, shake hands, say "we'll be in touch!" and leave.

Sometimes you gotta fire your interviewer.

/dev/null is pretty resilient

> Not because it’s trendy. Because ACID compliance

> The CAP theorem is not a suggestion. It’s a mathematical proof.

> this is not a philosophical inconvenience. It’s a compliance violation.

@dang, i thought we just banned AI written blog posts?

I know I will get downvoted for this everytime I post this comment, but I'm sick and tired of 80% of everything being written/heavily edited by AI.

Can the author not just use his own words? Is he really that bad at writing? But somehow the content is supposed to ne worth my time?

I'd rather read the prompt instead, so I know which parts the author actually meant to say, rather than wondering which parts the AI invented to make it sound flowery and punchy, the Tiktok style of prose.

something is off here.

naming "cassandra" is not designing a database.

it feels like there was some miscommunication along the way