Writing scalable backends in UDP
mas-bandwidth.com
In this article I share an interview question I've used for years at Network Next. How to implement a highly scalable backend in UDP. Can you solve it?
mas-bandwidth.com
What they mean to say is ".. an experienced senior engineer *with my exact skill set* could ..."
Senior engineers vary wildly in their skills. Would you expect a senior engineer with experience with (to pick a similar set) Ansible, Rust and AWS to implement this over the weekend using Golang, Terraform and GCP? Obviously not and, just as obviously, you don't need such a wide difference to quickly bleed from "over a weekend" to "over a week".
These sorts of bizarrely complex and specific code challenges would at least serve to greatly reduce the number of resumes you have to review. So that is one upside. Unfortunately what that selects for is probably not as obvious as it would seem as many of their candidates would spend that full week on the problem and I wouldn't apply for fear of a workplace culture that values their employees time that little.
I mean, is that OK? To have a technology stack that you are hiring for (Google Cloud, Golang, Terraform etc), and to look for candidates with experience with that stack, or who would consider learning that stack?
ps. This interview question was designed to also be a great question to learn that stack with, but I would not expect a senior programmer would be able to do so in one weekend -- hence I gave programmers as much time as they wanted, in many cases, several months to play around with the stack and see if they enjoyed it in their spare time. Some engineers even asked for some budget to play around with it in Google Cloud and I provided it.
Or should we just code each service in Rust or Elixir or Node or F# or whatever language stack a candidate prefers that day?
Also remember candidate are usually not only applying to your job. They will easily have 3-4 of these going in parallel and with each taking up that much time is untenable.
You're aware that a senior engineer will take a trivial amount of time to get up-to speed in a programming language paradigm they are familiar with.
And, an entry level engineer will have the same learning curve--their core source of inexperience is not the programming language.
So, no, the technology stack would be a very low signal for me if I were hiring.
So... we actually ask them to demonstrate to us that they can learn new stacks quickly. That's the point of the question. If somebody doesn't know Golang or the stack directly, that's awesome. They can learn it quickly and show us.
I feel for you. It does happen a lot. A portfolio matters. References matter. But it's really nice that you give unknowns a chance too, and to do that you really do need a filter.
And it's a fairly trivial filter, since you're asking the applicant to build a proxy.
I don't think you'll get too many applicants who don't already know the stack though because... think it of from their perspective: if someone is applying who knows that stack already, then they can beat the others to the punch, so taking 2x as long to learn the stack is likely to be a waste of time. I'm not sure how to make this "fairer" so that you really do measure learning time rather than comfort level.
Your filter is likely better at finding experienced Golang devs than experienced devs who can learn Golang quickly. In the short term this is probably very good for you, but the latter are probably better in the long run because people who learn quickly and well are more valuable than people who know a tool well.
It also adversely selects for something else.
In general it doesn't matter what specific technologies someone has used in the past, if they do the same thing. When you hire someone their first few weeks/months are going to be less productive because they have to learn your internal systems and even if you use AWS and they've used AWS, that doesn't mean it will be the same subset of AWS. Whereas by the time they've been there a year they'll have learned the relevant systems regardless of whether it's cosmetically different than the one they used before.
Selecting for the exact set of technologies your company currently uses is going to filter out attractive candidates. Worse, it selects for a monoculture. The last thing you want is to spend two years building something in Golang that exists in the Rust standard library because none of your engineers are aware of that.
And it's simple enough to ask the same question without explicitly specifying specific technologies.
Do we use it as a filter? Only to the extent that it filters out candidates who don't want to put in the work, or who can't actually code.
We're no longer hiring for this specific role, but I figured writing a blog post to share the solution to help people who for one reason or another, find themselves needing to implement a highly scalable UDP backend, they would now hit this article and at least have some pointers about SO_REUSEPORT and so on as well as increasing socket send and receive buffer sizes, ulimits and so on.
btw. This question actually comes out of a core part of the Network Next backend, which was problematic for us and a thorn in our ass in various ways over a period of 5 years. It took us an extremely long time as engineers not familiar with everything needed to solve this question, to correctly solve it and scale up past 25M+ clients. As such it feels like information worth sharing, after all is said and done.
But you want a senior applicant to solve it in a weekend. What you'll get is people who are already somewhat familiar with these issues to apply. Chances are such people are indeed senior and can think their way out of a paper bag. In fact, you'll get very valuable people if you get them at all. I hope you understand just how valuable.
This is honestly a toy problem for people with the direct experience that we are looking for, and there are a bunch of levels of implementation according to the level of the candidate that result in a pass. We've used this successfully to hire candidates who don't even know golang, and to filter out engineers who bluster and just wave hands and say "this is easy just XYZ...". OK cool, show me.
Like all good interview questions it has an infinite number of solutions and it's primarily used to determine if the candidate is "smart and gets things done" AND as a test to filter out candidates who talk the talk but can't walk the walk.
YMMV. It's being shared now primarily to help people who for some reason or another need to code backends in UDP, once the solution is published, at least there will be something that pops up in the google search results and gives them some hints.
Jesus. Next time I won't even bother trying to help people :)
Mind you, I do like this sort of interview question as a question not so much as a fetch-a-rock exercise.
The thing is that when you ask this sort of question you can very easily explore the candidate's ability to explore the solution space and show off what they know, but there's an end time, while with a task it can easily become a time sink, especially if the candidate chooses to go above and beyond and... fail to make the cut because in that time you picked a different candidate.
TFA seems much more interested in fetch-a-rock as a filter than the specific problem as a way to gauge how the applicant thinks.
- unikernel or user-land NIC driver and
network stack
- async/await or continuation passing
style to reduce the per-client/request
memory footprint
Translating a UDP packet into a HTTP request and back is reasonably easy. Yes, maybe one can do that in a coding interview with some pseudo code. But scaling it and making it reliable is yet another dimension.
Any candidate would need to understand that a single UDP socket itself would probably already a bottleneck for just running this on single machine, and figure out how to increase concurrency there. That's not easy - the amount of engineers knowing how SO_REUSEPORT works and when it doesn't work is low.
After that you start to dig into how you can actually spread load between hosts. Would an answer like I "I hope my cloud provider solves this for me" be good enough? Probably not. If it actually is, do candidates have to both understand the cloud providers native APIs and Terraform (mentioned in the blog post). Seems pretty unnecessary, terraform is just one tool out of the myriad of tools which can be used to configure cloud services. Not everyone will have used it before. Or would it even expect candidates to do a long writeup about the pro's and con's of client-side load balancing?
Are applicants required to talk about upstream connection pooling? Describe and implement a CDN like multi-tier architecture?
Last but far from least is that the requested architecture is very easy to misuse for denial of service and amplification attacks. Just being able to describe how to monitor and mitigate that is already a very very hard task, that very few specialists have worked on so far.
It's very fuzzy what would be good enough if this is a "homework task". At least in a synchronous interview the interviewer could give feedback that they are satisfied. So I think in a synchronous interview the question might be ok - but there will probably just be time to either talk about coding or about system architecture.
There are definitely degrees of correctness and completeness and depending on the candidate experience and level, certain solutions are acceptable. For example, a totally naive implementation in golang that doesn't quite hit the scalability requirements would be a good conversation starter and would pass a mid level or junior candidate.
A senior or above "badass" candidate would be expected to hit the scalability requirements.
An incredible candidate would teach us something new about this problem that we don't already know.
Over weekend? Either I'm misunderstanding it (are you supposed to write your own NAT-pinning solution? TURN?) or this sholudn't take more than couple of hours.
"You are tasked with creating a client/server application in Golang that runs in Google Cloud. The client in this application must communicate with the server over UDP.
Each client sends 100 requests per-second. Each request is 100 bytes long. The server processes each request it receives and forwards it over HTTP to the backend.
The backend processes the request, and returns a response containing the FNV1a 64 bit hash of the request data. The server returns the response it receives from the backend down to the client over UDP.
Implement the client, server and backend in Golang and Terraform such that it scales to more than 1M clients. Provide estimates of the cost to run the solution each month at a steady load of 1M clients, as well as some options you would recommend as next steps to reduce the cost.
IMPORTANT: When you load test this system, make sure you are communicating between the client, server and backend using internal addresses to avoid egress bandwidth charges."
1) The arrival rate is constant, vs the more realistic Poisson distribution
2) The response fits in a single packet
3) Requests are consistent size.
I don't think most companies are going to have the resources to take advantage of QUIC and HTTP/3 for a while.
While head of line blocking is theoretically gone, most programming is based around ordered delivery.
I see lots of projects doing single streams to minimize complexity, but that severely reduces large asset transfer.
I am sure the FAANG crew will gain a lot of advantages, but tooling needs to be there for the rest of us.
Obviously some will take advantage of it, but it is a shift in responsibility to the application and dealing with window scaling and out of order delivery is expensive.
IMHO, UDP is simply a target because legacy systems would make a new option that was superior difficult if impossible to implement.
As I have supported voip and streaming over the public Internet, I admittedly have a jaded view.
But I am not looking forward to supporting the quirks of every client when it was easy to hand that off to the OS.
Hopefully I am wrong, and SWEs will be good at dealing with ood and random failures and standards will offload the cost of implementing what TCP gave us for free.
But I don't think people realize how easy reliable, ordered delivery makes programming.
Edit: also i personally find TUN very convenient.
Sending the http request is easy, stuff incoming requests into an sqs queue from UDP side then have a lambda eating queue entries and sending them to the client. Because it's udp you don't need to keep endpoint state.
If api gateway doesn't do UDP you could network load balance the udp to a ecs autoscaling group with your udp listener in it. You'd have to figure out what metrics you need to trigger autoscaling. You could also just do route53 dns routing and have some oversized instances as your handler; it depends on your budget + how much time you have.
I'd probably stick them in redis, since it's fast and seems to be able to handle some ridiculous number of simultaneous clients. They don't need to be super-persistent since the map can be rebuilt next time a packet comes in.
Assuming the UDP packets come in on a timely basis you should be able to connect to the client relatively well. NAT tunnels expire in a minute or two generally, so as long as processing doesn't take more than a few seconds the NAT mapping should be fine. I'd configure the client to send every few seconds, just so if we fail over we can still talk (which latency = those seconds). I'd have to know more about the time envelope that's required tho.
Shouldn't take more than 2 weeks to implement/test in node. Learning go might add a week or two.
IPv6 makes it easier and harder. I'm not sure what happens if the ipv6 address isn't reachable due to firewall ingress rules, but I'm pretty sure you can't poke a hole in the NAT that doesn't exist. I would like to think that most home users prevent ipv6 ingress, but I have no way to test that.
It'll be interesting to look at the solution and see how they handled that.
https://blog.cloudflare.com/author/marek-majkowski
This question is designed to help find those people, in addition to people who are really willing to learn the stack with no experience and who are excellent problem solvers.
Things like iodine tunnels can trip IDS/smart-routers/firewalls to flag hardware for malformed traffic.
Seen people try this before, and it ends rather predictably unreliable/slow.
There are better languages that allow much higher performance ceiling for this kind of task.
Also seconding notes regarding encryption and the use of QUIC.
Either Rust or C# would be much easier in terms of vertical scaling strategy to reduce replica count and increase individual pod utilization - both languages offer better tools for scaling on individual many-core instances, reaching much higher throughput.
Edit: (though in wider context naturally the switch away from existing stack in a company would never make sense as long as it is a compiled language with sufficiently good average performance)
One company did Pion WebRTC + https://www.dpdk.org/ and is now doing 4x throughput. They tried switching to Rust (and still doing userspace networking) and it didn't make a big difference.
That brings me to another point: HTTPS isn’t preferred because it’s the defacto standard now, it’s the secure bit, the “S” in HTTPS, which makes it a better option than using plain HTTP over TCP.
By going completely bespoke means you’ll need to implement your own protocol that implements some level of security and redundancy in UDP communication.
Let's say I don't care about the initial handshake latency, have disabled nagle and have no packet loss. Is there still a measurable latency difference between UDP and TCP? It would appear that there is (based on yours and other comments) - but I don't know why.
See eg here: https://cloud.google.com/blog/products/networking/using-netp...
RTT of acknowledgements shouldn't normally enter into it, no? You can send while previously unacknowledged data is in flight (as long as congestion control state allows).
It'd be fun, but I'd rather be figuring out how to make box joints on my table saw. I do want to see how the solution code handles IPv6 client ingress issues, though.
Where is info about encryption, for example - very few libraries support DTLS or any UDO compatible encryption at all.
This gives everybody on Hacker News time to come up with their own solution, if they want to try.
When I asked this as an interview question, I gave candidates permission to implement this without encryption (although asking about encryption is definitely +1).
Note that the response size is significantly smaller than the request size (intentional), so you don't have to worry about being used in UDP amplification attacks. (Being concerned and asking about UDP amplification is another +1)