How Netflix really uses Java
infoq.com
infoq.com
How Netflix uses Java - https://news.ycombinator.com/item?id=37829395 - Oct 2023 (196 comments)
(Reposts are fine after a year or so, but when a story has had significant discussion within the last year or so, that's what counts as a dupe on HN. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.)
I can imagine that you need to keep track of all these services, along with information about dependencies, code version, API version and others.
Of course, monolithic services also require these this sort of management, but it should be less complex.
Some companies spend a bunch of engineering time making their monolith easier to manage (Google/Facebook) and some spend it making microservices work (Netflix/Amazon). Neither one is better, just different paths.
Where you get into trouble is trying to go either route without spending the resources to make management better.
> I can imagine that you need to keep track of all these services, along with information about dependencies, code version, API version and others.
This is the software Netflix wrote to do that:
Google has Borg.
They literally invented Kubernetees to make their microservices easier to manage.
---
Don't confuse mono/microservices with mono/micro repositories.
Someone just looked at what devs came to realize worked well, slapped a name on it and it blew up as a hot thing.
Since then, I've seen so much wrong done in the name of 'microservices'. It shouldn't be some aspirational goal to work with them.
Also, the fact that the word 'micro' is in the name makes the cargo culters split atoms to reach some tiny nirvana. The size isn't the point. The real point is in having the system/component boundaries closely match the problem domain, and that a single group 'maintains' it.
We call it distributed monolith. It has the worst things of the both architectures.
They should’ve been called domain services or something.
Some famous (Google, Meta, Microsoft IIRC) went into the "one repo to rule them all" approach, with requisite tooling, culture, and expectations to enforce that.
Netflix, Amazon (and I suspect a great deal others) went the micro services route.
As far as I can tell, its about whether or not you can reasonably maintain a good mono-culture around development practices and tooling or not ends up being the main question.
Now, there are aspects of micro services that cross these boundaries, but I strongly believe based on my industry observations that this is where it grew out of. If you think about how these organizations approach developing software it can give you a pretty good idea of why they ended up where they did on this frontier.
Are they still being developed internally and the foss repos are basically without the secret sauce? Or are these projects "done"? Did Netflix move on to replacement tools?
I'm referring to things like hystrix and .. a queue system I can't recall right now. This was a long time ago but that queue system was a bit of a rough roll out and IIRC I had to make a fair amount of code changes to get going.
Not biting the hand that feeds me, just curious about how netflix treats projects like those. I'm usually extremely apprehensive about deploying archived things but a few teams wanted to use them. I don't know if things are archived because they're done, perfect, terrible, cooked, deprecated, insecure..
edit: Oh I hadn't looked at it in a long time, the hystrix repo explains it well on the readme. I'm not sure if the other project I used did that.
That looks like an interesting solution to support asynchronous I/O without breaking all the APIs, and having the async/await mess that C# created.
> ...the async/await mess that C# created
Does C# have the same issue Python does with accidentally calling blocking code? In async Python - which I mostly quite like actually - it’s terrifying to bring in a library because if you’re unlucky it does some blocking network call deep inside that stalls your whole app..
> Does C# have the same issue Python does with accidentally calling blocking code?
For APIs, console apps, and such, it's not a concern for actual parallel-concurrent code. Of course, if you write a standard non-parallel `for` loop that has blocking code, it's going to block in a console app as well if you don't run it on a thread.
But I think that once you do enough JS/TS or C#, `async/await` doesn't feel very onerous if you have the basic concept.
This historically allowed it to take a lot of punishment but also led to many legacy codebases to be really shoddily written in regards to async/await, where it would take some time for threadpool to find the optimal amount of threads to run or devs would just set a really high number of minimum threads, increasing overhead.
In .NET 6, threadpool was rewritten in C# and gained the ability to proactively detect blocked threads outside of hill-climbing algorithms and inject new threads immediately. This made it much more resilient against degenerate patterns like for loop + Task.Run(() => Thread.Sleep(n)). Naturally it is still not ideal - operating system can only manage so many threads, but it is pretty darn foolproof if not the most foolproof amongst all threadpool implementations.
As of today, it is in a really good spot and with tuned hill-climbing and thread blocking detection you would see the .NET processes have thread counts that more or less reflect the nature of the work they are doing (if you don't do any "parallel" work while using async - it will kill most threads, sometimes leaving just two).
Function coloring can be handled by just blocking on an async function. Though the reverse takes some planning.
It's about mobile apps, but I talk about backend at great length, especially since my background is Java. The book is called "opinionated" because I cover Quarkus (https://quarkus.io/), monolith, Fly.io, and no K8s. No fancy stuff to pad your CV, just to get things done the simplest way possible, with the least headache in maintenance (I'm lazy).
But when I evaluated them, Render didn't support loading my own docker image, so it's incompatible with my build process and CI/CD.
The book is a living document, and I always try to find easier/cheaper/faster/better ways of doing things, so I'll definitely check Render again!
I think there is a great ecosystem for building large systems with go (check out cncf for a ton of go projects targeting the cloud), but a lot of big tech companies open source their java frameworks, tools, and systems. If you are looking for something really specific there may not be an equivalent in go.
I think go is great, and it is close enough for me to continue to choose it for this kind of work.
Go will give you lower memory usage, less abstractions, and the ecosystem has fewer observability tools.
That’s some shade.
Seems dumb, but I have 2 little kids and they both like to "pause" whatever show when it's time to turn the tv off. So after the first one does it, I have to do this dumb little dance to get the app to load again so the second one can do it.
My son also want to do this. Pause. Then turn the TV off. Maybe they believe the show will keep running with the TV off unless paused?
Better safe than sorry. ^S
You shouldn't have to buy the more expensive Roku 4K just to have a device powerful enough to run its own interface smoothly. Granted it's only $10 more but had I known the UI was sluggish I would have paid it from the start even though I don't need 4K. But now I've bought both.
Disney has _how_ much money to spend on their streaming services technology stack and they just... don't?
Just feels like they'll piss on the customerwhen given the chance because people will still pay for that sweet Disney juice regardless.
But now I'm gonna hear the comment every time I wait for it to load :P
Except for YouTube, that's just leagues ahead of everything else.
It only happens a few times per year for me, but I've never had it happen on another service.
For example, Disney is actually pretty terrible compared to Netflix + Prime. Worst part of the service is when it frequently loses where you were in a program. They also made it really hard to get to 'continue watching' for a long time.
Another one that has annoying UX issues is BBC iPlayer. If you click on the latest episode of a show that you're watching in 'popular', it won't resume where you were in the episodes, it will play that episode. Very frustrating. While technically correct, that's not what most people want to happen.
The service type is a bit different, but IMO Peacock is the worst. None of the others are as consistently slow for me.
Focus on your own house. At the end of the day both Disney and Hulu work more than fine for me( I'm guessing it's the same team ).
Netflix had what, a decade head start. I'd expect them to of perfected streaming.
RX has some interesting ideas, but from a practical standpoint, at companies not netflix, it just doesn't work
But you're right, the same things can be accomplished in other ways with less up front effort. But where it really shines is when you hit massive scale. Having RX and its concepts in place ahead of growth makes the growth easier.
I'd say a decent chunk of engineers just were not competent enough to get it in the first place and then there were those who just refused to put in the time to learn.
i.e. there are many engineers who I truly believe do not even want to put in the effort to be competent in the first place. That or false competence takes precedent as well.
Promises are all over the place too. I understand why someone thought this was a good idea ("I'm doing the scalable thing!"), but in the end it appears completely unnecessary (e.g. doing a blocking get call in all uses).
The problem with Java isn't Java. The problem with Java is that it lets so many people hold it wrong, and the Java ecosystem encourages it.
I've been coding in C recently and really enjoy the limitations. Usually there's only one way to do something.
I shared with the team and people complained about the complexity… so I ended up ripping it out. What people wanted was a test user creation service that worked well and could be maintained… not one that was the most efficient.
I was using RxJava initially because it was being pushed by the platform team and coupled pretty well with the GRPC interfaces we used internally. However, the rest of my team couldn't/didn't want to wrap their minds around the concurrently functional tasking that it brought (which was understandable).
I didn't want the code to have a bus factor of 1 (me).
This is essentially the same tension between static and dynamic typing.
No matter what, all bug free code is complex. Correct code can be written in any paradigm.
But productful code ships with bugs.
Does test user creation need to be bug free? I don’t know. It feels like something where productful test user creation can ship with bugs, even if in reality your test code harnesses should not be buggy.
You can step even further back and ask: when you have essentially unlimited resources, why ship buggy code at all? Netflix has unlimited capital, and it isn’t in a hurry anywhere. Engineers should never choose productful over bug-free. You learn nothing (Netflix’s product problem will never resemble your startup’s product problem) and you forfeit the opportunity to learn (ReactiveX is a good, expressive paradigm).
This is a startup forum, is this true for startups? IMO, yes. Your startup is going to fail. You will not learn valuable product lessons from the journey. I mean people certainly pretend and say they do, but even in the situation of success, Brian Chesky still hasn’t become Walt Disney last I checked. But once you have enough money to be employed for a year, as an engineer, you know, fucking learn how to do something the right way, bug free and performantly.
So I agree your coworkers are fundamentally making an error.
I disagree with this. It mentally changes the thought processes when debugging, which was the biggest issue that was brought up.
> Does test user creation need to be bug free? I don’t know. It feels like something where productful test user creation can ship with bugs, even if in reality your test code harnesses should not be buggy.
It should be bug free at Netflix scale. My service created more customers daily than Netflix signed up monthly. It also sat in the middle of Growth. People use it to gate deployments. It needs to work flawlessly (or at least consistently).
Are there any talks or blog posts on how y’all wired this up, lessons learned etc?
Sorry no talks.
In the second codebase, things are a bit simpler. The architecture allows for more readable code and far fewer cases where multiple streams might introduce loops or race conditions due to being in separate files and initialised at different times. We felt the benefits of a v2 build ;-)
I’d say RxJS is a great tool to help respond to multiple concurrent data streams (e.g. user interactions happening at the same time as video playback). If you’re not dealing with a lot of realtime behaviour, you probably don’t need it.
In my opinion the complexity ramps up massively if some consumers want a reactive streams pushing data to them and other consumers prefer to request a current static value of the same data. Watch for this and experiment to find comfortable solutions or acceptable compromises.
If you think you need RxJS, take a look at Redux + Sagas and consider a choice between the two. I prefer the rigour and dev tools for Redux + Sagas over the freedom to be hairy with Rx. But observable streams are a lovely abstraction.
Even if I do a hard reboot of my TV and don't start the app, my TV is still happily resolving away. I'm gonna have to set up a local cache or something just to save on my NextDNS quota.
Edit: I just asked Netflix support about it. Worse than useless, just kept telling me to either try a different internet connection(no idea why), then they told me to take it up with my ISP and immediately closed the chat. I'm gonna wait an hour or so, open another chat, and tell them my ISP said this was purely a Netflix issue, I suppose.
Ah, how many k/second?
> sometimes several times a minute
WTF?
We're talking 0.00003 thousand requests/second? Most apps do several DNS queries per minute. Otherwise every DNS update would require >1 minute to propagate. Slow DNS updates is a massive pain in the ass.
FWIW, ISP DNS servers and cheap home routers are both capable of caching DNS responses, updating the cache when they get a query for something that's about to expire, and serving thousands of queries per second.
> then they told me to take it up with my ISP and immediately closed the chat
This is the correct response. DNS is usually provided by the ISP via DHCP. If their server can't handle "several times a minute", the ISP would be at fault 100% of the time
I explained to them that I'm not even using ISP DNS. They still told me to talk to the ISP. Still think it was the correct response? It was obvious they didn't even know what DNS was, they were just slavishly following a script that said DNS resolving issues -> ISP. They didn't even grasp what I was reporting to them.