Hacker News Reader

Comment by hobofan | Hacker News Reader

> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 machines running at 35% CPU usage on NYE 2015. The response time was < 5 ms at 95th percentile, and < 50 ms at the 99th percentile.

> Geofence lookups are required on every request from Uber’s mobile apps and must quickly (99th percentile < 100 milliseconds) answer

For a 100ms total budget, a ~50ms 99th percentile for a single microservice doesn't sound like something to boast about.

View on HN

I don’t read this the way you do. I read it as their service had to answer under 100ms and they made it run under 50

If that's 40 bare metal machines... assuming 60ish HT cores, and 35% CPU... that's not that earth shattering. We do that in Java :)

Don't sell the JVM short. Using modern concurrency models (e.g.: Vertx) it will outperform go in throughput and latency.

I've changed my opinion a bit about Vertx combined with Java.

Personally I tried to push it for several years, but most Java developers (and I mean 95% in my particular case) seem to start resenting it over time.

It's hard to write good services in Vertx, mostly due to its asynchronous model combined with Java's verbosity and boiler plate.

Many teams have junior developers and IMHO it's simply not safe expecting them to write production grade services (albeit simple) mostly on their own. It can be done, but it drains the rest of the team. A more productive Java team would have used something like Spring Boot.

Also, with any technology, there are some gotchas .. and with Vertx they are much harder to figure out. In the end we changed to another Java framework.

I suspect that Loom Fibers are going to offer parity with golang in this area soon enough.

Loom seems very interesting. Do you know how far off they are currently?

Nope. They 'work' in a loose sense (you can run production-grade software like Jetty on top of them) but I don't understand very much about the current state of the project or what would be required for it to ship in a regular JDK release.

This is newest update on project Loom:

https://www.youtube.com/watch?v=lIq-x_iI-kc

My sense is that most people moving to Go are coming from Node and that explains their excitement about the performance. Not many upsides when you’re already on Java.

I mean there is a definite performance gain from Java to Go, although I would agree most people probably wouldn't hit it - but if its just as easy to write, might as well use the better language. On top of that Go makes concurrency so much easier to factor in and program with, which in the backend server space is a very important asset over Java.

As an interesting addendum, I found this to be a really interesting resource when comparing languages at a very base level, as it is very well written and you can actually look at the source code/ thesis papers for all of the implementations: https://github.com/ixy-languages/ixy-languages

> I mean there is a definite performance gain from Java to Go

I don't think thats true.

The GC isn't very good yet, but brute-force sequential code can keep more state on the stack where it doesn't matter. (Just please codegen that instead of expecting everyone to read it.)

Go's GC is better than most of the Java GCs except the new ZGC and Azul's proprietary one. JIT makes some Java code faster. Ultimately, it's more about programmer expertise in a language rather than the actual language.

It's all relative. Go is optimized for pause times and beats all other mainstream runtimes for that. JVM GC is very configurable and can be optimized for throughput, pause times, heap size, fragmentation, or whatever you want, and can beat Go's for pretty much all of them except pause times. Both have very smart people working on them and the techniques are well established, so it's about tradeoffs. (An oldish post but I think still relevant: https://blog.plan99.net/modern-garbage-collection-911ef4f8bd...)

That said, Go's focus may be the right one for many use cases. On a high throughput service where you would assume you want to optimize for throughput, 100 ms pause times can wreak havoc because they're unpredictable and can cause work queues to explode and such. This isn't easily mitigated by load balancing. Whereas "less efficient" GC is at least predictable and you can just add a server to balance that extra work.

There are dramatic memory and startup time improvements. Much less so for CPU once everything is up and running.

Is that still true after tuning the JVM GC for a low memory footprint? I'm genuinely asking, I'd like to see an article on the subject. I've read in the past that the default settings are geared towards long-running processes and trade off memory usage for higher throughput, as in general there's no free lunch with GCs. Better memory usage always implies worse performance (and viceversa), just like a higher throughput increases pause times. If Go has lower memory usage and lower pause times, I'd expect it to have lower throughput than the JVM GC.

There's two more variables you're missing here, besides "memory" and "pause times" - how well matched to GC the language is, and how advanced/good the GC is. Some languages might just be fundamentally easier to garbage collect than others - a simple example might be that of a language with no references and no threading, where the GC is going to be extremely simple. And GC quality is certainly a thing, since I doubt major companies would spend tens/hundreds of millions of dollars producing GC 'advancements' that just move a slider back and forth.

Yes.

Every object in Java has something like three words of overhead; also since it doesn't have value semantics (yet) objects are typically allocated out-of-line, so an ArrayList makes a linear number of allocations, whereas a Go slice makes a constant number. Plus the binary sizes are typically much smaller; not really an expert but our observataion at work is non-trivial Java services allocate a lot of memory through classloading / JITing whereas a Go binary will typically be very small.

Basically, agreed on the GC tradeoff, but the higher memory footprint of Java mostly comes from other areas.

> Better memory usage always implies worse performance

That isn't really true. Better memory usage implies better cache-friendliness (and possibly better locality too).

I hadn't seen the network driver comparison before, interesting. Go does really well here, maybe because this is the kind of job is was designed for originally? I'm not sure. For the kinds of things I generally write, this is usually a good way to compare something approximating real world performance: https://www.techempower.com/benchmarks/#section=data-r18&hw=...

C, Java, C# and Go all appear at the top there so it's basically a wash, +/-5%. I still think the main perception of Go as "the fastest option" on places like HN is because people are coming from some of the slower languages (Javascript, Ruby, etc).

Footnote: I don't think there's anything wrong with using a slower language, a lot of them have higher productivity overall. I also just happen to prefer programming in Java, so it's overall the best choice for most jobs for me.

Go has a little syntactic sugar for the same green threads other languages have, but it's missing a lot of immutable and thread-safe data structures.

> might as well use the better language

Which would be Java in this case.

Why is it that every Java application I've encountered in my 20 years of corporate IT, has been a performance pig? Is it just a case of bad developers, making shit code? Is it difficult to make Java perform well, but when it does, it shines? I can't have accidentally interacted with ONLY the crappiest Java apps in my tenure.

I have exactly similar experience. At this point I feel Java programmer/developer is misnomer. Most of the time they are 'framework' fiddlers. Be it J2EE, Spring, Spring Boot (nowadays) Vert.x and so on. When I talk about an http server, the only frame of reference in their mind is Weblogic, Websphere, JBoss, Tomcat etc. The stuff they work with is 80% auto generated and they are quite proud if it.

So few application I own have about 200 LOC of business logic and 10K LOC Spring Boot fluff along with some 60 jar files which I am not sure what they really do.

It's possible to make bloated crap in any language.

But enterprise programmers making internal software have little incentive to make their programs sleek and fast because they 1) have a captive user base who are forced to use their bad UIs, and 2) there aren't many users so you don't really need to optimize for minimal server use.

From the eyes of management, a good enterprise programmer can take a request from start to beginning and fulfill a business requirement in a reasonable amount of time. Architecture, UI, future maintainability, speed, security... that's not really on their radar at all.

There are plenty of managers who do not understand or respect feature creep and technical debt, so spending time refactoring is useless and opaque to them. Developers have no incentive to write good code, and bad developers who can talk a good game get hired on, so the codebase spirals into an unmaintainable sea of crap.

According to management, if you solve business problems, you're valuable to them. So if you care about good code, it's an uphill battle fighting for it against the other entrenched developers who really don't give a shit.

These programs usually happen to be written in Java because there are tons of Java developers out there, which is great if you don't live in a tech hub.

This isn't even internal stuff, all paid commercial applications. Oracle is notorious for it, but most recently dealing with SysAid(Helpdesk software).

Yeah, boring programming languages are criminally underrated. Fro instance:

> High developer productivity. Go typically takes just a few days for a C++, Java or Node.js developer to learn, and the code is easy to maintain. (Thanks to static typing, no more guessing and unpleasant surprises).

This is why the rest of the world doesn't use JS for everything.

> There is a lot of momentum behind Go at Uber, so if you’re passionate about Go as an expert or a beginner, we are hiring Go developers. Oh, the places you’ll Go!

If they did it in Java, they wouldn't have to recruit for programmers quite so hard -- they could pull them out of a hat, and fairly easily get a few wizards with two decades' worth of experience in it.

They also would have to sift through some really terrible programmers who have fully bought into the worst practices of enterprise-y OO development.

Is that really true? Honest question. Or is that just bias against things that are not new and fancy?

It's really true, in my experience. I managed a team in a company that used Spring Boot for services and batch jobs and Python for data and analytics (including ML). My team had its feet in both worlds so I had to hire for that. My experience trying to find programmers who could code and engineer first and not be slaves to the Spring framework was quite terrible in this context.

Another thing of note: the python side of the team could regularly move across and help with bugs or issues in our services and jobs, but the reverse was rarely true. In the 2+ years I had that team there was one person hired for the services side who could do it, and he preferred Go to either Java or Python.

I'm not sure what causes it—and I'm not even sure it's a bad thing, certainly it must not be hurting these folks careers too much, and maybe it's even helping them—but there's a certain path for developers that ends up leaving them helpless outside whatever narrow ecosystem they've grabbed onto. Usually it's Java or some Microsoft thing.

As someone who very much did not develop (as a person/programmer) that way, it seems baffling from the outside, but there's this whole world of programmers who work like that until they retire or are promoted into management. It seems bizarre to me but it must be working for them. Hell, they might even be the majority of all programmers. Bigcos, particularly the non-tech ones that nonetheless employ lots of developers, are full of such people.

Well, I think the difference between them and Silicon Valley programmers is that they don't constantly learn new technologies outside of work. It's just a 40-hour-a-week job.

I try to learn new things, but I don't do much programming outside of work. I could learn Node.js, for instance... but I've got other things I want to do.

Is that kind of what you mean?

Nah, I'm a 40-hour-a-weeker myself, I'm not sure it's strongly correlated with that.

Some folks, you say "do you think you can do [thing you're basically familiar with] in [language and platform you're not]?" and get a "yeah, probably lemme check it out... cool, compiler and language support's installed, see a couple tutorials here, I'll poke around and get something doing [subset of thing you need] then get back to you on timeline" and very likely it works out fine.

Others, you get "uh, I do (Java, .net), I don't... I don't understand what this is, is it a JVM language? Is there a jar for it?" and it's not really worth pushing any harder. And maybe some of them are deflecting because they just can't be bothered (mad respect), but most seem genuinely nervous and out-of-their-element at the mere suggestion of doing anything but their Java or .net thing (usually it's one of those) they're used to.

Though, again, the latter seem to get along just fine, career-wise. It's just a different sort of path, I guess. Seems really weird to me, but there it is.

Totally true, the whole reason half the popular programming languages exist is that their communities are overreacting to the bad/crazy things that happen in the other half of popular programming languages. Java and C++ communities have a lot of bloated ideas, and the languages themselves lend to that.

There has to be some reason that a language that benchmarks like Java is used to build so many heavy-feeling bloated programs that eat way more memory than seems reasonable, even from big "tech first" companies. Either lots of highly-experienced Java programmers aren't using patterns suited to good performance, or the language actually sucks a lot more than benchmarks/theory indicate. I suspect it's mostly the former.

It's been this way ever since I can recall. "Java's so fast you probably can't tell the difference most of the time". Well, OK, but, I can. So something's going on here.

At my old company, we would do hiring screens by having people send us simple code exercises in whatever language they'd like. Every once in a blue moon, I'd see a good Java sample, but most of them were bloated crap.

FWIW, the JavaScript samples were also pretty bad because it was before async/await, and they'd all get twisted into callback hell waiting for IO.

This might be related.

From pg: "...you could get smarter programmers to work on a Python project than you could to work on a Java project." [1]

[1] http://www.paulgraham.com/pypar.html

More popular languages get more early developers, so will be naturally predisposed to have objectively bad developers. You'll definitely find more bad Java devs than bad Haskell devs, but you'll also probably find a higher rate of bad devs in Java or PHP than you will Haskell or F# or something.

Yes, hiring for mainstream languages have this cost. It's true whatever the quality of the language and how many good people are available to work on it.

In my experience: yes, this is true. I’m not going to speculate about why, but I suspect this will eventually happen to Go as well when/if it becomes a more mainstream language.

I agree about performance. Java would be totally able to do it.

I feel this article is about as exciting those Java article describing 'Hello World' http server running under 256MB memory as earth shattering.

> This is why the rest of the world doesn't use JS for everything.

What?

I think they’re referring to static typing.

As some other posts have pointed out, their algorithm was holding them back not the language.

Well, switching from nodejs to Go for this use case was probably already a big win even if the algorithm didn't change.

Per machine, that is 4,250k rps. For any http service we write in Go, that lines up with our general sizing estimates before we start work on a generic new service. After we start to hit the service with load, we start profiling and go from there. Sometimes it gets faster and sometimes it gets slower, but it is usually in the right ball park.

With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.

The latency to human user is what matters. If I get an answer in under 100ms as a user 99% of the time, I don’t care whether 100 or 100K times per day someone else hits that 1% worst case.

It has nothing to do with your experience being 99% good enough and more to do with the size of the revenue opportunity that could be lost messing up 1% of requests.

As an individual you don't care as long as you aren't in that 1%, but as a customer-obsessed service owner I absolutely care if a bad experience is happening 100k times per day. If you're operating at internet scale you need to look at both percentiles and absolute numbers to assess customer impact.

Also keep in mind how percentiles compound when you have more than one service involved in serving a customer request. For example, let's say it takes 5 internal requests to serve an external customer request and each of those services measures latency SLAs at the 99th percentile. The customer request may only finish inside the SLA 95% of the time (99%^5)

> The latency to human user is what matters.

Agreed, which is why the most surprising part of this article for me was this phrase: "In our main data center serving non-China traffic".

The transit latency to the one main datacenter in the (non-China) world is a lot more significant than the server time here. (Over 50 ms just to cross the US; compare with their server-side latency numbers of 95%ile < 5 ms, 99%ile < 50 ms.) If they're serious about latency, their deployment is holding them back much more than their choice of programming language or algorithm.

Or maybe the client is not the user's phone, but some other service running within the same datacenter?

I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.

Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild.

Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.

And that would be a good signal and worth investigating. Why timeouts? I run a few services doing billions of daily queries. I watch max in addition to percentiles.

> and almost certainly had timeouts, the "max" would also likely be at the timeout

You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.

Sure but now your "100th percentile" would be nothing but the timeout.

Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else.

.... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.

Reply on news.ycombinator.com