TLDR: they saved 400 to 1000 cores by switching off the switching off of the GC during requests.
The fact that they are running Ruby and are spending 1000 cores on GC is o_O.
The fact that they are running Ruby and are spending 1000 cores on GC is o_O.
I mean a reactive application needs three things:
- a fast network connection to persistence. If you can't talk quickly to your database, latency eats your response time no matter what you do.
- Maybe some LRU caches across requests, but caches inside an application tend to become a maintenance nightmare quickly. Stales reads are fun, and stalled cache writes are too.
- And beyond that, it should be possible to handle most request data by stack allocation/deallocation with references to a cache version if you have a cache with little overhead, depending on your security requirements.
I understand the ease of use and simplicity of a GC'd language, especially given the time GC'd languages came around big time. But I've always been wondering if you really need a GC if you think properly about request and object lifetimes.
I just write crappy code for microcontrollers. Over 30 years I've seen the amount of ram available for stack grow. From maybe 20-30 bytes to a couple of k. That said it appears to me that stack allocation is severely under utilized for historical reasons. The idea that stack space is a precious resource. Which doesn't fly on machines with 10's of gigabytes of memory.
Think programs that put vast quantities of ephemeral objects and short strings on the heap. Using call tree analysis would allow you to put a lot of that on the stack resulting is much better performance and bounded latency.
> Moving objects from the heap to the stack
> Many sources report that escape analysis moves Java objects from the heap to the stack. As Aleksey Shipilёv points out in his article about scalar replacement, the JVM does not do this implementation. It's just a misconception. But it's an interesting one, and I wonder why the JVM doesn't implement it.
In my defence the sentence I quoted isn't so much deliberately misleading, as outright false. It essentially states that 'new' always results in a heap allocation, which just isn't true.
A profoundly unhelpful writing style, going out of its way to punish skim-readers. I'm reminded of this TV Tropes page: http://tvtropes.org/pmwiki/pmwiki.php/Main/ByNoIMeanYes
From that article:
> Java doesn't store any object in the stack. It does store local variables on the stack: primitive types like int or boolean. It also store the pointers to objects on the stack. But it doesn't store the objects themselves on the stack. Anything you create with new is always created on the heap.
Is that true? For years now there have been articles from serious sources discussing JVM escape-analysis-based optimisations.
Is there something mistaken in the analysis in this DZone article?
https://dzone.com/articles/do-not-let-your-java-objects-esca...
How about this StackOverflow answer, which even goes into the detail of distinguishing escape-analysis, stack allocation of objects, and object deconstruction+scalar replacement:
If stack allocation was really done, it would allocate the entire object storage on the stack, including the header and the fields, and reference it in the generated code. The caveat in this scheme is that once the object is escaping, we would need to copy the entire object block from the stack to the heap, because we cannot be sure current thread stays in the method and keeps this part of the stack holding the object alive. Which means we have to intercept stores to the heap, in case we ever store stack-allocated object — that is, do the GC write barrier.
Someone told me that the team at his previous job with a "large cloud provider" based in Seattle was told not to go above 30% CPU usage, at which point they'd buy more hardware. I'm sure not all teams there do things that way, but, coming from Google, 30% (95th percentile over a week) is very low.
consider a deployment (deployment app) update it will first create a new container and than kill the old one, which means depending how much memory you request at container creation is the least minimum you would have on spare.
if you schedule java apps with at least 1 Gi heap you would at least need 1Gi spare capacity and that is just with one pod/container (it will be worse if you need to do blue/green deploys, since you need the same capacity as your live cluster)
Even then, deployment updates don't necessarily need to surge above their replica count. You can also configure them to terminate X replicas at a time before bringing up new ones. At Google, all teams have Borg quotas, so it's not unusual to max those up by running as many replicas as possible. During updates, Borg does not allow an user to temporarily oversubscribe their quota (unless you're changing replica count and replica footprint at the same time, but that's another fun story), so it will always take down Y tasks first.
But as a PM I have a different question. Which is the impact of the response time of Github on the user experience? 25% reduction on an average of 2 sec improves greatly the user experience, on a 200ms for a service that has basically no competition is just a marginal improvement, on a 50 ms average is not noticeable.
How is it distributed across multiple operations? Improving the latency of loading a diff is probably more important than the latency of approving a PR. In one case I'm trying to work, in the second I'm done with the work.
Now, my experience with GitHub is that it is already reasonably fast unless I'm doing something stupid. So the CPU utilization is a good metrics because 10% less CPU means 10% less server I need to pay and that goes to their bottom line (I don't know their economics to decide if it is substantial or not). The 25% reduction in latency is just icing on the cake...
Isn't that exactly the opposite of what the article is saying?
> An OOBGC is not really a Garbage Collector, but more of a technique to use when deciding when to collect garbage in your program. Instead of allowing the GC to run normally, the GC is stopped before processing a web request, then restarted after the response has been sent to the client. Meaning that garbage collection occurs “out of band” of request and response processing.
> This graph shows the difference in core utilization before and after OOBGC removal. In other words “number of cores used yesterday” minus “number of cores used today” ... We saw a savings of between 400 and around 1000 cores depending on usage at that point in the day.
So it sounds to me that by switching the GC on during requests they saved that many cores.