Everything I Ever Learned About JVM Performance Tuning At Twitter
slideshare.net
slideshare.net
-XX:+UseCompressedOops converts many 64-bit references into 32-bit, trading a bit of CPU for a lot of memory
If you use a lot of threads, lower your stack size (-Xss) (no real method here, just drive some trucks over the bridge until it breaks, and that's your weight limit :P )
You don't want to swap! A good rule of thumb is to make your max heap (-Xmx) about 60% of total RAM if you only have 1 JVM running
Adaptive GC tuning algorithms can become unstable in long-running processes. I like to use -XX:GCTimeRatio for the throughput collector which effectively turns off the adaptive stuff. CMSInitiatingOccupancyFraction/UseCMSInitiatingOccupancyOnlydoes the same thing for the concurrent mark-sweep colletor. Keep the ice pick handy :P
i'd prepend the advise with "use large pages, Luke". Set Xmx to almost 100% of large pages, while the total of large pages set to 60% of the total RAM in your case or whatever your application specific is. For non-filesystem-IO application (middleware between DB and client applications) in my case the 14G on 16G machine works fine.
Despite this, Hotspot is very good and the stock JVM GC is very useful and it is reasonably straightforward to get visibility into the state of the heap and what is going on with the GC. I've never worked in a professional capacity that has anything comparable to VisualVM, although I gather Smalltalk and commercial Lisps have comparable offerings, and you only have to program in Ruby for a while to learn to appreciate the GC quality.
google docs: http://goo.gl/g7y02
A restart transformed a JVM with a fragmented old gen into a JVM with fresh old gen using X1 wall-clock seconds and Y1 cpu time/watts.
A full GC transforms a JVM with a fragmented old gen into a JVM with fresh old gen using X2 wall-clock seconds and Y2 cpu time/watts.
Have you ever compared X1,Y1 vs. X2,Y2?
My point was that a full GC does not need to re-profile the code, re-JIT the code and warm data caches, like a newly restarted JVM does. Sending a user request to a JVM to be run in an interpreter with cold caches is not good for user satisfaction, so a newly restarted JVM needs to be given mock requests to warm it up (like a script for PGO static compilation). A JVM doing a full GC does not need all that to become fully ready, and if it has enough free RAM to defragment quickly, the process should be much more efficient.
I suppose in theory you could use JMX instrumentation and tie it to your load balancer, but I've never heard of it being done.
(It's actually quite a good idea! hmm...)
Really? It had to? There was no possible alternative?
As often as I get the "fail whale" page on Twitter, I'm always skeptical seeing them present stuff like this and release code.
Maybe I'm just not grasping how large and popular Twitter is, but of the popular web services I use, Twitter fails more than all the others combined.
Though it was an interesting read, it seems suspect that they're having so many more problems than everybody else.
So true; i don't even know what a "facebook overload" looks like because i have never had one.
Here's why. Compare these two:
http://hg.openjdk.java.net/jdk7/hotspot/jdk/file/9b8c96f96a0... <-- Doug Lea's java.util.concurrent.ConcurrentLinkedQueue
https://github.com/afeinberg/lockfree/blob/master/src/lockfr... <-- my port of above to C++0x
https://github.com/afeinberg/lockfree/blob/master/src/hazard... <-- essentially an implementation of garbage collection that's needed to to work around the ABA problem
tl;dr Shared memory concurrency is surprisingly hard with manual memory management. Not impossible, not infeasible, not impractical. Just hard.
C++ is still a valid choice for many products: JVM isolates you from the underlying OS, the VM subsystem, and there are cases where the cost of garbage collection is prohibitive.
However, I'll argue that vast majority of a site like Twitter (or LinkedIn, another high-scale JVM powered property) is best served by runtime like the JVM or CLR. Erlang is another great option, but it's more of something you'll have _along with_ JVM/CLR and C/C++: Erlang's model is (highly efficient, well abstracted) concurrency with message passing -- which is awesome, but not a full substitute for shared memory concurrency -- i.e., it's a great tool for some jobs, but not others.
C++ makes more sense for things like a B+Tree implementation: I've been using a pure Java B+Tree implementation -- BerkeleyDB Java Edition, and can certainly mention the negatives of that approach.
On the other hand, look at something like the routing layer of Voldemort, multi-Paxos implementation in ZooKeeper, or (as an example of another high scale, Java based service) the modified Paxos implementation in Google's MegaStore, non-storage parts of Amazon's SimpleDB (written at Java at one point, in Erlang at another -- not sure what it's written in now).
I'll also argue that I'd rather use a less verbose and more functional language than Java -- and honestly, in some cases C++ far outdoes Java in terms of expressiveness. Go, Scala, languages in the ML family and Erlang (especially with tools like dialyzer and quick check, to get around the dynamic typing) are the right way forward for building highly concurrent user-land "systems-y" software (think more databases or distributed middleware than an OS) -- the parts where memory layout is critical and which tend to produce a lot of garbage can always be implemented in C/C++.
tl;dr Programming language choice involves trade offs, Java is too verbose, but garbage collected languages/runtimes have their role in building highly scalable applications or service
That said, since I've now taken the time to read your code, I have a suggestion that would drastically clean up the parts that are making the manual memory allocation feel "overtly manual": rather than allocating memory and then using the default placement new to construct your object, you could define a placement new/delete that operates over the allocator.
This not only would remove all of the reinterpret_cast<>s and sizeof()s from the queue code, but it would also allow a future modification where the queue was able to store values other than void *, while retaining exception safety (as when you template the Node by way of the Queue itself, its more complex constructor would suddenly be allowed to throw, and the memory would need to be collected).
You're also completely correct about placement new: I am working on a cleaned up version of this class, this was essentially a first pass to get myself more familiar with concurrency in c++0x. What complicates things a bit is that allocators are (much like all else in STL) meant to be used as class template arguments, which makes main separate compilation impossible -- hence the need of an adapter from an STL-style allocator to a pure virtual class. Separate compilation is why I also made a void* version of this initially.
I have a much cleaned up version in the work that will handle more than void . There's an implementation I call it ConcurrentLinkedQueueImpl that handles just void , that is compiled separately -- and there is generic version ConcurrentLinkedQueue that is specialized for void * (ends up just proxying the calls to ConcurrentLinkedQueueImpl), with the generic version (in turn) using ConcurrentLinkedQueue<void *> and placement new to hold any type.
Once again, the version posted there was a rough pass to get myself familiar with 0x concurrency constructs and hazard pointers -- the code is fairly messy.
[1] http://www.amazon.com/Art-Multiprocessor-Programming-Maurice... [2] Everyone should read this book cover to cover -- http://jcip.net/
http://hg.openjdk.java.net/jdk6/jdk6/jdk/file/b58af78ac79c/s...
That said, the main drawbacks are:
* At most, I can give the JVM about 18-22 gb of heap, out of that, at most 8-12gb (usually 10) of heap can go to BerkeleyDB's cache. If I give BDB-Je too little heap, I run the risk of it not having enough space for buffers (for cleaner thread, for scanning through the entries). If I give Bdb too much cache, memory pressure becomes an issue.
So now there's 10-12gb of memory (per machine) that's neither use for caching data by BerkeleyDB itself, nor used for the OS page cache.
* The index takes up surprisingly much space. In most cases, the index still fits in memory -- but it could definitely be smaller.
* Couple of implementation issues that are being addressed by the BDB-Je team and aren't really Java specific. I won't go into them.
That said there are many pluses to BerkeleyDB JE:
* Log structured B+Tree. Awesome for use with SSDs. SSDs also make the penalty of going outside the in-memory cache much less costly, a random seek for reads is 0.2 ms, and due to log structures design random writes -- not very efficient on SSDs -- don't happen. Additionally, log-structured design means less wear on an SSD. That said, I'd still strongly suggest using an SSD even with a conventional B+Tree (e.g., MySQL InnoDB) -- but benchmark your application first.
* Avoids overhead of JNI
* As I've mentioned in my comment, being on the JVM means you can spend a lot more time thinking about concurrency. The locking/latching design in BDB JE is very well made.
And when you go with C++ you've forsaken type safety (i.e. memory safety). That's a substantial thing to give up, IMO.
Here's an example. You allocate a chunk of memory as type Foo. You use it as Foo. You hand out references to Foo. Everyone is happy. Then someone says, "We don't need this instance of Foo anymore" and deletes it. Someone later comes and allocates Bar and the memory allocator correctly goes and grabs some of Foo's old memory and uses it for Bar.
All good, right? Well, except there's this other thread that is using a reference to that old instance of Foo. Suddenly you have a type safety issue at hand.
Now you can say, "Don't write programs with bugs". And indeed that is just about the only alternative.
I have been responsible for projects with on the order of hundreds of thousands of line of C++ (such as for a video game we were developing), and the only calls to "delete" were in the stack-locked reference counting implementation I designed, and a few optimized containers. It isn't even hard to think like this in a language like C++, and isn't highly different than a language like Haskell (you build abstractions from unsafePerformIO: you don't routinely code with it).
Sure, you might be less likely to crash; but you're far more likely to get silent data corruption -- and I know which of those two I'd prefer to see.
You're more likely to get silent data corruption without memory safety, than with it.
It has nothing to do with Java specifically, FWIW. Ruby, Python, shell script, C#, Haskell, they are all memory safe. It's a big church.
Well, sure. And that's true of C as well. The question is what happens when that memory gets reused.
I'm just going to have to throw up my hands here. The masses on HN are apparently proudly ignorant, and unwilling to learn even the slightest thing before wading in.
That's completely different from type safety.
It's not completely different from type safety. You're right, they're not the same thing, but GCs prevent a common form of type safety violation. With that said there are other techniques that exist that are orthogonal to GC.
At the same time there are problems, like lack of bounds checking, which can result in type safety issues.
My point isn't that you need a GC per se, but you do need memory safety. GCs happen to be one of the most prevalent ways to achieve it. Someone else in this thread noted that they build a complete runtime that guaratees memory safety and requires their devs to code against that runtime. That's fine too (although I think a lot less common than that poster might lead one to believe). It's almost like the memory safe subset of other popular existing runtimes.
Memory safety is not dependent on having a GC - if you have an appropriate runtime, a smart enough type system, or a simple enough language semantics.
It's independent of how you write or design your program. It's a property of the language + runtime combination.
(Folks out there downvoting me: you could do with some education too... Parent is a completely ignorant (in the best possible sense - easily fixed) comment.)
Yes: in a couple places in your software you will have a cast, but it is about as useful to complain about that as it would be to claim that the JVM itself has a cast operator in it; if the program code isn't using it, then the program code can be proven to be as safe as the runtime, and C++ does not remove your ability to make statements like that.
So no: I believe that your comment is "completely ignorant" (and rude, to boot).
Or threads.
That said, I'm having a difficult time figuring out how that could actually cause a type safety problem, given the same constraints (using a safe memory management framework: garbage collection or stack-locked reference counting). I am totally willing to believe I'm missing something, however; care to provide an example (that does not involve type casts or unbounded pointer arithmetic)?
Nor even is the parent of the post I responded to (as you might then say "well, I am arguing that, and you can't have both"): you easily can have both, as all that these people actually end up doing is brick allocating blocks of objects (either by using C# structs, such as the Stack Overflow articles we've been seeing recently, or doing a poor man's "column store", splitting the fields into arrays, which then causes cache performance issues).
So, as that parent post's parent pointed out, if you are going to go through hell to do that, you may as well do so in C++, as it will be a million times easier (even doing this for their existing managed objects in Managed C++ would have been easier, and it is unfortunate that they didn't evaluate that).
Therefore, with that performance and tuning argument totally removed from the rest of the conversation, I am focussing instead on the irritatingly strong statement "That's a substantial thing to give up, IMO."--a statement I very explicitly pulled away from the unrelated argument in the previous paragraph of the comment--which does not seem at all warranted given the situation, or the facts of these languages.
I mean, the entire premise of this argument is flawed... there are very few systems that actually /are/ type safe, and yet people seem to like using them. We don't even need to go to silly examples like Haskell's unsafePerformIO keyword: C#, a very similar language and one that has been being talked about a lot with respect to GC performance on HN recently (the Stack Overflow article), is quite clearly not type safe, as it allows you to type cast pointers using the "unsafe" keyword.
Of course, you don't have to like people using the "unsafe" keyword, and you can enforce that people on your team not use it; but then that's the real issue: if you are allowed to use the entire specified system, as opposed to restricting yourself to "the known safe subset", then almost no systems of note are actually type safe, partly because users wouldn't stand for it.
With this understanding, we can now go further: Java?... /not type safe/ (sun.misc.Unsafe, as used in the ConcurrentLinkedQueue from earlier in this thread; or more simply, JNI, which people, including myself, use to do all sorts of craziness, even going as far as the JVM itself by backpatching its code at runtime... and I'm not kidding: I actually do that).
Therefore, that anyone is even arguing some hard line that Java is type safe, C++ is not, that the definition is clear, that I'm ignorant for supposedly not understanding that definition (despite spending years researching programming languages and virtual machines in academia, and having implemented multiple), and that this type safety "is a substantial thing to give up", is ludicrous.
The ease of building lock-free algorithms and changing threading models in Java was probably a significant factor.
My company's web services are exactly that: we do all of the heavy graph traversal stuff down in the C++ layer and do the web services and XML parsing and all of that stuff in Java. The Java types that correspond to our graph elements basically just are a reference to our persistent graph store (basically a bunch of mmaped disk structures) which means that to do graph traversal on the Java side we never have to actually load the graph into the JVM. And the glue code between the JVM and our C++ still weighs in at under 1k LOC.
Unless you are running Azul's Zing JVM, some tuning is usually required for non-trivial systems. We have tuned our system to be very fast with low latency most of the time, and handle timeouts by using a load balancer in front of a few servers. The chance that all servers are having a GC pause at the same time can be made sufficiently small.
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+CMSIncrementalMode causes more frequent, smaller pauses. It was designed for low CPU core count, small heap size applications where your willing to pay a CPU premium to gain better response times. It is specifically not recommended for 2+ CPUs and/or large heap sizes. With large heap sizes you'll be paying a large CPU penalty and if your are using / freeing a lot of objects it may not be able to "keep up" with its short pauses resulting in the heap filling up quickly.
The usual approach for low latency systems are to spend the GC pauses incrementally over time. But for us, even though we require consistent low latency, big pauses are perfectly fine because we can run several replicas of the system behind a load balancer.
Not all systems can be designed to do this, and it usually implies a idempotent architecture where you can run the same transaction multiple times without any problems. But when it's possible, you can start guaranteeing low latency with a very high probability. Worst case: simply run the transaction on all instances and use the result from the first one that answers.
That is to say, optimizing the JVM's GC settings and effectively writing/debugging C and C++ memory management seem to be wizardly-level tasks of about equal standing. TANSTAFL applies.
10% of your code being performance critical would be a lot, in my experience. The usual number where I've worked is probably closer to 1%.
Getting Java's garbage collection tuned isn't trivial, but it's still easier than solving any of those problems in C++ at serious scale.
I don't understand. C++ has exactly the same model for collections of polymorphic types. But you can also pursue data oriented designs that aren't feasible in Java.
> knowing types at runtime
RTTI works fine. Querying object types at run time is probably a sign of horrible design.
> stability
By this do you mean unhandled runtime exceptions? If anything it's easier to write C++ free of fatal "null pointer" exceptions by avoiding pointers in favor of references and using RAII well.
Partially. Mostly I mean that bad code can make your runtime unstable by corrupting memory.
You could say "but I write code that never does that". Pretending for a moment that I believe you, you're still assuming nobody else does that, either.
RAII is great, modulo the problem that you have to handle the whole "no exceptions in constructors" problem to avoid resource leaks.
Querying object types at run time is probably a sign of horrible design
Querying object types at run time is usually a sign of doing something that wasn't anticipated at design time. As systems get larger and older, and as they get pieces patched at runtime, doing unanticipated things becomes inevitable.
Java isn't designed to be maximally clean and flexible for a tiny team. It's designed to build behemoth software for industry which is very forgiving of problems.
C++ doesn't do that as well.
I don't understand. C++ has exactly the same model for collections of polymorphic types.
Yes and no. C++ has more restrictions in terms of using only RTTI-capable types in mixed collections.
What types aren't RTTI-capable? Primitives and structs.
Java has the same problem, patched by autoboxing in such a way that you can't actually screw it up thoroughly. C++ lets you get into unfixable problems pretty quickly.
You're right - use only instances of classes with virtual functions, never a primitive or non-virtual class, and you can avoid it. But "with enough auditing, the problem goes away" assumes some pretty specific things about your final environment, including the assumption that errors either don't make it through or are very recoverable.