The Java browser plugin runs unsigned applets in a sandbox. Sandboxed applications can only use a subset of Java that should make it impossible to cause serious damage to the host system. However, if an attacker can break out of the sandbox, they can do whatever they want, so this is a pretty common attack vector.

If you're interested, I spent some time figuring out how these attacks work and I blogged about it here: http://jsaxton.com/fun-with-wireshark-and-ie-java-exploits-p...