It technically compiles it to memory before running it, but the point is that you don't have to invoke javac first. It's a relatively new feature.
Oh so actually, you can use a shebang, just without the .java extension and with the "--source" flag, it's specifically written out in that spec. I was doing my experiment wrong!