What aptitude are you trying to test for? Do you already have a code base in Java you might use for inspiration? You mention framework doesn't matter, but the difference in what you can accomplish in 15 minutes is significant if you're comparing, say, core Java EE and Grails.
Agree with brudgers that 15 minutes is short. You're going to need a fill in the blanks type of assignment, where they import a project into their favorite IDE and can just add methods to a skeleton class. This means you need to test importing/running in at least Eclipse and IntelliJ, and maybe a regular text editor (Emacs/VIM/Sublime) with a simple build/deploy/test script (Maven or Gradle).
In other words, you have to factor away all the time sucking annoyance associated with spinning up a project you haven't touched before.
This is a hard problem, would be interested in discussing it further with you.