From my experience at University, the JVM uses an indirect method of accessing and storing objects. So each reference to an object is effectively a handle, hence as big as a pointer. It points to a table that contains the actual pointer to the object in memory (or something to that effect), so that the garbage collector can move the object in memory.
Therefore an object that just has a single integer will require 2 pointers and an integers worth of memory (at least). Any less than that and it would be garbage collected since there are no live references in the code to the object.
Garbage collection isn't free.