Why do you want to sit near other people? Then they can see what time you come in or leave the office.
If you feel the need to hide the hours you work then there is something very wrong.
I very often work 6a-2p and I will occasionally get odd looks or the random comment when I am packing up for the day an hour or less after people return from lunch. If you're there when someone arrives, they assume you got there 5 minutes ago, not 3 hours, at least subconsciously.