Think beyond yourself. In a team environment, there are times at which other team members will need to communicate with you to get their work done. If you're working on a different, arbitrary schedule, you can impede your team members.

If you want true flexibility and control over your schedule, the best option is to become a freelancer or consultant. But keep in mind that in many cases your clients will also have expectations around your schedule and availability.