Your address isn't a public key. It is the formatted hash of a public key.

https://en.bitcoin.it/wiki/Technical_background_of_version_1...