Are there any problems we have a proof for, and a corresponding proof that no algorithm can exist to provide said proof?

Basically, why can't I write an algorithm that can be given input for "is the halting problem decidable?" and return "no"?