Yes. The IRS is in the money refunding business.
It's NOT INCOME. It was already my money. If anything, I should get it back with interest, since they had it for a while.
For state/local taxes, as smoyer points out, if you erroneously deducted a larger amount of state taxes than you actually owe, then, when you get the difference back, you have to correct that on the next year's form, if you had claimed the larger amount as a deduction. If that weren't the case, you could game the system by purposely having your state taxes massively over-withheld, deducting the inflated amount from your federal taxes, then receiving a state tax refund.
It's as easy easy as (not pie) flying with wax and feather wings.