That would incentivize waiting to post an answer. That's definitely not desirable.
Except that someone else could answer while you are waiting.
Now it's a dilemma. I can just write up my answer and wait to post it until someone else posts, maximizing the amount of reputation I'd get for the same answer. If everyone does the same, the net outcome is a longer waiting period to get answers to questions. I'd rather keep game theory out of it.
A possible solution: When someone answers, the karma clock is "reset", so it is then based on the time since the last answer given.
Snap! That's a good idea. :)
It needs to be coupled with question votes, as it would be too easy to game by writing a terrible question with a false account, letting it sit for a long time, then answering it with your real account.