I think paying them is perfectly reasonable, but I'll also note anecdotally that having done this for many candidates for years, I think very few people have ever declined.
You're putting the candidate in a very vulnerable and stressful position:
You're potentially asking a candidate to solve a problem with your application (even if scaled down)--
The question/assignment could be a long-standing issue your staff have been unable to address on their own and you want to try to find out whether you can get a free solution from qualified people.
The candidate(s) could get the correct answer, yet you could walk away from the process with a free solution.
>> "I think very few people have ever declined"
That's short-sighted - they're not declining because the want/need a job and are already vulnerable during the process; many would not, understandably, have the presence of mind to decline.
I just don't see a couple hours spent on a interview programming assignment as being any different from the 5-6 hours you spend in the office being interviewed, which by the way, you don't get paid for either (excepting reimbursements, etc).