I'd like to qualify this with some additional advice.
Advertising, on SU, a site that's not build for the SU audience, is likely to fail exactly as you've described.
If you're going to advertise on SU, you should craft the page that you do advertise to be instantly attention-grabbing, short, and tuned to give you whatever result it is that you want.
http://www.christmasbingocards.com/ , for example, is clear and immediately obvious, but not very attention grabbing. If I'm stumbling around looking for interesting stuff, I will not be grabbed by this page, no matter how well targeted to my demographics it might be, because it's not targeted to my frame of mind.
On the other hand, if you'd published a cartoon instead, for example, something obviously funny, chances are most stumblers would have read it. Then, your job would have been to create a cartoon that's funny and also encourages the reader to click around the site.
That's no easy feat, I'll grant you, but the point is, if you want to get useful traffic out of StumbleUpon, you have to tailor the content to the channel-surfing frame of mind.