Very rarely will someone tell you that you're not charging as much for something as you should (perhaps with the exception of patio11 because he wants us all to make more money :) ) but generally there's always someone who will say something costs too much.
As explained elsewhere, as soon as you want a product that will be sold by distributors you're looking at a retail cost of ~4 * the build cost. This is because distributors will be looking at a 100% markup to cover their costs (and give room for discounts etc) and the manufacturer will be looking at about the same to cover their non-direct build costs (research, development, documentation, customer support, staffing, taxes, returns, damages, profit, etc). If they don't price like this then the product is unsustainable.
The name you're looking for is probably "non-volume electronics" or "small-run electronics" where the cost of developing a small-run product is spread over a smaller number of units and thus appears to be more expensive than some apparent mass-manufactured equivalent device (e.g. a low end cell phone).
Let's take your examples of $3 + $4, add $1 for the case, and $2 for the PCB & other components and you're already at a retail price of $40 without considering assembly. It gets expensive fast.
But, for the right market, it's still seen as a fair trade of time/money.
(I've seen a really good spreadsheet somewhere that can be used to make this sort of calculation--but I've not been able to track it down. Any pointers welcome.)
For disclosure: I've met and chatted with Marcus on a number of occasions and contracted with Sparkfun in the past but otherwise have no connection with the product in question. I just know how surprisingly expensive hardware manufacturing can be and how disappointing it can be to hear "it's too expensive" from people who may not be aware of the costs involved.