The logistical challenges look daunting to me. My recollection is that the problem of coordinating and optimizing deliveries in such a point-to-point system with time constraints is NP-hard (I could be wrong about this, but that's my recollection). Then one has to deal with all sorts of real-world problems like order-fulfillment errors and substitutions (intentional and otherwise), constant shrinkage of merchandise, etc.
Obviously there's some price at which the service can be profitable, but I'm not sure it's $4. (Consider that the former CTO of Kozmo, Chris Siragusa, has been running a one-hour, point-to-point $3 delivery service for a number of years, MaxDelivery.com, but he has kept the service restricted to a relatively small, dense area in lower Manhattan, making the economics worthwhile.)
In any case, I sure hope you're right!