My 23c handmade FMB tubulars (definitely not cheap) are a much nicer ride than cheaper tyres.
Rolling resistance is far more complicated, on anything other than a wooden velodrome, the road surface actually means that there is a sweat-spot in terms of pressure - too low and rolling resistance will be high, as you say, this decreases up to a point at which the effects of the microbumps in the road start to cause energy losses through hysteresis and eventually the tyre bouncing over tiny bumps.
For a 23c tyre and a 70kg rider the optimum tyre pressure is actually typically in the 80-90psi range (5.5-6ATM) although again this is dependant on tyre quality, an expensive tubular tyre with very supple silk sidewalls can be run at higher pressure than a touring tyre with tough reinforced sidewalls.