Location dependent. I hazard a guess 80% of UK people going on holiday take out travel insurance, it's practically pushed down your throat at every turn and in the media too about dangers of being medically insured abroad without insurance. The airline travel insurance for lost baggage and delay is pretty scummily pushed and generally doesnt cover medical. Luckily it is usually only 5-20GBP per person per trip or 50gbp-100gbp for an annual policy (with max time out of country limited to X months usually 6). I suspect this is because we dont generally have medical insurance in the UK due to the NHS where as I guess your $$$pm medical insurance covers you abroad so primary reason for holiday insurance doesnt exist.

IMO, trip insurance mostly makes sense if you have a large non-refundable pre-pay in which case the insurance is also covering you if you have some injury or other covered reason for not being able to go. There are also some trips that have a greater probability of injury and expensive evacuation, e.g. treks in remote areas.

Some argue that you should always buy it because there's always the possibility of bad things happening that could be very expensive. But personally I (like I assume most people) am very selective.