I don't think that buying coffee is necessarily a recipe for disaster financially, but if you are strapped for cash and have a $100/month latte habit then I think you should probably address that.
I'm all for spending money on things that bring value to your life, but I think if people cut the latte habit they would see it doesn't provide a ton of benefit. It's more valuable to occasionally enjoy a latte than have one every morning.