The current median home sale price in the US is $416100[0], so with 3% down for a first time homebuyer ($12500), and assuming a Starbucks coffee costs $3.50 including tax, it'd take ~3570 coffees, or a little under 10 years of a daily habit.
That ignores any interest on the savings. So yeah it seems like "small" regular spending in your 20s (e.g. $100/mo total) can add up to an entire down payment in your early 30s.
Alternatively, assuming your rent would be at least $1000/month, you might live with your parents for an extra 13 months (or fewer for higher rents) before moving out on your own.