You don't have to. But I am someone who does. There are many, many people like me who love their phones like the Galaxy Note who feel the same way. If I made a list of the top 5 reasons I really like my phone, 1 - 3 would be screen size.
I have been surprised at how many people have gone from, "let me see that phone, it is ridiculous that you carry the around" to "wow, I never realized how hard my screen is to use compared to this." I know three people who have gotten a larger screen phone because of me in the past 9 months and I don't evangelize at all. They just see my phone and want to try it.
I almost didn't get my Note 2 because I was afraid of carrying around a "small book". To honest it is less obtrusive in my pocket than my Motorola Droid 3 was, that thing sat in my pocket like a thick brick. The single time it can be an issue is getting it in and out of my pocket while sitting and wearing jeans.