I agree it's definitional, so ultimately there isn't much to argue about.
What I wonder is what you think the upper class (as you define it) have that's so great or special that you don't. Sure you could cabin it so narrowly as to be people that e.g. get personal access to the President, or can get there kids into Harvard with a single phone call or something, but that definition is so narrow as to be useless (which gets back to the whole all Americans are middle class thing).
Having known several people with net worths in the 7-9 figure range, from brand new money through came over on the Mayflower, it really is a continuum. There's no secret handshake.
But that doesn't mean that somewhere along the continuum you really are living a different lifestyle than a family of four living in Northern California with both one parent working as a teacher and the other as a hygienist, each making $40,000 a year. They've got credit card balances, student loans, a long commute because that's where the affordable houses are, stress over how they are ever going to pay for college, and so on. You may shop at the same stores, but is that really a good indicator? Call what you have upper middle class if you feel like it, but that family is solidly middle class by any reasonable definition. So if there's a difference in kind, and someone has to give up the label, it shouldn't be them.
Finally, I should note that I find this little quirk of American culture fairly harmless. The one area where it can be pernicious is when it effects certain policy areas. For example, many people agree that the "rich" should be paying higher taxes, but since hardly anyone sees himself as rich, it is always a little farther out that the tax burden needs to fall. A good litmus test is the home mortgage income deduction, which is massively tilted to the high end of the income scale. If in justifying a high phase out you start to hear yourself say things like $1.5 million doesn't really buy you that much -- then you may be in the problematic zone.