I would even be okay with getting rid of the $100 bill. Let's face it, people who deal in VERY large amounts of cash $100,000+ are not good people. Drug dealers or politicians (same thing in my opinion) don't use $20 bills as amounts that high get very cumbersome. If you find yourself in a situation with so many $20 bills that you physically can't carry them around, I have literally zero sympathy for you.
Slippery slope? Maybe, but as it stands I have no issue.