Non-compete clauses will try to limit the usefulness of the "in your mind" knowledge by restricting the domains in which you can work post-departure. It's my understanding that such clauses are generally held to be unenforceable except in an acquisition scenario.
Really egregious is taking the sales data. Business analytics around leads, customer satisfaction, pricing, etc are not the same as retaining general knowledge. If you left and remember the point of contact you had at a customer, that's allowed (barring non-solicitation agreements). If you leave and you take a list of customers, data that the business has generated about them, etc, that was never yours and it's not your knowledge. It's clearly the business's and there's usually dozens of people involved in the creation. That's clearly theft, especially since it was never yours to begin with.
While I have jumped to competitors, I moved to roles that weren't in any form competition to my former team/role. That makes it easy, even if I would accidentally take things with me, I wouldn't be tempted to look at it, as there would be no point.
So yes, I take all my growth, knowledge and experience, but nothing that is really unique (say trade secrets) to old company would directly apply to my new role, so there has never been any problem. Once one is willing to jump to a competitor in a manner where you trade secret knowledge would benefit your role directly, one is creating a problem.