My cholesterol has been creeping up since I hit 30. At the time it started increasing, I was doing something like 15 hours vigorous exercise a week and eating pretty healthy. Not perfect, but pretty good. Doing that much exercise meant I ate a lot.
Arriving in my 40s (a lot less exercise now.. maybe 6 hours a week), my cholesterol got to the point where the doc was talking statins. I didn't like the sound of this and cut out all dairy and avoided fatty meats. The year I ate like that my cholesterol went up by 15%.
As a last resort before starting on statins, I went completely vegetarian (but not vegan) and my previous year's gain in cholesterol reversed in 6 months.
Reading up on it, that may be the one loss I get from cutting out meats and I'm destined for statins anyway.
But, fwiw, that's my experience with cholesterol and dietary changes.