Grains and sugar have been the biggest lie I was ever told.
But here's what I propose to you HN reader: take 30 days. 30 measly days, eat eggs, meat and dairy. No grain, no processed shit. Give it a try and see how you feel. Ignore the rest.
I expect the main two mechanisms of action are that these diets tend to reduce total caloric intake (lower variety = lower interest in eating; less opportunistic snacking since what's on offer is often not allowed by the diet; and so on) and drastically reduce how much outright crap adherents eat.
10 days to evaluate a diet seem low. Not many people in the history of humanity have had a pure carnivore diet, there's a reason we have evolved longer intestines than carnivores. Another interesting fact is that the inuk people who have mostly a carnivore diet due to inability to farm in the artic, live 15 years shorter than the average person in the developed world. Omnivore is the way to go, not carnivore.
Remember that how you feel isn't the same thing as your long term health risks. My belief, and that of mainstream science, is that this diet will raise your cholesterol and increase your risk of heart disease. You won't be able to feel your heart health worsening until things are very bad. It might be a trade-off some people want though.
I also believe that what works for one might not work for another. So yes, trying different things and getting measurements is good advice. I'd warn against chiropractors (they call themselves "doctors", but not of medicine) on YouTube telling people that cholesterol doesn't matter; it is a risk factor, but only one of many factors.
My understanding is that they've done a wide variety of studies continuously over decades.
They observed only, people with higher cholesterol were more likely to die.
They fed various animals high fat diets, the animals died sooner than control groups. The animals who are a lot of fat had higher serum cholesterol.
They tried diet interventions, no medicine, just getting people to eat different, those who ate less saturated fat tended to live longer. Saturated fat is correlated with cholesterol levels, as shown in other observational studies.
They created medicines, did trials, some of the medicines tended to lower cholesterol and extend life, others lowered cholesterol and didn't extend life.
It's complicated, but there's a wide variety of studies that have looked at cholesterol from different angles and taken all together paint a crude but coherent picture.
Today, we know that total cholesterol is quite crude, looking at LDL is better, and there's other blood markers that are even better than LDL.
It's complex and beyond what can be discussed effectively on HN, but there's a lot behind the conventional wisdom on cholesterol. Dismissing it all as "bunk science" is dismissing a strawman.
Furthermore, mainstream recommendations are completely reasonable. Eat a variety of food, avoid processed foods, limit saturated fat but some is ok, only give statins when cholesterol is very high or their risk for heart disease is high. If more people followed the guidelines, the net effect would mostly be people eating more whole fruits and vegetables, hardly the great evil the industry is sometimes accused of.