What rules can you possibly have that distinguish the expert and the manipulator in all cases, without abuse?

I think free speech comes from the same base as universal vote: any selection mechanism would be corrupted and in the end cause more harm than good. That is why the solution is to let everyone speak / vote. If you have some uncorruptible people or mechanism for selection, just use that to make policy decisions directly.

I think the solution is to elevate critical thinking in the populations, so people can be less vulnerable to psychological warfare. Otherwise you're just picking a different manipulator - whoever writes or enforces the speech limits.