But just because a bunch of people agreed on it, doesn't mean it's "absolute truth", like how doctors for a long time didn't want to wash their hand, because they didn't believe in germ theory. Countless of examples where we've ("scientists" and/or humans) all agreed on something, which later turned out not to be true.
And believing that truth is just "consensus between people" would lead to people never trying to go against that, even if their truth seems more truthy than the "established" truth.