This is exactly my point, the community defines the term. The OSI definition does a good job of making the legal aspect of (their vision for) "open source" explicit, but it also adds additional definitions beyond what the average layperson might consider "Open Source".
Take section 5 and 6 of the "Open Source Definition"[0]. It states you can't discriminate against "persons, groups, fields, or endeavors". So if I wrote some software, put a MIT license on it, with a single additional clause that says the CIA can not use this software. Magically, it is no longer "Open Source" according to the OSI, even though 99.9999% of people can freely use it under the MIT.
[0]: https://opensource.org/osd
They have a definition, they even have a pretty good definition, but the OSI shouldn't be the definition. All OSI licenses are open source licenses, but not all open source licenses are OSI licenses. (All thumbs are fingers, not all fingers are thumbs).