> Free speech is a negative right, which is a limit on government, which in turn means it cannot be infringed and that it has priority over rights of the government
Saying that it's a negative right only begs the question of what the right actually is. What does "free exercise of religion" mean? In both cases there needs to be an objective, shared understanding of what religion and speech are, and what it means to freely exercise them. Quite obviously the free exercise of religion can't mean anything each individual wishes it to mean. For the same reasons freedom of speech can't mean communicating however, whenever, and whatever by any means you desire.
There's a chapter discussing freedom of discussion in the classic 19th century treatise on English constitutional law, "The Law of the Constitution", by A.V. Dicey. The basic gist is that the author thinks it's a stupid phrase, even though the very concept derives from English legal theory. The TL;DR is basically that in the English conception of government everybody is free to do whatever they want so long as it doesn't interfere with others' rights. Government only intervenes and prohibits activity when it has some legitimate interest, and then it must use a fair means of seeing to that interest. Fair means that, among other things, the government must craft rules of general application that don't unnecessarily single out individuals for arbitrary and capricious treatment. And the executive can only enforce the rules through a separately constituted court system whose basic function is to apply the rules in a fair manner. Likewise for private individuals seeking damages for harm inflicted upon them by others. This is basically Due Process in a nut shell, which all rights in the Anglo-American tradition can be traced back to.
Note that the phrase "freedom of speech" doesn't actually give us any hints about how to gauge the limits of what individuals are free to express, or the limits of how the government can regulate that speech. It's almost entirely context dependent and a function of the extent to which the speech can or does cause harm.
You say that it's a negative right, which I take to imply that we're to construe it as meaning that everything is allowed except when there's justified reasons for not allowing it--an "exception". Well, that's fundamental to Anglo-American law. That's how all rights work in the sense of government's ability to regulate behavior. What Dicey is saying is that there's no need to have an independent concept of "freedom of speech" because for various conceptual and technical reasons we're going to end up applying the same tests as we do for any legitimate government interest and its pursuit generally. And because most speech is essentially harmless--in the Anglo-American tradition we don't consider ideas to be intrinsically harmful--people would naturally have exceptionally wide latitude to say whatever they wished, and the extent and scope of government's legitimate interest would normally be quite narrow. This is actually how it worked in both England and the U.S., at least up until circa 1930s. And, indeed, on the rare occasions we did consider ideas harmful--i.e. communism, incitement to treason, etc--the courts were fairly liberal in permitting government regulation and even censorship.
Arguably the jurisprudence of free speech in the U.S. changed in the mid-part of the 20th century for some interesting reasons. But you could also argue that it didn't change at all, just that that the new jurisprudence was simply more explicit in saying that there's a high bar to meet before we can consider speech harmful. But to the extent there are harmful consequences of the speech, well then naturally the government is free to regulate it, though only to the extent necessary to mitigate the harm.
Contrast that with other rights, like the right to a jury trial. There's no balancing test where we weigh the consequences of exercising that right. You have it, period. For other rights there's more of a grey area. But for freedom of speech, Dicey makes a persuasive point--that throwing around the phrase "freedom of speech" doesn't actually accomplish anything--you're just always begging the question, and you're ultimately going to use generic legal tools and theories to answer the question. Not that I think it's not worthwhile to have it spelled out in our constitution. Americans especially, but also many other countries, do build a complex jurisprudence around the phrase. And it's helpful to be able to put a finger on some specific text when you're resolving a dispute in favor of an individual and against the government, which is pretty much the only reason we have a Bill of Rights anyhow--remember, many people argued it was superfluous and unnecessary.
It's also worth pointing out that though we often claim to hold free speech to be sacrosanct, we rarely bat an eye at the fact that copyrights, patents, and trademarks severely limit the things, including ideas, we're permitted to communicate and how we can communicate them. The FOSS community is maybe more concerned than others, but in any event it's a telling illustration of how we accept that free speech is basically being able to say anything you want to say so long as saying it doesn't impinge upon or "harm" some other legitimate interest--slander, safety, or simply the ability to sell a book at a high markup.