Even if it wasn't, the international courts have no power to compel any judgement. But I don't think it would even get there.
I doubt it would work, not because of national security, but because of jurisdiction. Unlike some other countries, China has not accepted the general jurisdiction of the ICJ. That greatly limits the ability to get a case against China before the ICJ. A case could be brought with China's consent, but obviously in this case China will not consent. The Joint Declaration itself has no dispute resolution clause. I think the UK's only hope is to find some other treaty it has with China (or maybe even Hong Kong SAR) which does have a dispute resolution clause, and use that to try to backdoor the issue of the Joint Declaration into an international court (ICJ, PCA, etc). That's not impossible, because there are actually some treaties to which both the UK and China are parties with mandatory ICJ jurisdiction. However, these treaties are on seemingly unrelated topics like racial discrimination or health. Trying to squeeze the dispute about the Joint Declaration into a dispute about some other such treaty is going to be legally rather difficult, and the odds are against the UK succeeding if they tried, but I don't think we can rule it out as completely impossible–lawyers can be very inventive.
If we assume the UK manages, against the odds, to overcome the jurisdictional hurdle, and some international court accepts the case – I'm not convinced they'd give China as much deference on the "national security" issue as you think. If "national security" means whatever China wants it to mean, it turns other provisions of the Joint Declaration (such as the guarantee of freedom of the press) into a nullity.
> Even if it wasn't, the international courts have no power to compel any judgement
That's true. If an international court rules against Beijing, Beijing will just condemn the court and ignore the judgement. International legal action against China has no hope of making any difference on the ground in Hong Kong. But it does have some hope of infuriating Beijing and causing them to lose face. That's not much, but it isn't nothing.
It doesn't actually turn for example the free press clause into a nullity, it just means there is a conflict and one supercedes the other. It's not uncommon in law. If Apple Daily and Jimmy Lai had no contact at all with foreign agents, it would have applied unambiguously.
So clearly it doesn't turn it into a nullity, it just limits it.
Beyond that, fundamentally, the law was negotiated by the UK from a position of massive weakness, as they militarily simply could not hold Hong Kong, and legally wouldn't be able to supply it either. It is expected that debilitating clauses are included.
One such clause is that China has total sovereignty in anything that concerns foreign affairs or defence.
I think a lot of the UK's weakness was self-inflicted. They were one of the first Western nations to switch recognition from ROC to PRC, all the way back in 1950. The US didn't make the same move until almost 30 years later. If the UK stuck with ROC for longer, they could have bargained a better deal on Hong Kong with PRC, in exchange for PRC recognition. Instead they gave up that bargaining chip for nothing.
> as they militarily simply could not hold Hong Kong
Would one nuclear state (PRC) would rush into war against another (the UK)? PRC loved to threaten, but what if the UK had stated that it reserved the right to respond to a conventional attack on Hong Kong with a nuclear counterattack? Consider also that the UK had a 10 year head start on nuclear weapons (1952 vs 1964).
The UK seriously considered threatening PRC with nuclear weapons in the 1960s [0]. However, it appears the UK plan was not to threaten PRC with its own nuclear weapons, but instead try to convince the US to threaten the PRC with US weapons. The plan apparently fell through when the US, after expressing some initial openness to the idea, began to get cold feet. The threat was never publicly made.
In any case, if push came to shove the PRC would have simply invaded HK. There is nothing the UK could have done to prevent it.