I doubt it would work, not because of national security, but because of jurisdiction. Unlike some other countries, China has not accepted the general jurisdiction of the ICJ. That greatly limits the ability to get a case against China before the ICJ. A case could be brought with China's consent, but obviously in this case China will not consent. The Joint Declaration itself has no dispute resolution clause. I think the UK's only hope is to find some other treaty it has with China (or maybe even Hong Kong SAR) which does have a dispute resolution clause, and use that to try to backdoor the issue of the Joint Declaration into an international court (ICJ, PCA, etc). That's not impossible, because there are actually some treaties to which both the UK and China are parties with mandatory ICJ jurisdiction. However, these treaties are on seemingly unrelated topics like racial discrimination or health. Trying to squeeze the dispute about the Joint Declaration into a dispute about some other such treaty is going to be legally rather difficult, and the odds are against the UK succeeding if they tried, but I don't think we can rule it out as completely impossible–lawyers can be very inventive.
If we assume the UK manages, against the odds, to overcome the jurisdictional hurdle, and some international court accepts the case – I'm not convinced they'd give China as much deference on the "national security" issue as you think. If "national security" means whatever China wants it to mean, it turns other provisions of the Joint Declaration (such as the guarantee of freedom of the press) into a nullity.
> Even if it wasn't, the international courts have no power to compel any judgement
That's true. If an international court rules against Beijing, Beijing will just condemn the court and ignore the judgement. International legal action against China has no hope of making any difference on the ground in Hong Kong. But it does have some hope of infuriating Beijing and causing them to lose face. That's not much, but it isn't nothing.