To add to this (because the difference was really not clear to me from your suggested rhymes):

The difference is in the sound of the final s. "loose" ends with "s" from "set", whereas "lose" ends with the z from "zoo".

And while the vocal sound is the same according to https://en.wiktionary.org/wiki/lose and https://en.wiktionary.org/wiki/loose, I feel that there is (or can be) some difference there. Just like in "jews" vs "juice".