Wikipedia tells us that Dutch "ui" is the diphthong /œy/ and "ou" is the diphthong /ɑu/. The English MOUTH vowel is conventionally /aʊ/ in American pronunciation, and the CHOICE vowel is conventionally /ɔɪ/. I will analyze them as if they were /au/ and /ɔi/.
It's clear that the MOUTH vowel is the best available English match for Dutch "ou". The difference between [a] and [ɑ] is minimal.
Dutch "ui" is trickier. /œy/ starts with a front vowel, which isn't true of /au/ or /ɔi/. /ɔ/ is a better match in terms of vowel height. (According to the notation. This is a case where you're probably better off just listening to samples; vowel notation is quite sloppy.)
/y/ is the high front rounded vowel, a sound which, like /œ/ (mid-low front rounded), does not exist in English. It has two obvious approximants: FLEECE (/i/), the high front unrounded vowel, and GOOSE (/u/), the high back rounded vowel. If you choose FLEECE, prioritizing the vowel's frontness, you'll end up thinking that "ui" should correspond to the English CHOICE vowel -- the first part of the diphthong has been moved back, and the second part has been unrounded, but compromises have to be made. If you pick GOOSE, then you'll end up on the MOUTH vowel -- in this case, the first part gets moved back and lowered somewhat, and the second part also gets moved back but preserves its lip rounding.
Is either of those choices objectively right? No. You might prefer thinking of "ui" as CHOICE on the grounds that that makes it different from "ou", but that runs into the problem that (according to wikipedia again) Dutch also includes an "oi" diphthong which is a close match to English CHOICE.
You could think of the root of the problem as being that Dutch has three high vowels /i/, /y/, /u/ (spelled "ie", "uu", "oe" among other ways, according to https://en.wikipedia.org/wiki/Dutch_orthography ), where English has only /i/ and /u/. You're going to end up with collisions somewhere.
We have a tongue-twister for German and English people that goes like this "Ik eet uitsluitend gefruite uien in mijn geruite keuken" (you can have Google Translate read it out loud). Meaning is "I only eat fried onions in my checkered kitchen".
That sentence is more ambiguous in English than you might think! I'm curious which of these meanings are compatible with the Dutch:
1. The only thing I do in my checkered kitchen is eat fried onions. (Other activities don't take place in the checkered kitchen.)
2. Fried onions are the only thing I eat in my checkered kitchen. (Other foods must be eaten elsewhere.)
3. Fried onions are the only type of onion I eat in my checkered kitchen. (Raw onions must be eaten elsewhere, but eating steak in the kitchen is OK.)
4. My checkered kitchen is the only place in which I eat fried onions. (I'll eat steak anywhere, but I won't eat fried onions outside the kitchen.)
5. My checkered kitchen is the only one of my kitchens in which I eat fried onions. (I'll eat steak in any kitchen, but I won't eat fried onions in my plain white kitchen.)
6. My checkered kitchen is the only checkered kitchen in which I will eat fried onions. (I won't eat fried onions in your checkered kitchen, but I will eat them in the bedroom.)
The only one it's not compatible with is 4, in that case the word "uitsluitend" ("exclusively") would appear elsewhere in the sentence.
Each of the others are ambiguous when written, but should be clear given pronunciation (emphasis), or the tiniest bit of context.
As for which ones work in Dutch, all of them.
It mostly gets confusing for me because of moving back and forth between English and Dutch grammar, but any bit of context would make it a lot less ambiguous :)
If you're interested in the theory:
Vowels are generally thought of as being points in a three-dimensional space. The dimensions are:
1. Whether the vowel is high or low -- if your mouth opens wide (chin closer to feet), it's a lower vowel; if your mouth doesn't open far (chin closer to eyes), it's a higher vowel.
2. Whether the vowel is front or back. The traditional way to explain this to linguistics students is to give them all lollipops. You put the lollipop on the middle of your tongue with the stick jutting out of your mouth. You can observe that the stick juts farther forward when you're pronouncing the FLEECE vowel than it does when you're pronouncing the GOOSE vowel.
3. Whether, as you pronounce the vowel, your lips are rounded or not.
Dutch "ui" is a diphthong, a transition from one vowel to another vowel. It appears to be a transition from (lowish, front, rounded) to (high, front, rounded). This is difficult for English because, in English, all front vowels are unrounded.
If you'd like to preserve the lip-rounding aspect of "ui", you need to use back vowels, at which point the MOUTH vowel is a near-perfect (low, back, xx[1]) -> (high, back, rounded) match.
If you'd like to preserve the frontness of "ui", the English FACE vowel is available, which is a transition from (lowish, front, unrounded) to (high, front, unrounded). CHOICE is a different option; it's (lowish, back, rounded) -> (high, front, unrounded). With CHOICE, you're choosing to match the rounding (and not the frontness) of the beginning of "ui", but transition into an end that matches frontness instead of rounding.
Which part of a sound is more important for purposes of matching a foreign sound to a native sound is a very interesting question, and to the best of my knowledge nobody has any idea why different languages make the choices they do. My (American) ears think the MOUTH vowel is the best match for Dutch "ui", even though it's an even better match for Dutch "ou". But someone else might disagree.
For an example of "differential splitting", consider that in modern English, W, V, and F are three separate sounds. In Old English, W and F were "real" sounds ("phonemes"), and V was a variety ("allophone") of F. In modern Mandarin Chinese, W and F are also phonemes, and V is an allophone of... W. Clearly, if you speak modern Mandarin, what's important about the V sound is not the same thing that would be important if you spoke Old English.
[1] Rounding is less obvious for low vowels -- a certain amount of lip rounding is induced by opening your mouth so far.
As for
> My (American) ears think the MOUTH vowel is the best match for Dutch "ui",
I personally stand by my choice of the CHOICE vowel also after reading your comparison of the different options, but it's definitely a close contest.
Using the MOUTH vowel in that word would reveal an English speaker immediately ;-) (Germans mess it up differently ;-) )