You need enough RAM to store the model and the KV-cache depending on context size. Assuming the model has a trillion parameters (there are only rumours how many there actually are) and uses 8 bit per parameter, 16 H100 might be sufficient.
And the 'search' part of it could use many of these clusters in parallel, and then pick the best answer to return to the user.