You can just have a conference call with the 5-10 speakers and use broadcasting software to stream it to the audience, why do they need to be in the conference?
Giving all the data to zoom probably means also giving it to most US law enforcement agencies (should they request it), that would be a big no no for me.
Oh, and you can also do sub-rooms with Zoom, which has some applications in these types of meetings.