This is an extremely bad idea. Firstly, it reduces the value of the conference itself as a means to share the best possible information. Secondly, it lowers the esteem and respect of the speakers by allowing the audience room to question their abilities. Lastly, and most importantly, it allows politics into the selection of speakers, and moves the system further towards a popularity contest from a meritocracy, where people argue for speakers based on increasingly superfluous factors.
These are technical conferences, and you guys need to keep them this way. The selection of speakers need to be done on merit.