That will introduce a lot of delay - they make synchronous cuts by frame buffering every input.
A video mixer acts as the source and sink for the output and the inputs respectively, where a HDMI switch will just physically disconnect and connect the output port to a different input port, meaning the HDID negotiation has to be re-done every time.