Home Assistant uses the Wyoming protocol to talk to Whisper, Piper, and the various other components of the speech pipeline. Might be something to check out?

https://github.com/rhasspy/wyoming

This subproject looks interesting too

https://github.com/rhasspy/wyoming-handle-external