You should indicate somewhere which languages are supported. You mention Deepspeech German in the credits, but did you use it to implement support for German or just to inform your architectural choices?
Was this before the streaming API was added to DeepSpeech? I recently did some testing with it, and it provides text within ~100ms of last audio block on my PC.
edit: that is, the most significant latency I had was from having to wait a bit to detect end of speech.
Fortunately for me I only needed it for command recognition, so the process was quite quick, and results were very good. No need to retrain the net.