Yes. The big parts are the models (neural networks (NN)). You can make them quite small (sparsification, quantization, transfer learning to smaller models, ...). But this is non-trivial.

Kaldi is the most complete open source speech recognition software, covering HMM/GMMs, the conventional hybrid HMM/NN, and much more. If you care about end-to-end only, there is much more, e.g. Mozilla DeepSpeech, and most modern open source encoder-decoder-attention frameworks can do that (e.g. Lingvo etc).

Most of these encoder-decoder-attention framework can also do text-to-speech (TTS). Maybe you need a separate vocoder, but you will also find open source code for that.

For the assistant engine, there are also a couple of open source projects, e.g. Mycroft, LinTO, Rasa, Leon, Open Assistant, Susi, ...

Putting it all together is non-trivial, and requires lots of expert knowledge, and lots of work and time to get a good result. This is hard to do as a hobby project.

Have a look at https://snips.ai/