I've been having great success building homebrew voice assistants with Rhasspy [0] and voice2json [1] (they're sister projects from the same maintainer). The code's not ready to share yet, but I have a voice-controlled Raspberry Pi music server in my car now, working 100% offline (got it ready just in time to never have to drive anywhere, lol).