328 karma · joined January 4, 2021
What we are doing is automating bespoke account qualification and research. Say, for example, you are a company that sells software to chemical synthesis labs. Your ideal customer synthesizes more than 5 chemicals. They need to synthesize them in large batches and they need to have more than 10 chemists on staff.
To go out and find those ideal customers, right now, you would need to spend hours upon hours searching and manually qualifying each account. We are going to automate that. By providing more accurate leads we will reduce the amount of time that companies need to spend on prospecting while also increasing the open rate.
I totally agree with you about latency. It is very very important for use cases such as a voice assistant. I also think there are use cases in which latency doesn't matter that much. One thing I think I may have understated about S.A.T.U.R.D.A.Y. is the fact that, at it's core, it is simply an abstraction layer over vocal computing workloads. This means it is 100% inference implementation agnostic. Yes, for my demo I am using whisper.cpp however there is an implementation that also uses faster-whisper.
I also want to call out that I have spent very little time optimizing and reducing the latency in the demo. Furthermore, when I recorded it I was on incredibly shoddy WiFi in northern Scotland and since the demo still depends on OpenAI a large chunk of the latency was introduced by the text-to-text inference. That being said there is still a ton of areas where the latency in the current demo could be reduced probably to the neighborhood of 1s - 1.5s. This will get better in the future :)
I want to touch on something else you mentioned. GPUs. I intentionally tried to avoid using any GPU acceleration with this demo. Yes, it would make it faster BUT I think a large part of making this kind of technology ubiquitous is making it accessible to as many clients as possible. I wanted to see how far you can get with just CPU.
In regards to your comments about NLU/NLP I have not dug into using them in place of LLMs but this seems like an area in which I need to do more research! I am very far from an AI expert :) I have a bunch of ideas for different ways to build the "brains" of this system. I simply have just not had time to explore them yet. The nice part about this project and demo is that it doesn't matter if you are using an LLM or an NLU/NLP model, either will plug in seamlessly.
Thank you again for your response and all of this information! I look forward to hopefully chatting with you more!