OSS framework for voice first multimodal assistants

Hi Hackernews, we're Maitreya, Prateek and Marmik. Over the past few months we've been working on building a platform to build, scale and monitor voice based LLM applications. Demo (https://www.youtube.com/watch?v=OSrOmyR7oQs)

1. Open Source orchestration: We're open-sourcing our orchestration to quickly setup and create LLM based voice driven conversational applications https://github.com/bolna-ai/bolna/

2. Hosted API Platform: Exposing our managed solution via APIs to build voice driven applications https://docs.bolna.dev/api-reference/introduction

3. Normal LLM telemetry tools won't work in giving visibility for audio bytes in and out of the system across multiple models. So, we've build our own observability layer fully integrated with the dashboard as well.

4. 3 different modes for creating agents - Lite (Intent classification based) (useful for basic calls and really pocket friendly). Normal (<2sec latency but only one llm call means it's cheaper than nitro), Nitro (<1sec latency and but multiple llm calls means really expensive)

5. Follow up tasks like webhook integration, summarisation, and extraction.

6. Modular and extensible architecture, which means connecting two different llms yet parallel paths(for example code and english to automate leetcode screening interviews) is really easy, albeit you'll initially need some hacking until we're able to release that to both hosted and open source versions).

7. Vector and Scalar caches to reduce the entire cost (by 3x compared to deepgram + mixtral + elevenlabs) and latency by 300ms - 500ms.

8. Semantic routing

Over the next weeks we'd be doing a lot of small releases here starting with a Speech language model only pipeline, integrating gazzele

We'd love to welcome you guys to our community, give us feedback and together build "langchain for voice first AI assistants".

Great to see an open source implementation for this.