Just the last 2-3 weeks have I had talks with enterprise companies regarding this topic. It seems to be on every CEO's agenda. I have talked to a couple of startups who wanted to do a similar thing to you. But they all feared Microsoft Copilot is not beatable. So they don't even try.
The thing is, you don't need to beat copilot. Copilot may well be the worse system and Microsoft can still win because they offer the most enterprise-y solution. I wouldn't be worried about competing with them on functionality. But I also would never even try to outdo them on a business level.
The hope is that by working with the community, we'll be able to incorporate the best ideas and contributions from a large pool of like-minded people to build something everyone can benefit from!
The OSS space has absolutely taken off in the NLP space and the excitement has bled over to developer platforms resulting in some outstanding projects, hopefully the same will happen with LLM applications.
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 - Mixture of Experts model like GPT-4 but this one is open source doesn't cost an arm and leg to run.
https://huggingface.co/mixedbread-ai/mxbai-rerank-xsmall-v1 - This is a very recent reranker, we will probably try it out. It's really nice that it's open source and small enough to run on CPU. I think an overall trend of the space in the next years will be to maintain/improve performance while shrinking the cost and hardware requirements.