Fair worry.
> What can I do to change things?
You should switch jobs.
I've been the top technical person at a couple of companies (not the size of your company, however). It can be exhilarating to be "the person". You get a lot of respect and autonomy. But it can be easy to fall into the curse of the expert beginner: https://daedtech.com/how-developers-stop-learning-rise-of-th...
While you can mitigate this to some extent by participating in groups (virtual and physical) for other great developers, the easiest way to handle this issue is to switch jobs.
So, I'd suggest you start interviewing. Either you'll find a better place, or you'll discover advantages of your current situation.