Don't ever want to build a web app? That's pretty much Elixir's sweet spot, IMO. You'll get great productivity, great scalability, performance that's better than popular interpreted languages and a code base that's easy to reason about.
You'll also be able to do a lot from within the VM instead of relying on external services.
It will never be as popular though.
And it's not like other languages stay stagnant. JS and TS, Java with green threads, C# with actor and LiveView like libraries.
I migrated the code base for my last startup from Node to Rails at the end of 2015, then from Rails to Phoenix at the beginning of 2016. Pretty much all of that Elixir + Phoenix 1.1 code is still fine today. There haven't been any breaking language changes and the most arduous part of upgrading Phoenix to the current version would be the front-end—replacing Brunch with Webpack and removing Bootstrap. Other than that it's just a few lines of code.
The Rails app is slightly more dated but still fairly trivial to update. Most of the gems still do things in the same way and there's generally a clear way to go. The JS app on the other hand, is a pretty depressing mess of broken dependencies and multiple libraries I was using have dramatically changed their APIs.
For me, productivity and ease of maintenance matter much more than popularity does when picking a tech stack.
Why migrate working code to a new language?
And to do it twice?
The second time was because I'd gotten it to where it was getting users but I couldn't afford the server costs (given that it was free to use and my users were largely in Vietnam and Thailand). I needed something that could scale much more cheaply.
If I'd had millions of dollars in the bank, I'd have just stuck with what I knew and added people and servers as necessary.