Elixirls is finnicky and works 50% of the time in my experience. Don't know why.
Try using Zed it comes with Elixir support out of the box, and uses nextls, which works always for me. Snappy as hell! Like Sublime Text 4.0 https://github.com/elixir-tools/next-ls