The problem for aaaaa is it gets to this parse

  (a(a(a(a)a)

and can't complete the (aXa) parse for the second a, so it backtracks to try the (a) parse

  (a(a)

The issue is since the parse of the third a as (a(a)a) succeeded, it will never try the (a) parse for the third a.

That is incorrect. Feel free to try it here: https://pegjs.org/online, you will see that aaaaa is not a match. And no, that is not a bug in pegjs.

I think it will parse "a", it will parse "a a a", and "aaa a aaa", and "aaaaaaa a aaaaaaa" and so on, but nothing else.