Could you elaborate how you switched to Elements? I am curious if it makes sense for one to go through both books in sequence.
ISLR introduces you to many of the same topics in a less rigorous way. Once I was familiar with the topics and had worked through the exercises, Elements became much easier to learn from.
I'm not sure if reading Introduction will prepare you for Elements so much as it will just give you some knowledge you can use and see if it makes sense for you and what you want to do to go and (re)learn some of the math tidbits that you need for Elements.