Yes, it did occasionally distract from the novel, but I'm the kind of person that doesn't mind reading and re-reading a passage.
Hover has poor usability on mobile. I don't think that I'm the only one who reads the phone on even short trips.
However, most of the interesting uses of footnotes people talk about, like nested stories, are not like this. Novels like _House of Leaves_ or _Infinite Jest_ for example. But these are fine also, because you can just assume you want to read the footnote and follow the reference. It just makes reading it more nonlinear and like a CYOA.
However, in between, there is no golden mean. It is very frustrating to have a book where half the footnotes are mere bibliographic information but the other half are interesting comments, digressions, meta-commentary, and so on. You either spend a lot of effort breaking from the text with wasted clicks/context-switches, or you miss a lot of important things. This is bad enough when it's a book using footnotes and you only have to keep glancing down at the bottom to see what it is - but it gets far worse with endnotes, and if they require multiple clicks each, you just give up!