We know that it's not just our modern language coloring our reading of it because Henry James's family wanted these passages removed from the published versions of the letters, precisely because they didn't want them to be 'misconstrued' as expressions of sexuality. The first volume of letters referred to in this article was published in 1920.

> I earnestly feel like that's our modern language coloring our reading of it.

You are reading modern language. Sexual identity was already a thing at this point in history and had been for several decades.

It's always important to be earnest...

https://www.youtube.com/watch?v=oVQIB-QuooU