It is absolutely NOT a reference to his homosexuality.

It is a reference for his taste for under-age boys, which seems to be from what I can tell, the reason he lived in Sri Lanka