"Visual novels" are definitely niche, but they are somewhat games and have very minimal interactivity. Yes some have branching narratives but others are just "click click click". They just tell a story and that's perfectly fine.
You wouldn’t say that “platformers” are not games just because we have a name for them.
I think it is fine to count them as games as long as there is some interactivity, but that obviously puts them at the outer edge of what games are.
To get back to the point, I think a visual novel should be eligible for a literature award, but that is because the interactivity is inherently limited.
The market finds simple denominations to help consumers navigate the offering, but these are not always accurate or very descriptive.
Focusing on the content as opposed to the label, why do you not think that these are games?
As I mentioned above, to me interactivity is the core element. Certainly a VN can have interactivity, although that is usually limited so I would put them at the outer edge of games. If it has no interactivity, then I wouldn't count them as a game though. A picture book is not a game.
Interactivity is just the ability to respond to a user's input. Even VNs with very limited branching do that.