"Literary fiction" might sound redundant, but in fact it's jargon used by critics, usually to describe a certain range of writing styles in which all of the implied drama takes place in the characters' inner emotional states rather than anything explicitly described in the narration or dialogue. It's usually contrasted with "genre" fiction, I guess because they wanted a broader term than "pulp."