I think part of the problem is that it became a bubble. Literary authors weren’t primarily writing for the general audience, but rather their specialized peers.
As this bubble became more separated from the mainstream, interest in these literary magazines shriveled.