The prose is almost certainly inspired by H. P. Lovecraft. "At the Mountains of Madness" may be a good start, if you like this sort of stuff: http://www.dustylibrary.com/horror/5-at-the-mountains-of-mad...

Try a book called "Random Acts of Senseless Violence" by Jack Womack. The text doesn't morph into gibberish, but the lead character transforms from entitled middle class to borderline psychopath street kid, and the way the language changes as the story progresses is _wonderful!_

Mark Z. Danielewski's House of Leaves

William S. Burroughs

It reminded me mostly of the gibberish from the Cylon hybrids in Battlestar Galatica.

Robert Anton Wilson

Schrödinger's Cat trilogy especially

Hmm, Joyce's Finnegans Wake? You did say "probably back".

Thomas Pynchon?