The bottom line is, there are many periods of time when stocks and bonds are inversely correlated and many times when there is a positive correlation. That's the whole point.
This is a gross simplification. Historically, to the 1930s, stocks and bonds were inversely correlated. Only recently has that behavior not held true (around 2008, according to the very PIMCO paper you cited).
To put it simply, it all comes down to interest rates. The only reason stocks and bonds are no longer inversely correlated (in my opinion) is the Fed's QE efforts. Historically, funds would move between stock and bond asset classes based on market sentiment (stocks when bullish, bonds when bearish).
Now, with an excess of cheap funds available (thanks QE!), both asset classes are being inflated artificially.
http://www.rba.gov.au/publications/bulletin/2014/sep/pdf/bu-...
http://blogs.wsj.com/moneybeat/2013/07/25/why-arent-stocks-a...
http://money.usnews.com/money/blogs/the-smarter-mutual-fund-...
http://www.bloomberg.com/news/articles/2014-08-26/something-...
http://time.com/money/3981692/what-the-bond-market-says-abou...
Not true. In 1927, the correlation was around 0.18. The correlation was close to 0 in 1928 and it was 0.4 in 1929.
>Only recently has that behavior not held true (around 2008, according to the very PIMCO paper you cited).
Again, not true. From 1965 to 1995, there were only a handful of years when stocks and bonds were negatively correlated. See figure 1 in the research report from PIMCO if you want to find this.
Bonds with higher risk (states, municipal, various corporate) will tend to track the federal bonds with some higher yield due to the higher risk you're taking. The amount of risk you're taking is a function of the state of the economy since in a bad economy e.g. corporations are more likely to default (overall). So one could say that yields going down are indicative of increased risk except they often lag. The phenomena we've seen over the last several years has yields going down while the perception of risk is that the risk is reduced. People considered the risk to be highest during the financial crisis and dropping since. So stocks have sky-rocketed because yields are down and the perception of risk has lowered. At the same time bonds went up because yields are down due to QE and low short term interests (that's just math).
The stock market is a form of a risky investment. While some stocks have a yield directly in the form of dividends others supply it in the form of growth. The price of a stock today vs. some expected price in the future is similar to the yield on bonds. If the risk did not change and bond yields are down one can expect stocks to go up. Now obviously stocks are very much influenced by people guessing how much growth is in there but at the same time given some fixed guess there is a price that correlated to some return %... Stocks are also influenced by volatility as people tend to want to get a better average return if the outcome is very volatile.
So at the end of the day, all investment options "correlate" (negative or positive depends on what you measure) with each other because if there was a single investment that had a better return for the same risk then there would be an arbitrage opportunity. At the same time they respond differently to changes in the perceived risk because the riskier assets are a lot more sensitive to risk (duh)...
EDIT (some more thoughts): Inflation expectations also influence the relative pricing of bonds and stocks. That's because stock prices will tend to go up with inflation (as corporate revenue will tend to track inflation almost by definition).
The really tricky bit is that the economy as a whole is I think a chaotic system. A butterfly flapping its wings in China can send oil prices down in the US. So it's hard to say something like these things used to correlate and therefore they will always correlate in some non-trivial way. (EDIT: esp. when thinking about things like growth, inflation and even policy which are the core things that move these other assets around)