The New Bond Market: Bigger, Riskier and More Fragile Than Ever

Not true. In 1927, the correlation was around 0.18. The correlation was close to 0 in 1928 and it was 0.4 in 1929.

>Only recently has that behavior not held true (around 2008, according to the very PIMCO paper you cited).

Again, not true. From 1965 to 1995, there were only a handful of years when stocks and bonds were negatively correlated. See figure 1 in the research report from PIMCO if you want to find this.

Bonds with higher risk (states, municipal, various corporate) will tend to track the federal bonds with some higher yield due to the higher risk you're taking. The amount of risk you're taking is a function of the state of the economy since in a bad economy e.g. corporations are more likely to default (overall). So one could say that yields going down are indicative of increased risk except they often lag. The phenomena we've seen over the last several years has yields going down while the perception of risk is that the risk is reduced. People considered the risk to be highest during the financial crisis and dropping since. So stocks have sky-rocketed because yields are down and the perception of risk has lowered. At the same time bonds went up because yields are down due to QE and low short term interests (that's just math).

The stock market is a form of a risky investment. While some stocks have a yield directly in the form of dividends others supply it in the form of growth. The price of a stock today vs. some expected price in the future is similar to the yield on bonds. If the risk did not change and bond yields are down one can expect stocks to go up. Now obviously stocks are very much influenced by people guessing how much growth is in there but at the same time given some fixed guess there is a price that correlated to some return %... Stocks are also influenced by volatility as people tend to want to get a better average return if the outcome is very volatile.

So at the end of the day, all investment options "correlate" (negative or positive depends on what you measure) with each other because if there was a single investment that had a better return for the same risk then there would be an arbitrage opportunity. At the same time they respond differently to changes in the perceived risk because the riskier assets are a lot more sensitive to risk (duh)...

EDIT (some more thoughts): Inflation expectations also influence the relative pricing of bonds and stocks. That's because stock prices will tend to go up with inflation (as corporate revenue will tend to track inflation almost by definition).

The really tricky bit is that the economy as a whole is I think a chaotic system. A butterfly flapping its wings in China can send oil prices down in the US. So it's hard to say something like these things used to correlate and therefore they will always correlate in some non-trivial way. (EDIT: esp. when thinking about things like growth, inflation and even policy which are the core things that move these other assets around)

Credit default swaps for mortgages didn't even exist really until some smart speculators noticed that the bond market for mortgages was unstable (and full of deceit) and only needed a bit of default to create a cascading waterfall of default which nearly took down the entire financial system.

Honestly, CDR's were a good idea but it made no sense for the same banks to sell them that were also baking the mortgages. It would be like selling insurance on your own car. If you crash you are out a car and have to pay someone else! It would have made sense for banks to hedge their mortgages by buying CDR's from other investors. But then, that would have affected their bottom lines and they just plain got greedy.

HFT is debatable as many claim it creates liquidity. I'm not sure I buy that but I also wouldn't confuse HFT with financial instruments.

For instance, packaging mortgages into rated bonds of various risk tranches in the 1970's was a brilliant innovation which enable more people to qualify for loans as their risk was distributed and sliced up among many parties. Of course, that system broke down after being abused - but the initial concept still survives and is remarkable.

Other instruments like derivatives allow for affordable hedges, the ability to buy and sell at a future price you want or collect premium on that offer. They aren't just for speculators.