First, a lot of companies don't pay out dividends or buy back stock these days, so as time passes, removing their stock price from the price at IPO, their stock price becomes based on perception--not even perception of the reality of the company's value, but perception of the stock's value, which is increasingly just speculation. The stock price might remain tied to the performance of the company in broad strokes, but without regular dividends, sales, or buybacks to tie the stock back to the company, there's nothing to keep it from becoming disproportionate with regards to the company's value.[1]
Second, when companies do pay dividends or buy back stock, it's sometimes done by borrowing money. This actually drives down the value of the company (since now the company has to pay interest on those loans) but drives up the value of the stock--the value of the company and the value of the stock are going in opposite directions.
Third, with the wealth disparity in the US, even if 90% of people pull out of the stock market, it's quite possible for the stock market to go up, because the other 10% own >80% of the stock market. 90% of Americans can divest completely from the stock market, and it could at most lower the stock market by 20%.
This is why stock market metrics are not metrics I care about when determining how the economy is doing.
[1] EDIT: What I mean by "broad strokes" and "disproportionate" here is: Events occur which change the value of the company and the value of the stock, and at least the direction of these price with regards to these events is likely to align. In broad strokes, because people believe the value of the stock is tied to the value of the company, if a "good" event happens, the stock price goes up, and if a "bad" event happens, the stock price goes down. But it's pure speculation how good or how bad these events are. If big bad events are downplayed so they only are represented as slight drops in stock price, and small good events are marketed well so they are overrepresented as big upticks in stock price, then over time this can result in a stock price that goes up, when the value of the company is actually going down.
It's actually even more complicated than that.