I used to be a Director in Engineering team at Box. I now run a unique bootcamp in the valley (
http://InterviewKickstart.com), where we train engineers to do better at technical interviews. In my perspective, a combination of following is happening:
0. The companies the OP interviewed at, can afford to not give a benefit of doubt. If they hire a few less engineers, their revenues are not going to see a significant dip. Not true of startups. e.g. If an e-commerce startup misses a holiday season, they are done forever. They'd hence be more forgiving.
1. Valley thinks it is a meritocracy, but it is not. Valley is full of biases when it comes to engineering talent. (It's true for all meritocratic systems). They want certain kinds of engineers.
2. Regardless of the hiring market, the companies OP interviewed at, are not forgiving in their interview processes. e.g. use of substring() instead of passing indices in a string recursion will kill an interview with most interviewers. So will use of pow() in atoi(). And you won't know ever.
3. Interviewing is not a test. Interviewing is closer to a date, than it's to a test. There are no "failures" on a date. It's about whether that relationship clicks. "Hiring market" has little bearing on a date.
4. Even if the interviewing panel and hiring committee knows that there is a shortage of engineers, they cannot knowingly give a benefit of doubt to a candidate, or that starts to affect morale in the company.
5. They cannot afford to give a lot of time in interviewing. e.g. they cannot spend a week working with you and knowing you better. If they do that, they will never be able to hire enough engineers. They need them in hundreds a month. They hence have to have an interview process that's short. And short interview processes are bound to be imperfect.