Yeah I guess in any field the low-hanging fruit will be yanked. One situation we could end up in is where we feel like we have more possibilities but counter-intuitively they don't work (otherwise, by definition I guess we could say, they would be low-hanging fruit that had already been yanked). This is the situation you may be describing. Let's call this situation A.
Alternatively, we could end up in a place where there's no more low-hanging fruit and we also can't even come up with ideas that seem promising. I guess at that point the field will just kind of die and people will move on to other pursuits. Let's call this situation B.
I guess a field should hit A at some point and may continue to progress to B eventually or splinter into sub-fields.
Yeah so from a very meta perspective I guess I should expect that things that I expect to work should fail. But at the same time I can't ignore the fact that I honestly felt surprised for at least a few of my ideas that they failed (others felt more like a gamble so it's no surprise they failed). So in some sense I should expect to be surprised but I am still trying to get to the bottom of why I felt surprised in the individual situations...
Your point does raise some other psychological questions. Like maybe we're wired to delusionally believe our likely-to-fail ideas are likely-to-succeed. If we get rid of those delusional beliefs, will we become unmotivated to try new ideas, or will we just seek out actually promising ideas?