If you wait until all the studies are complete before trying to do something, you will never take action. Life is more about trial and error than knowing how everything is known.
Most of the time I think it's a matter of spotting which the heavier factor is.
Sounds a bit like confirmation bias in action to me; there are almost certainly times you are crystal clear on what's going to happen and then it doesn't happen.
Assume a person has a bunch of algorithms available to them. They may not be able to tell if algorithm X works with problem Y until they start interacting with problem Y and find out that heck algorithm X does work.
With this approach, we can imagine knowledge that actually can only exist hidden within multiple of level not-knowing, at least if knowing is being able to fully characterize one's abilities.
argmax(sum(probability_of_event * utility_of_event) given action)
So do you open a door ? There could be a bear behind the door that will eat you if you open the door. The probability of that is not zero (and even if it was, it would not matter). The utility of that is -inf. The utility of not opening the door is also -inf. There could be a bear behind you and the door could be your only way to safety.
So a rational actor would never do anything at all, as the rational function would never converge.
Ergo, nobody and nothing is rational (as soon as the world has a certain minimal level of complexity).
There are various suggestions given out of this conundrum, but none are even mildly satisfying. For example, one could claim that you don't know those odds, so you don't care. But that is the same as saying you can't really be rational, so it's not a solution.
Every decision has a ROI. The line where ROI meets costs is your maximum budget for a decision, after which point it costs you more to remain in the decision phase that it costs you to follow the best of the few "imperfect" decisions you have in mind (one of them includes the decision to stop "deciding", do nothing, and move on - that's still deciding).
That idea so far is rational. If you wear nothing you'll remain naked. If you don't eat you'll die.
And most of all, being stuck deciding forever just about anything is not rational.
So this person is unable to sense his "decision budget" and decides forever, while most people have facilities (impatience, impulsiveness in some degree, boredom etc.) to work around us being stuck forever after our decision budget is spent.
Spending to get positive ROI is rational. Spending infinitely beyond that point is not.
(In that, at least in some sense, hunger is an irrational desire for food, it isn't reasoned out)
> The utility of that is -inf. The utility of not opening the door is also -inf.
Humans don't value anything--even their own continued existence--at inf, just at pretty average-levels-of-high numbers. We seem to get "intelligent" things done despite this. What's wrong with building an AI this way?
But aside from that, humans are not rational. Depending on your definition, humans aren't even intelligent.
When it comes right down to it, it's pretty obvious what the human mind does to solve problems. It copies behavior. That's all it's capable of. It doesn't try to be rational, good, ... It is like DNA in that way (where 99.9999999999% of behavior simply comes from the parent(s)). People keep forgetting that evolution didn't create a "human", evolution is simply trying to duplicate itself. Your brain is nothing more or less than another trick to accomplish that.
More detailed, here's what it does :
1) it looks at it's inputs, and models them : it finds how the feedback loop works between outputs and inputs (e.g. you move muscles 382, 182 and 138, your hand moves in the image of your eyes. That sort of thing and more complex)
(technically it doesn't really do this. Your brain models the outside world as if it's part of your brain. That it models accurate prediction models is just a consequence of that)
2) it "shortcuts" the feedback loop. Essentially if neuron B always sees something happen 10 ms after neuron A sees it happen, they will form a connection and neuron B will report the event when neuron A starts seeing it's other event. If this means anything, it's a huge autocorrelation machine (with only positive offsets)
3) it copies behavior from it's predictive model to it's outputs. A human effectively tries to be the most "average" human in existence, based on his/her idea of what that is. It also copies (much less) from other events, like trees moving, animals, even just random signals that somehow get into your nervous system (some were put there on purpose). And lastly it copies from various internal sources that were put into our nervous system, like various clocks (I'm sure you can see why a clock would be useful to correlate things with), and chemical signals that get translated (like hunger, pain, or exitement, or ...).
Keep in mind that I can split it up into functional things as much as I want. "Phase 1", "Phase 2", "Phase 3", but they don't really exist. The brain starts with a random signal and then does one operation continuously. That one operation implements 1,2 and 3 as a consequence of
The informal definition of rational is having good sense and good judgment. If your formal definition results in agents that never do anything, or are in basic conflicts with millions of years of evolution and survival, then it's far more likely the formal definition is wrong, or your understanding of it is wrong, and not humans.
I understand the benefits of strict formal rational analysis. You can reason about it. But then if you'll talk about that, don't compare it to intelligence. Strict rationality is not intelligent. It's a tool we've made ourselves that has specific strengths and weaknesses born out of its restrictions.
And strict rationality is absolutely dependent on the model you apply it on. Wrong model + perfect rationality = disastrous decision. No one talks about the models, ever, it seems.
This is the real problem with AI - that we try to implement our very rough (at this point of time) idea of what intelligence is. If we knew all the details of animal/human intelligence, we wouldn't be sitting here debating it, we would've just implemented the AI with it.