You have an interesting argument here. It sounds like you're saying that once you've proven safety in Phase I trials, and you can skip to the large, post-market Phase IV studies to do the dose-finding and efficacy-proving that would traditionally be done in smaller Phase II and III trials. As long as the drug is not dangerous, which can be proven by the Phase I trial, there is no need for the masses to wait for slow Phase II/III trials because in the worst-case scenario, the drug is safe but not effective, which is an acceptable outcome. So far, I find this pretty convincing.
However, I would raise two objections.
1. Without a Phase III trial that proves effectiveness, payers do not have sufficient motivation to purchase a drug that may or may not work. One solution would be to have the producer of the drug offer the drug for free/cheap until it demonstrates effectiveness, but that becomes a lot more expensive when you have a larger cohort. This is admittedly an implementation detail that could be sorted out, but it's a very important detail that would have to be addressed for this plan to work in the real world. The bigger problem is:
2. The best way to prove that a drug works is a large, randomized controlled trial. Your proposal is definitely an improvement over current trials in terms of making them larger, which would help to improve statistical power (and also get potentially helpful drugs into the hands of patients, of course). However, without the "ritual" of a Phase III trial, you lose randomization and you lose the placebo control. These are big losses that make statistical interpretation of results very hard because they open up the door to many potential confounders. As you mentioned, Bayesian signals can try to control for some of these confounders, but you can never be confident that you've considered every possible variable, whereas randomized placebos can eliminate almost all confounders.
So ultimately, I think that Phase III effectiveness trials are necessary in order to prove effectiveness because the statistics with post-market Phase IV trials will never be as "clean" as an RCT. And if we were to delete Phase III trials and skip straight to Phase IV, we would never again be able to confidently say whether or not a drug works. Thus, the ethical cost of making patients wait a few years for the Phase III trials to finish is outweighed by the benefit of being able to say with confidence that the drug works for the rest of eternity.