One might think drugs would be more dangerous if the FDA were less conservative, but I suspect the opposite. If it takes you ten years and 2 billion dollars to release a new drug, and then you find out it has some terrible side effect in 0.1% of the population, you don't get to fix the problem. You might not withdraw it either, since the drug would still be a net good for society. And if one drug is approved instead of ten, patients are forced to stick to the flawed solution.
The far bigger problem than drugs with side effects is a lack of drugs. Around 1/3 as many drugs get approved today relative to 1970, and the costs of a drug approval have risen 5 times faster than inflation since then. Medicine has been an anti-technological field for the last few decades - it's doing less with more. The most valuable biomedical companies are the ones that have been around since the 1800s - one could imagine what computers would look like today if IBM were still the largest IT company.