Inception wasn't that big a risk. Nolan had proved he could do ideas (Memento), he'd proved he could do big budget (two Batman films) and he was willing to stick major stars in it (including arguably the biggest star on the planet). They then backed that up with a $100m advertising budget.
In many ways it was an art house sci-fi film but in many others it did everything by the book for a big budget movie (proven track record, stars, special effects and action then hype it to the hilt).
You're right that the pattern is towards the more conservative but I'm not sure that Inception was a massively out there move.
Requiem for a Dream is a completely different beast and shows the other way you can do it even now. That was a cheap film - £5m or so. Adjust for inflation and you get maybe double that, which is still less than 5% of the budget of the Avengers.
That sort of film can be funded by smaller production companies, by larger studios (often as vanity / credibility projects - Schindler's List was seen that way when it was signed off though I doubt the studio will admit that now) and in a host of other ways. I can certainly see large studios funding things like that almost as part of an incubator model - giving new directors who seem promising a stage to show what they can do before they let them loose on something larger.