If you want to know what they are talking about, here are two free courses that go over all of that stuff and more. I particularly liked error oracle decryption exercise from coursera crypto class:
That's the value of using established libraries. It's not necessarily that those library authors are so much smarter than the rest of us (although maybe a few are). It's that they already know a lot of the ways that their code was wrong.
If you write it yourself, who's going to catch your bugs? Then who's going to catch theirs?
This is not specific to crypto code, but crypto code is probably harder than most to get right, and more likely to be used on serious stuff.
Aside from your point (which is valid itself), it's not even necessarily "smarter". Comparably smart people more specialized at task X are likely to be better at task X.
Good crypto code needs a good understanding of crypto and a good understanding of code. The person who has both is rare. Far too frequently, the situation you've described comes about because the only people writing crypto code are the people who understand crypto.
And I think the admonishment "don't write your own crypto code" is peer-pressuring people who have a good understanding of code from abstaining from gaining a good understanding of crypto.
Folks saying "don't write your own crypto" are really talking about production systems that will serve customers, employees, governments, etc. Crypto is not going to be a market differentiator; it just needs to work. So why not stand on the shoulders of many others and use a library that is already well tested and patched up?
But in terms of spending your own time to learn, I doubt many folks would say don't do that. Matasano even provides a bunch of freely available materials to do just that.
See, this is why I hate these sorts of statements. They assume it's impossible to be innovative in well-established fields. Or they assume the person they are saying it to is incapable.
It's a sort of arrogant condescension that I don't think should have any place among polite people