When working with JSON, you must load the whole JSON document before you can start processing it. If the JSON gets big enough, you start having performance / memory issues. One might argue that you can just get more powerful hardware, but that's only half of the equation: you'd solve your problem, but not that of your clients. Are you going to tell every single one of them that they just need to spend more money on hardware?
When working with XML, you can choose to store the whole document in memory, but it's merely convenient, not compulsory. Using a SAX-like parser, you can stream the whole process and get a much smaller memory footprint.
Having never worked with YAML, I don't know whether the same argument applies.
This is more of a personal issue than an actual one, but another problem I have with JSON is that once you support it, people are going to demand you support JSONp - terribly convenient, but a major security hole.
Aside from that, JSON is obviously a great format for small, self-contained responses that can be consumed by both browsers and heavy applications. It just doesn't scale well with large responses.