If the domain is the same one that the rest of your site runs on, it almost certainly does mean you have a security issue. The only practical way to protect against CSRF attacks is to use a secret token in a hidden form field to authenticate all form submissions. allow-access-from-domain="*" means that an attacker can steal your CSRF tokens using a hidden Flash applet running on their malicious page, then use that token to construct a CSRF attack form submission. That means they can perform any action on your site as if they were the targeted user. That's bad.

If you want to provide access to an API, put the API on a separate subdomain. That's why api.flickr.com has an open crossdomain.xml file and flickr.com doesn't.

Adobe actually have a pretty good explanation of the security issues caused by an open crossdomain.xml file:

http://www.adobe.com/devnet/flashplayer/articles/cross_domai...