On several occasions I've used the Tomcat Security Manager and it have not given me any problems. I find it fine-grained enough for my purposes, although not as fine-grained as you wish. I've used it to limit JVM's access to a specific set of hosts and TCP ports, restrict (RW and RO) access to files with and without wildcards, restrict access to specific methods, properties, classes and what-not. One can't use it to restrict the access to specific databases, or topics in the message queues as you suggest, but I don't think it's necessary. I think it's out of scope for the VM to restrict access to a specific DB and this should be the responsibility of the specific DBMS. Otherwise I'd imagine the overhead would be quite serious. I haven't heard of any VM manager that would provide such a thorough and deep access control, do you know of any?
If you still need this functionality in a Java security manager I believe you could build it using the existing hooks, they look quite powerful and flexible.
Now, the real pain I've had with the Security Manager in Tomcat 5.5 was writing the rules for a pre-canned application, not written with SM in mind. It was quite a tedious process, but all MAC systems are tedious to set-up initially. That's life.