Redhat have added new filesystem support after a .0 release. For example, they added XFS to the kernel (without xfsprogs) in RHEL 5.2 and added the tools to the base repo in 5.4, then officially supported it in 5.6.
Between lvm/dmraid and how much effort they've done to make it simple to use lvm storage for libvirt/openstack/etc., I see no reason why they'd ever support btrfs in RHEL. They would spend more in engineers than they would ever get back in support.
XFS is a fairly simple filesystem. That said, the issues they had were integrating it in VFS. There are still some bugs at that layer today, albeit minor ones. In 5.x, most of the issues were race conditions in VFS and raid controllers (sas bus resets and such). XFS itself is actually very mature but any time they add a new FS, the VFS layer requires a lot of work.
Btrfs is still very much a moving target, with large patches every release.