A number of years ago, I set up a similar configuration with two OpenBSD boxes (which support PF and CARP out of the box) running Apache + mod_proxy_balancer for really granular load balancing and routing. It was a super-flexible and cheap way to route traffic to a mid-sized app server cluster, and generally worked really well.

We did have some network issues after the new topology went live that we unexpectedly tracked down to the LB pair, for one simple reason: CARP generates a lot of multicast traffic. Depending on how your hosts and network are configured, this can easily get routed out into a fairly large portion of your local network, and use a lot of bandwidth/router capacity with no benefit.

So, if you're setting up a CARP pair/cluster of your own, pay close attention to your multicast setup. Ideally, put the CARP multicast traffic on a dedicated subnet and watch your router and switch stats to make sure you aren't flooding the rest of your network with multicast spam.