This was discussed by the WG. The best rationale is given in the thread starting from this mail[0] from Will Chan at Google, followed up by this one[1] from Amos Jeffries (Squid Proxy). The summary is: waiting for SRV lookups can be slower than just doing straight-up A records, so no client in a competitive space is going to do it.
Everyone agrees that in an ideal world this would be great, but in practice no-one is going to be the first person to move on that front. The wins aren't big enough and the cost is too high.
[0]: http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/... [1]: http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/...