I understood what pjax was doing in about 5 minutes. Backbone.js is more complicated than that, and while I'm certain it's not that hard to figure out, I felt a bit like "ok, it's doing stuff I don't immediately understand, but it's also not doing any deep magic that makes difficult things super-easy, either". Very subjective, but it just didn't feel like a sweet spot. Angular.js actually seemed like more of an "all-in" play with the fancy stuff, but that datepicker issue was a wake up call about getting to a point where even doing something simple might be quite difficult.
Backbone and pjax are not similar so maybe the issue is picking the right tool for the right project. PJax is great if you are looking to help navigate. Backbone is useful with managing objects and collections while keeping them separate from the DOM. It also provides its router to coexist with its views.
you aren't the only one. i too went through troubles and concerns when i started venturing out to use some of the plugins out there, notably datepicker, select2, jqmodal, actually just about any other "cool" plugins out there. creating your own directives to get something to work is something i'd just rather stay away from.