I'm not sure why that's surprising to you.
One characteristic that's common to both mainstream conservatives and mainstream liberals, though heavily more emphasized in the latter, is the deeply-held belief that government exists not just to create and enforce laws, but to establish and enforce social norms. Of course, one side wants to promote social norms discouraging racism and sexism, and the other side wants to promote social norms discouraging drug use, but they are on the same page about whether it's legitimate for government to be getting involved in any of that.
Libertarians almost universally reject the idea that government should be in the business of promoting social norms, which is distressing to liberals and conservatives alike. Conservatives, however, tend to get along with libertarians to the extent that even if they disagree on the function of government, they often desire the same ends. Lower taxes, less business regulation, etc.
Liberals, on the other hand, don't even share common ends with libertarians, with the exception of a few niche issues like civil liberties. Libertarians don't have a recipe for how limited government will end sexism in corporate America or fix the educational gap between black and white children. Worse of all, libertarians attack the very principles underling cherished liberal policies.
Take, for example, the Civil Rights Act. I imagine very few libertarians would disagree with the goal of eliminating racism in commerce, but the means Congress used to achieve it was pretty anti-libertarian: sweeping interpretation of the Commerce Power to regulate very personal sorts of private conduct. If you own a restaurant, and so much as a potato traded in interstate commerce end up in your soup, Congress gets to tell you who you have to invite into your private establishment.[1]
[1] Libertarians and liberals alike cite with disdain Gonzales v. Raich, where the Supreme Court held that growing marijuana fell within the Commerce Power because the drug trade involves interstate commerce. Nobody mentions Daniel v. Paul, a case where the Supreme Court held that a recreational park in Little Rock (located about as far from an interstate border as you can get in Arkansas), had to serve black people because, among other things, "three of the four food items sold at the snack bar contain ingredients originating outside of the State."