Next time they offer you "unlimited" vacation say that's very generous of them but you don't need unlimited vacation days you know exactly how many vacation days you need and that's 28 days.
If they don't reject it straight up they might try to haggle over that which proves the point.
Bigger companies will even force you to take it once you've accrued a number of hours, as it can be a liability on the books (as unused holidays roll over indefinitely).
The US is an outlier in this regard.
Being fired over taking too much vacation time given the mentioned policy is potentially lawyer fodder.
Interestingly, in the UK, managers are also liable for health and safety to some extent. So if you get burned out because you haven't taken any holiday for the last 3 years, your manager has to be able to say "I made him aware he had holiday left and encouraged him to take it" or something along those lines, otherwise they/the company is also on the hook (possibly).
My company has unlimited vacation and I didn't take any vacation from Jan 1 to May 28, then I took off 10 consecutive weeks for the birth of my daughter and a move from Seattle back to Portland. We also treat Dec 24 - Jan 2 as company holidays because so many people use their personal vacation days for that time of year and to prevent people from trying to work through the holidays and dragging others into it with them. I then followed it up closely with a few days off for a wedding I was in, a couple of fishing trips, a couple out of office days for unexpected things. I've probably taken more vacation this year than I have in my entire career excluding my honeymoon.
We have a system for placing vacation requests that allows anyone to see what a given stretch of time looks like as far as people being gone, and we can track how managers respond to vacation requests as well. I guarantee that if we found a manager pulling the kind of shit that the OP's manager was pulling they'd be dealt with, probably even let go without a second chance unless they were stellar in every other way. Also, all requests go through the system, so there is none of this whispering back and forth and getting you to withdraw your request. The back and forth happens and is recorded for posterity.
The only problem is, just like most other culture fits at a company, how do you know until it's too late? Also, would you personally feel comfortable using an 'unlimited' amount of vacation?
My recommendation has always been to treat unlimited vacation days like bonuses and options. They are perks, but the number you're generally after is the salary, so don't discount that too much in favor of some nebulous benefits that may not materialize.
A few times, I had interviewed at such companies with unlimited vacation. I would ask about their vacations, and how many weeks they took off this year or last. Most developers would usually say they took 1-2 weeks off last year, to which I would ask that you get that everywhere, why not more? Then the answer is "oh we are so busy and we don't abuse this policy. But if we ever need to take more days off, we can".
This just seems like a warning sign. If a company is advertising unlimited vacation policy, I would expect more vacations not less. All my jobs had at least 2-3 weeks off. So I always assumed unlimited vacation means less than usual vacation time.