> Everything has to have story points assigned to it just so we can say we did an arbitrary number of points per sprint

If you have a manager that's treating N points as a target, then you just have a bad manager. Sprint points are just a signal for how over/under your estimates are on average, to help inform future planning. If someone goes on vacation, how do I know how much work the remaining team can handle? It's also a good signal for measuring the impact of team/process changes (to be clear, the idea here is post hoc analysis and not +N points as a target).

That said, how would you prefer to handle capacity planning? Points aren't perfect, but trends should stabilize over time (you can have predictability/consistency without precision). You can even map point values to ranges of time (e.g. 1 point = [2, 8] hours) if it helps.