If the quality of your estimations has ever come up on your annual review, that's them bargaining you down by making you feel bad about yourself.
Someone in a video I watched recently pointed out that story points per week is a graph plotting time on both axes.
One of the earlier agile methodologies (FDD) had one thing figured out: the law of large numbers works just fine for long-term estimation, as long as you can identify the stories, and the range of story 'sizes' is within an order of magnitude (eg, a day vs 2 weeks). You don't have to give a shit if a story is 4 points or 7. That's a waste of everyone's time and especially energy. It's horizontal aggression condensed into a management model. We need to start refusing, as a collective, to engage. The only discussion you need to have is whether this story is less than two weeks, more than two weeks, or way more than two weeks. Those happen at a much lower frequency.