My big problem with estimating in story points is that it ignores the reality that how long something takes is directly tied to who is doing the work. A junior eng, a senior eng, someone who just started, someone who isn't familiar with that piece of the system, the person who wrote and has always maintained that piece of the system. These people will all do a given task in a different amount of time. Possibly an order of magnitude differently when comparing a junior eng not familiar with the code to the codes author. This is okay. Embrace it. Plan for it.