First, the more difficult a task is, the more inherent difficulty there will be in "accurately" estimating the difficulty of the task. Fibonacci is used to represent the inherent lack of accuracy in more difficult tasks, since the numbers get _very_ far from each other as they go up the scale.

Second, the numbers _are_ arbitrary. Completely, 100% arbitrary. It's a _relative_ difficulty scale. Say you've got 3 tasks - A, B, and C. A and B are approximately as hard as each other, they're 1 story point. C is more difficult than either one - it gets 2 points. That's it. Story points are not, and should not be used as, a unit of measurement. The biggest utility is to identify big, scary tasks with lots of unknown factors.

The fact that they are _numbers_ is what tricks so many teams/PMs/management/etc into thinking that story points are more meaningful than they were ever supposed to be. Incidentally, this is also why some planning poker teams use t-shirt sizing (S,M,L,XL,XXL, etc). No numbers means people are less tempted to punch them into a spreadsheet while deluding themselves into believing that showing numbers going down is the same thing as "showing progress".