There are two kinds of "cite" here. Citing a non-academic source is different from citing a (published) paper; you should cite anything that precedes your work in the second sense, whereas the first is only obligatory for work that you actually took something from. If you work on calculus you're obliged to cite Leibniz even if you didn't read him, but you are not obliged to cite Newton's unpublished work unless you read it. Unpublihed arXiv papers fall in the latter category.