There's a branch of economics, usually lumped in with "heterodox economics" that says that interpersonal comparisons of utility are impossible. This is called the Austrian school of economics, whose figures are von Mises and Rothbard.

Their basic unit of the market is the transaction. You know you value the coffee more than $2 and the coffee seller less than $2, and the other person values it more than $1, but until you and the other person actually make a trade you can't figure out the relationship between your preferences. That is, you are exactly right that you cannot say you prefer coffee twice as much.

Here's a video that explains this idea, called "praxeology" in these circles. He explains it in the first five minutes: https://www.youtube.com/watch?v=TI5fjTz1Rbw