The courts. See the link.

In practice, this definition has rarely [1] been a problem. There are more serious concerns.

For example, a real problem with possession-based laws (as opposed to distribution-based laws) is how comparatively easy it is to frame innocent people [2]. Law enforcement likes possession-based laws, because they are an order of magnitude easier to prove and prosecute.

[1] Which is not to say "never".

[2] http://www.bbc.co.uk/news/uk-england-london-11397515

A jury of your peers, applying the "man in the street" logic.