So, JSON has non-string values in other positions (as elements of arrays, or values in an object). Wouldn't your argument also lead to the conclusion that we don't need numbers at all, since we could get by with

{ "foo": "42", "bar": ["1", "2", "3"] }

There's also the issue of values with multiple equivalent string representations. I want 42.1 to equal 42.10 and 42.100. I also want {"foo":1,"bar":2} to equal {"bar":2,"foo":1} but with just strings you don't get that:

{ "{\"foo\":1,\"bar\":2}": 1, "{\"bar\":2,\"foo\":1}": 1, "42.1": 2, "42.10": 2, "42.100": 2 }

should have 2 keys but has 5