Common Lisp has literal syntax for vectors (and it wasn't the first Lisp to). Its standard representation of a set is just a list, but if someone were to add proper sets to the language they could add read syntax for them easily
CL also has read syntax for maps (as has every Lisp) in the form of alists; it doesn't have literal hash tables, but those aren't terribly useful. Alists perform better for lookup for less than 20 or so keys (their writes are always constant-time), and they aren't going to be a bottleneck until you have at least several hundred keys, at which point you probably won't be writing those as literals in your source code. As well, alists have several other nice properties over hash tables, like not being tied to any equality predicate.
Alists are a standard Lisp map with literal syntax that's been around since the beginning and are often preferable to hashes, but if you really did want hash literals specifically, it's a trivial read macro.