Dollar ($) as prefix to variable names is actually quite good. You know instinctively what is a user created variable and not a language keyword. Easy to find variables when scanning large chunks of code and easy to use with string replacement to avoid string concatenation or sprintf.
Opening tag at the beginning of file is not really worth of doing anything about. If you separate concerns in your code, you don't do HTML in your model code, therefore the only tag in a the vast majority of PHP files is the opening tag a the top of the file.
Paradoxically, today you don't use PHP as templating language in modern PHP, because it is clunky. You use a templating language like mustache or twig.
Standard library, yes. That ought to be fixed. My proposal to solve that is to implement function autoloading and support for callable types (parameter and return type signatures to callables). With that you could start implement outside of PHP core and start deprecating old core functions and supply a library shim for those who need it.
And also with those two language features, it will be much easier to do functional programming instead of the now dominating OOP i the PHP community.
PHP since 5.5 is quite good and with this release it is even better. It is flexible but at the same time rigid with very good performance. No built in state, which is perfect for stateless API:s.