Most programming languages manage to get this right. You have a quoting syntax with an escape character that lets you express the quotes, the escape character, and any arbitrary character (as a byte sequence, as a Unicode codepoint in some encoding, etc.) as well. Now you can do everything. Why not do this?
I'm not going to say this won't be a pain if you decide to write out every file name possible, because it will be, inevitably. But you can supply alternative syntaxes by way of more convenient (if limiting) shorthand - Python has its r"""...""" notation, for example, allowing you to express the majority of interesting strings without needing to escape anything.
You might argue that I've just punted the problem on to the text editor and the filing system. You'd be quite right.
I have recently decided to stop using $ in passwords (defined by me) for that same reason: of course they are valid char, but they are such a big pain to support in usual contexts that it is simply not worth it.
(Thanks for correcting the nul reference)
But I'm going to go back to my original point. What I do have an opinion about is how reasonable it is for tools not to support a character that is valid in file names, when that character is straightforward to support with everyday, well known syntax.
And when that character is ' ', the standard English word separator, as straightforwardly supported by approximately ever kind of quoting or escaping syntax ever, my opinion about a lack of support is: it's crap.
Now do the same, assuming that the password can have a dollar in it.
You can do it. It is not as easy, readable or maintanable. Avoid it if you can.
Extra complication to avoid if possible.
Somehow we started with a single dollar in a string, and now I am dealing with sql, here docs, and TDD?
There is a place for everything in life: for small helper makefiles, which I write simply to support me in my workflow and even to remember some interesting commands that I need to run for a certain project, I can assure you that making some simple assumptions helps me staying halfway sane.