& notepad.exe
Which isn't complicated I agree but annoyed me.
Just typing "notepad" is enough to launch it. The problems with powershell and executables only really come in to play when argument passing, where some escaping is necessary.
If you want to execute the EXE stored in a variable or whatnot, then yes, you need `&`, which I think is borrowed from Perl. But otherwise, the rules in PowerShell are the same as a Unix shell: if it's in the PATH, just type the command name (with or without the .exe); otherwise, use the full path.
The SO answer isn't wrong. The EXE there has a space in the path. The answer uses a string literal for it and thus needs &. It could of course escape the spaces with ` instead; then it wouldn't need a string literal and thus not need &.