If you don't mind hotkeys, I eventually discovered that cmd+up arrow will take you to the parent folder.