What you describe is almost exactly what Pebble is offering: one of their new watches + the Pebble Core (GPS + Cellular + headphones jack)
Add to that the fact that it requires its own 3g connection, which will add a monthly cost. I'd much rather have built-in storage sufficient to store audiobooks and podcasts...
> You can also use Core’s 4GB of built-in storage to access your favorite playlists, podcasts, and audiobooks.
Not sure if it will work with Audible specifically, but it should work with normal audiobook files.
You do not have to have a 3G connection to use it. It has WiFi for offline syncing.