Darn, on Heroku even the "maintenance mode" (redirects all routes to a static url) won't kick in.