I ended up doing something like the following:

    # Dont recompile assets unless they hange
    task :precompile, :roles => :web, :except => { :no_release => true } do
      from = source.next_revision(current_revision)
      if capture("cd #{latest_release} && #{source.local.log(from)} vendor/assets/ app/assets/ | wc -l").to_i > 0
        run %Q{cd #{latest_release} && #{rake} RAILS_ENV=#{rails_env} #{asset_env} assets:precompile}
      else
        logger.info "Skipping asset pre-compilation because there were no asset changes"
      end
    end

I'm not entirely happy with it and don't even know if it's something I should be doing. I rarely change my assets, so this at least allows me to do quick deploys until an asset change comes. (shrug) It feels incredibly hacky.