Offline batch learning is not contradictory to reinforcement learning. It just requires that it be an off-policy RL algorithm, which happily, many, like DQN, are.
> Similarly, the amount of driving data collected by Tesla from all its cars will be much larger than the data received by any single car.
Well, Tesla benefits from the fact that its cars are already agents, acting in the real world. So using it as a counter-example doesn't really work... You would need to imagine some sort of counterfactual Tesla which eschewed any real-world deployment or simulators. Which no one would ever do because the usefulness of trying to create a self-driving car database without agents or reinforcement learning is so obviously zero.
And anyway, most of Tesla's data, however, will be totally useless. You can train a simple lane-following CNN with a few hours of footage (as Geohotz demonstrated), but another million hours of highway driving is near-useless and doesn't get you much closer to self-driving cars (as perhaps Geohot also inadvertently demonstrated). What you need is the weird outliers which drive accident rates. So for example, the Google self-driving car team doesn't depend purely on collected data even from its agents, but simulates very rare scenarios as well. (If I may quote myself: "You need the right data, not more data.")
> This suggests that most learning will happen as a batch process, ahead of time. There may be some minor adjustments for personalization, but it doesn't seem like it's enough for Agent AI to outcompete Tool AI.
Even if we imagine that the dataset covers all possible sequences and doesn't need any kind of finetuning the loss, the agent AIs in this scenario are still going to benefit from actions over (going by section): 'internal to a computation', 'internal to training', 'internal to design', and 'internal to data selection'. In fact, if you want to have any hope of running effectively over extremely large datasets like hundreds of millions of sentences, you are going to effectively require some of those to keep training time tractable and runtime cheap (even Google can't afford a million TPUs for Google Translate). For example, regular hyperparameter optimization using a few dozen examples doesn't look very good when it takes a month on a GPU cluster to train a single model, but using a meta-RL RNN to do some transfer learning and pick out near-optimal architectures & hyperparameters in just a few iterations looks pretty nice.