What has mattered more? The only reason oil is valuable is because it powers our technology. The contribution of agriculture to global GDP is about 4%. Yes you can't eat a smartphone but absolutely anybody sound in mind and body can grow food. Not so many can design smartphones. And they do matter[0].
[0] https://qz.com/901357/selfies-are-great-but-smartphones-have...
Take all that away and you're Zimbabwe, with 0.3 tonnes.
Humans already command or control some 25-40% of all primary plant production (NPP, or the photosynthetic ceiling), much of that made possible through modern mechanised architecture.
In the works already mentioned (Smil, Weissenbacher), the immense increases in ag productivity through such advances as assisted irrigation, mechanised plows, disks, seed drills, fertiliser, harvesters, etc., etc., are addressed in detail.
GDP, and economic prices generally, are exceptionally poor measures of net contribution and value. (Or of true real costs.)
Northwestern University economist Joel Mokyr, otherwise quite a technological optimist, discusses the failure of GDP to capture the true impacts of, say, antibiotics or vaccines. Their price-based contributions to GDP are de minimus, but their impacts on overall well-being are tremendous.
http://www.nationmaster.com/country-info/stats/Agriculture/C...
Perhaps you only know tech oligarchs. But if you look at the list of billionaires, you find also mining oligarchs and fossil fuel oligarchs. So much for the argument that raw materials do not matter...
Physicists invented nuclear racketeering, which made them celebrities in the 1940s and 1950s. We live in the age of the MBA, not in the age of the scientist. That golden age is gone, unfortunately.