Technological innovations that produced the shale revolution

Fracking is banned in Europe I think pretty much everywhere. It results in earth quakes, underground water and air quality contamination.

But it it provides vast amounts of energy. The US gov cares about different things.

While we have models from geologists and geo-physicists, I think it’s very difficult to work out the best course of action in each case without a lot of prior examples (typically trial and error as you point out).

In modelling oil wells, oil industry engineers typically have lots of variables but far fewer data points (well head pressures, flow rates and hopefully some downhole pressures/temperatures) which makes building a good physics based mathematical model tough. The structure and permeability of the rock matrix is difficult to characterise both before the fracking and after. All situations are different and once you start operations which are expensive, you can’t go back and start again if you make mistakes.

There have been considerable improvements in model theory, sensors/instrumentation and computer power in the past decades. Hopefully this can be harnessed to provide cheaper geothermal power for the future as well as managing oil fields more effectively.

That's most engineering in new fields. Lots of trial-and-error, and empirical data collection. Modelling comes after experimentation.

Edit: for example chemical industry starts with laboratory experiments, then pilot plant, only then scale-up.

I once asked one of my oil majors clients why they hadn’t invested in nonconventionals, and he offhandedly replied, “We don’t understand why fracking works, and we don’t invest in anything where we don’t understand the downhole dynamics.” Basically, there were lots of theories as to why you could get oil and gas out of tight shale after shotgunning it to hell and back, but no consensus, and you can’t get the kind of logging you need under those conditions to really test which theory holds water, so to speak.

Conversely, the big fracking players were much less concerned with, to be completely frank, safety and well integrity compared to the major players, so moved faster, often with troubling but quite profitable results. (Even so, those companies have invested quite a bit in university research to figure out what’s going on, while my client’s company made a major onshore bet, albeit after he retired from leadership.)