If you're #50 in the PGA in terms of average ball speed [0] and drive a regulation ball (46g)[1] at 175 mph (78 m/s), that ball has an energy of 140 J. If it accelerated from 0 to 78 m/s in 0.5 ms [2], it accelerated at 156 km/s^2 for that short time, at a power of 280kW (375 horsepower). The force applied to the ball was therefore 7100N, corroborated here [3].
A driver is 45" standard, but assuming that you don't hold the club at the tip we can call it 42" or 1 meter. You don't rotate the club where you hold it, but rather roughly at your shoulders (I will neglect hip rotation), and the average arm length is 25". Arms are not locked straight so let's call it a little shorter, and add 0.6 meters for the arm length.
You now have a 1.6m moment arm applying a force of 7100N through the ball, requiring 11.4 kN-m of torque, or 8400 lb-ft. Edit: I have discovered that wrist torque is significant, and that your wrist and your arms individually supply about 150 and 400 N-m of torque respectively. [4]
To compare, a high-end sports car might have 500 HP (compared to the golfer's 375 HP), and the Tesla P100D has 920 lb-ft of torque.
Of course, the golfer sustains this output for only half a millisecond, and a lower output during the rest of the swing (around 1 second). A car can do it until the tank or battery is empty.
Also I have neglected hip rotation, differing driver and arm lengths, and energy loss due to deformation. But this is a decent first approximation, and at very least in the correct order of magnitude.
[0] https://www.pgatour.com/stats/stat.02402.html
[1] https://en.wikipedia.org/wiki/Golf_ball
[2] https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/11470...
[3] https://hypertextbook.com/facts/2001/EmilyAccamando.shtml
[4] http://rspa.royalsocietypublishing.org/content/465/2102/551