I've been reading that the Raspberry Pi 4 tends to run very hot. I saw reports as high as 80 deg C with the case and no active cooling.
You will certainly need to cool the chip to get any meaningful overclock. In that case it is different from previous versions of the Pi, when you could overclock a bit with no extra cooling at all.