Single point of anecdata: I found that using the flirc passive cooling case + putting the whole thing on its side dropped my average temp a full 20°C (from about 65° to about 45°) while running a Plex server.
Have you used a heat sink in your tests or did you used a barebones board?
previously I was using the heat sync that came with the 4B, inside the ABS plastic case that came with it (both in the canakit set)