Say you have 64GB of RAM, of which 4GB are dirty pages that are never, or at least very rarely, accessed. With 64GB RAM + 4GB swap, you filled up your swap with them and have 64GB RAM left to use actively. With 64GB RAM + 0GB swap, you only have 60GB RAM left to use actively.