Let's say your GPU uses a constant 300W. It takes approximately 1.16 Wh to heat a liter of water 1 degree celsius under 100% efficiency. This setup won't be 100% efficient, so let's round up to 1.5 Wh per liter per degree (roughly 80% efficient). A typical domestic water heater in the US 40-50 gallons, so roughly 160 liters.
Let's say your incoming water temperature is 18C and you want it preheated to 50C, which is 32C degree differential, which means you'll need 32 * 1.5 * 160 = 7680 Wh, or 25 hours straight to heat the buffer tank from scratch.
You'll need to purchase a small water-to-water heat exchanger ($50), two pumps ($100 each), a power supply for said pumps, hose and/or copper pipe and fittings, and various other sundries, plus the cost of a buffer tank ($600ish), so figure all in roughly $1000, plus the cost of electricity to run the pumps.
At $0.22/kWh you're saving roughly $450 a year with this setup in foregone water heating, but because it's not 100% efficient you're spending $500 in electricity to run your GPU 24/7/365, and that's the maximum you can possibly save with the above assumptions. Scale up for more GPUs and down accordingly for less usage as you see fit.
Alternatively, use air as the heat conductor by placing the GPU laden machine in the same space as a hybrid heat pump water heater.