Usually, you can. But occasionally you get mildly defective tools that require some directory to exist, even though it's empty. It's easier to add a gitkeep than fix them.
Your point is valid though. It would be much preferable to include build/ in your root .gitignore so that the directory is never tracked.