I haven't tested this since then, but this wasn't the case in 2014[1]. I haven't noticed these issues since then, but back then it cost hundreds of microseconds -- 8 orders of magnitude more than "a few ns".
EDIT: I just re-did the relevant tests on Go 1.11.5. It's significantly better than in 2014 but it still costs between ~20us and ~50us ("only" 4 orders of magnitude more than "a few ns").
goos: linux
goarch: amd64
BenchmarkPut-8 50000 27502 ns/op
BenchmarkPutDefer-8 30000 46774 ns/op
BenchmarkGet-8 50000 29812 ns/op
BenchmarkGetDefer-8 20000 89701 ns/op
PASS
[1]: https://lk4d4.darth.io/posts/defer/