I completely agree - the more I used it the less I liked it. I really enjoyed its implicit simplicity at first. The code I wrote had purpose and was clear, consice, and easy to understand. However, as the codebase grew larger I really ran into the issues more people talk about. Specifically lack of generics. I was using reflection to simulate this, however my performance was terrible on large sets of data. I had to end up relying on a code generator, which seems to be the direction Go is heading.
Edit: Package versioning was another pain point, as someone mentioned below. Their built in package manager is almost too simple. As with anything simple, it's great at first but not so great when more complexities arise.