1. There seem to be a consent among Gophers to avoid frameworks where possible and use libraries instead. Frameworks are good for a quick start but when you hit their limits it is very difficult to break out of these limits. Libraries allow to compose an application out of independent building blocks. Your main package is always the one in control.
2. "backend/server" can have different meanings. If you can get more specific about the kind of backend or server you want to build, it will be easier for you to find the appropriate libraries (or maybe even one or another framework). Two good places to start are (a) a GitHub search with the search terms "language:go sort:stars <your search term>", and (b) if you prefer curated lists, awesomego.com. You might even try searching GitHub for "language:go sort:stars django" to see some Go projects based on Django's patterns and ideas.