Sometimes that means better documentation and self-serve content. Or adding better search terms to your support content so people can find it more easily. (If someone writes in on a question your help center doesn't cover, copy your answer to them and paste it into your help center CMS. If someone writes in with a question you already answer, pull phrases and keywords from their email and add them to the answer / tags in the CMS.) Sometimes it's making it clearer within the UI what your app will or won't do. Sometimes it's writing a new feature to appease people.
You want to maximize the number of ghost tickets — the ones that never come in, because they never had to get written in the first place. As with code, "No ticket is faster than no ticket."
Making your support content more present within the app itself, or hooking up your "need support?" request flow so the customer has to go through the support content first can also help. (Obviously, there's a balance between making them jump through hoops and just having an open line to you.)
Eventually I hired one of my most vocal customers (opinionated, but polite) to handle support part-time. Hiring her at an hourly rate meant I didn't have to worry about a lot of the normal staffing issues that come with a full-time employee.
If you do go the route of hiring someone, I highly recommend looking for someone with a librarianship/infoscience background. They're trained to communicate well, to discern the question behind the question, and to do self-directed research. They're also in a market where jobs are harder to come by.