Why do you like managing your own memory exactly? Modern GCs are quite good, so the domain in which you can beat their performance is narrow and will continue getting narrower as time goes on.
GCs are useful in the same way that Python is useful even though it is slow. Simplicity of use has intrinsic value but that is a tradeoff against other objectives.
You can say the same about any abstraction, such as having a generic memory allocator, using a standard library, actually about any code reuse. For example, many programs use printf, but don’t need all its features, so hand-rolling a custom version may lead to smaller, faster code (things get complicated here on systems with shared libraries)
The question always is whether lost performance (execution time, memory usage) is worth faster development.
My point is that you can start out without a GC and do a series of sane and effective optimizations that end you up with a GC. Just as you can start with a GC and optimize by moving more and more of your allocations to non-GC memory and end up without a GC. Which endpoint is faster depends on the workload.
I have also worked on projects (web apps) where GC pauses became annoying to end-users. I wouldn't say this is the fault of GC perse, but writing code without a GC, seems to prevent some performance problems for me, and can also make performance problems stand out sooner in the dev cycle.
Of course, many GCs have indeed gotten better in their own right, but not using the GC at all remains the most potent optimization.
Not in my experience.