Yes, this is the key. Helm charts should basically be manifests with some light customization.

Helm is not good enough to develop abstractions with. So go the opposite way: keep it stupid simple.

Pairing helm with Kustomize can help a lot as well. You do most of the templating in the helm chart but you have an escape hatch if you need more patches.