Yes, the Creator knows what is best for us, but, no, It has given us free will and honors it until we beseech It to help us.
If you make a prayer with all your heart asking for the Creator to take Its Spirit (our conscience) back into Itself so that we can cleanse and purify our souls of our vice-oriented ("vice-eous", the opposite of virtuous) tendencies, thereby becoming a friend to all, becoming, by degrees, consumed by compassion and a possessor of wisdom, you will be changed. It is the meaning of the 1st Beatitude ("blessed are the poor in spirit, for theirs is the kingdom of heaven"), and is the prerequisite to (IIRC) the 5th ("blessed are the pure in heart, for they shall see God").
This is a subjective experience. Most people are too wrapped up in their selfish desires to see how different a person is who has undertaken this process of moral self-evolution. I have met two such purified men (Murshids) and a third (from a different faith) whose teacher was also such a one. I am on the way for a couple of decades now but am not purified, only semi-cleansed. Happiness is different for me; it is a sublime joy, a pleasure to help a person in any small way.
Yet, I am rejected by my family and lifelong friends, but they are only rejecting happiness because their forms of religion (especially those with none at all) reject the path I am on. Our son is an open chess champion (3x!) and our daughter is an accomplished seamstress while still in her teens (Singer 503a is badass). We do not reject any form of religion, but understand that we are all one human race and should care for one another and create peace here on our blessed Earth.
This is the subjective truth that we live but is rejected by nearly all; such is the world we live in, of selfish idiocy and destructive strife. But what we Sufis teach and live is truly the science of the soul. In history, all groundbreaking science is rejected, be it that of Eugene Parker, Boltzman, or even the doctor who found that most stomach ulcers were caused by a simple bacteria. That others reject the truth that we live -- and they can, if they try -- is not a testimony about us, but only about them.
I love you. I have no selfish motive in telling you the truth here. I want nothing from you. I only want for you to be happy, and to spread happiness to all those you experience, by both your actions and explanations. You can verify this for youself, but only if you jump in with both feet.
The great Islamic mystic, Rumi, said 800ish years ago, "The Way goes in."
The truth is that we human beings could be creating a must happier world, but selfishness is ruining it for everyone. There has been talk around HN lately of "mathematical thinking" but who contemplates how we would change things if selfless compassionate service was our systems' intention and goal, instead of their current motive of profit-at-all-costs? We Sufis understand that calculus and understand that it begins with each of us. We inhabit a world that pushes us to be selfish animal-like creatures, when we could instead be humanitarians that care about everyone.
We could create a simulation of such a transformation now, but there is a force within each of us that pushes back against such notions, decrying them as impossible. You will feel that push back as you read the deepest truth in the universe. It will say that I'm crazy or don't know sh_t. Just please remember you are free to choose, and pay close attention to what that negative inner voice says. Perhaps it's not you, and is, as Castaneda's Don Juan explains, a parasite of our mind. We Sufis have a much simpler explanation: there is an enemy within.
Peace be with you.