I disagree. While both are grammatically sound, "of you" sounds better to the native ear. This is largely due to repetition (humans tend to process language probabilistically, almost as if language were merely a series of Markov chains). Ponder, for a moment, these famous lines:
"Christine, that's all I ask of you." - Andrew Lloyd Weber
"Nor would I ask of any fellow American in defense of his freedom that which I would not gladly give myself - my life before my liberty." - Manchurian Candidate
"And ask of you forgiveness; and so we’ll live," - Shakespeare (King Lear)
These classic lines have made their mark on our culture and/or the English language.
The notable exception to this is when "to" is used to show cause, i.e.
The student asked permission [in order] to go to the bathroom
The man asked a question [in order] to understand the answer