Let's say we pick a random person and make them learn a completely unrelated language. Sometimes they will learn faster than others, like Spanish or Swahili vs English or Mandarin. Why would we expect them all to be the same difficulty?
So I totally believe that it's easier to learn English than Mandarin (and maybe easier to learn Spanish than English) because the writing system gives you more hints, but that doesn't have anything to do with the (spoken) language, strictly speaking.
On another note, it's not clear to me that "as phonetic as possible" makes a (written) language easier to read, since our brains process words/morphemes as chunks, rather than sounding them out letter-by-letter (this is how reading Chinese is even possible). So semantic vs. phonetic writing should be thought of as a tradeoff. As an easy example: spaces between words are not phonetically justifiable (there is no pause between words in natural speech), but they sure help reading comprehension a lot.
I totally believe that it's easier to learn English than
Mandarin (and maybe easier to learn Spanish than English)
because the writing system gives you more hints, but that
doesn't have anything to do with the (spoken) language,
strictly speaking.
I realize that this isn't practical to actually do as an experiment with all the world's language pairs, but I do think it demonstrates that inherent difficulty is a thing.
One thing that sort of shows this, though it's kind of distorted by writing systems, would be charts like this from language learning companies: http://www.effectivelanguagelearning.com/language-guide/lang... For example, they think for an English speaker Swahili is easier than Nepali is easier than Finnish.