The problem with invented languages is that it is almost always possible to find some easy criticism of it (e.g., inherent sexism in vocabulary), and since it's invented, you can't shrug it off with "historical baggage" like you can with natural languages. When trying to design it for an international audience, and trying to avoid tying it to a specific language or language family, it's basically impossible to find sufficiently universal constructs.

Phonetics is the easiest thing to find a universal subset for--there's basically only so many ways you can physically structure your mouth to make sounds (although it does operate rather on a continuum basis). We can also objectively say that some sounds are easier to produce than others. Even then, the intersection of all languages' phonemic inventories is probably empty. The basic structure of a dozen or so consonants (m, n, b, p, j, s, z, k, g, d, t, w, h, say) with 5 basic vowels structured in a strict CV syllable format, with no diphthongs and vowel length, aspiration, and tone all being non-phonemic is the basic outline for how to design the phonotactics of a universal language.

Moving beyond phonemes is virtually impossible, however. Trying to design a basic lexeme for the language obviously runs into problem of where you source the lexemes (particularly since borrowing from several different languages tends to make it completely incomprehensible to everybody, although you may find that more tasteful than prejudicing one language's speakers as having an easier time mastering the lexeme). But even choosing which words to make root lexemes and which ones to make compounds is difficult. Should blue and green be separate lexemes? What about blue and azure? Should right be "not-left" but south not be "not-north"? Languages often attach multiple meanings to the same word, and these meanings don't always match up across languages.

Grammar is even worse. It's hard to find any grammatical structure that's truly universal across languages; maybe the separation of function between nouns and verbs, but the proclivity to verb nouns is so common in languages that it's questionable how fundamental the distinction actually is. Even the basic meaning of things like mood, aspect, and tense can be hard to settle, since the forms that exist in languages can partially overlap certain combinations (past/present+future versus past+present/future forms being the most well-known). Having to mark cases that don't exist in your native language can be difficult; English users' trouble with the subjunctive is well-known.