Anybody can come up with a new language that happily condenses the concepts you need to have in your new world, but forcing the reader to read an encyclopedia to decipher it is just lazy. You are meant to weave the explanations in your narrative, so the reader picks them up seamlessly.
It's a bit like building a UI that is unusable without reading a boring manual.
There are no rules except "be good". I've thoroughly enjoyed Dune every time I've read it and it had quite a profound effect on me as a youngster. There are countless thousands just like me and you can't argue with that really.
> It's a bit like building a UI that is unusable without reading a boring manual.
Well, I think the purpose of a UI is slightly different to that of a novel. One exists to be useful while other exists to be enjoyable. If it's useful it's useful. Doesn't matter if it requires a manual or not.
The best part of Dune is clearly the worldbuilding rather than the prose. Yes it can be used, and the software is great when you get to know how it works, but there is a learning curve that similar software does not have.
You didn't make the distinction "is not necessarily" in your first comment. That changes the connotation.
My favorite part of "A Clockwork Orange" (book) was that very thing: the narrator just talking naturally and leaving the reader to figure out the slang (Nadsat).
There too was a glossary in the back of the book but I soon dispensed with it and found I could get the gist of the slang well enough to just roll with it.