Words are considered a "discrete unit of meaning", i.e. 3/4 of a word doesn't really mean much. So words like "red" and "grass" are "standalone" in the sense that the mean something by themselves. I agree that words are very much related to each other, in the sense that you can combine them.
I was trying to draw a connection that the "disentangled representations" ML folks often talk about are but a special few-word case of grammars for combining distinct concept.
The discrete unit of meaning level is generally somewhere between a syllable and a word, with a few exceptions for shorter modifiers.
Unfortunately, in linguistics, the concept of a "word" is only as well defined as "planet" was pre-pluto losing its status.
Similarly when you look at riddles and crossword puzzle clues the idea of words being discrete also falls apart. Words, very much like variables in algebra only have meaning in relation to the other pieces of the context they are attached to.
While the mechanics (all the pieces of language, syntax and semantics are not discretizable. Just talk to anyone working on a dictionary.) you talk about don't seem to hold, I do think the idea you're talking about does hold.
Or in other words, I believe the surface area of "edge cases" has a similar surface area as the rest of the language. The difference being they aren't invoked nearly as often because they require more effort and creativity.
Just look at the rise of words like "hangry". There are types of mashups that show up in creative uses of language that defy nearly any rule for any language you can come up with. In many languages, if you choose any of those supposed rules you can probably construct an algorithm to generate odd, but understandable words that defy that rule.
I do mention this in my blog post - although I think Generalization is Language, I don't think it's possible to create a formal framework of language, for precisely because of "adversarial examples" that can be supplied for any formal definition.
Natural language itself, ignorant of formality, is able to account for these exceptions insofar as language is sufficient for people to convey a bare minimum of meaning. I am proposing to define language and generalization via the implicit understanding of large language models, in the same way you might use an image classifier to define "cat images" or "hot dogs"
DL is already far from formal models, that's why deep learning “works.” And even at the current level of DL models, those exceptions are represented to some extent.
So ultimately, your idea is to push the models toward further generality, which in my option, will bake these “exceptions” deeper into the model.
And my question is, what does that mean for your idea? In my mind trying to exclude them would break what works. On the other hand, ignoring them means you can't direct development towards your goal because there’s no map from language to generalizations, so that you would be relying on random chance for progress.
If this is off in left field, let me know, but that's what I can see from your description.
Edge cases or exceptions do tend towards being highly used; this is because language is more likely to change the more it's used, so the most highly used words/phrases/sentences/etc tend to accumulate changes. One example of this is that if a language has verb conjugation and irregular verbs, then odds are some of its most common verbs will be irregular.
> Just look at the rise of words like "hangry". There are types of mashups that show up in creative uses of language that defy nearly any rule for any language you can come up with. In many languages, if you choose any of those supposed rules you can probably construct an algorithm to generate odd, but understandable words that defy that rule.
There are rules for that that would work, weirdly enough. There are just a ton of them.
I certainly think most linguistic typologists would say that there is no cross-linguistic unit that corresponds to our intuitive understanding of word, which is really grounded mostly in orthography.
And I think it's fairly easy to show that orthography should not have much say in this matter, though. Of course you can't get around it in language didactics, but in scientific description we need to be very careful with it. Bob Dixon and Alexandra Aikhenvald give some examples from Bantu languages in their Word: A cross-linguistic typology. In Sotho, the sentence "We will skin it with his knife" is written "Re tlo e bua ka thipa ya gagwe", while in the orthographies for Zulu and Xhosa, the same sentence would be rendered as "Retloebua kathipa yagagwe". You really need to look at each language to find a sensible set of analytical categories, and be very explicit about your criteria, be they syntactic, semantic or phonological.
Funny you use the term morphology like that. To me it's basically synonymous with inflection, very traditional, where morpheme is very much a structuralist term. But all my teachers were cognitive-functional linguists, so everything was cut rather different and sometimes it's hard to talk.
You're definitely right that the definitions aren't cut-and-dried and that makes typology rather difficult.
Language, man. It's weird.
Technically the smallest indivisible unit that bears meaning is the morpheme, not the word. For example the word “cats” in English consists of two morphemes, cat+s. The first morpheme can stand on its own as a word, but the second can’t.
I don’t think you could really give an uncontroversial symbolic definition for any natural word.
Using a classic Latin example from Monty Python, Romani ite domum / Romanes eunt domus;
the "Roman" part of of Romanes/Romani actually carries very much meaning and the -es/-i has information that's largely orthogonal to that.