Do Language Models Know How Heavy an Elephant Is?

A good starting point is a system that has a pretrained list of objects it can recognise. For example, we can train our image model what a book is, so when a user asks for a book, we can correlate the "bookness" feature (trained part of the output vector from the vision system) to it 1-1, and connect the bounding box from the vision system to the named entitity "book". However, say a user instead asked for a "novel". While this is also a book, we need a way for the two to be connected. For this, we use a quite different type of language model to connect the named entity "novel" to a list of known objects, one of which is "book". It does this by scoring how similar words are in the language model, and picking the closest match that is in the list of known objects. Over some threshold of similarity (measured by distance in abstract word-vector space), it is determined that this is a new, unknown object. What happens next varies, but is usually some variation of the system guiding the user to teach it what the object is that they meant (something like: point to the object, or put the object on top of a special evenly lit blank background for the visual system to learn it's features), so it is known for next time, and given an identifier from what the user called it.

This is the most basic form, but (as with everything) it gets more complicated. Two areas I am looking into at the moment are past reference language learning, which is about learning new visual features from users in operation. For example: user refers to this particular book as an "open textbook", later asks for a "closed textbook". Model knowns "open" and "closed" are mutually exclusive, so looks for a second match for book that is dissimilar in some visual features to the first, and learns some weak relation in those features to "open" and "closed", possibly to be reinforced later by repeated user expressions.

The other area I am looking at is learned spacial relation grounding. That is, trying to be able to resolve something like "The third book from the left" or "the book on top of the blue table". This is interesting because it breaks a lot of current grounding approaches, in that it requires some recursion (you can chain spatial descriptors in natural language e.g. "The book on the table in the room in the leftmost house on the hill...") and self-referential techniques. Ways of doing it aren't as well researched, and we're working on a method for doing it atm :)

I would expect any language model to be robust to minor errors like that though so I doubt it makes any difference.

It seems to me that these kinds of questions could be handled but require some adjustments or tuning more than we need fundamentally different approaches. We can't do everything at once so there's going to be simple things that don't work for quite a while.

I guess it depends on how many hairs you want to split and how precise you want the answer

Half of the weight of an element would be well defined since it is just a number.