stephantul

Not that I know of, but encoding orthography in a fixed width vector usually carries the assumption that words with the same prefix are more similar. So there’s an alignment problem. You usually solve this using dynamic programming, but that doesn’t work in a vector.

For example “parent” and “parents” are aligned, they share letters in the same position, but “skew” and “askew” share no letters in the same position.

I was trying to say “at least for alphabetic languages”. I don’t like to say things about languages I can’t speak or write. So, no, it wasn’t my intention to say “non-logographic languages”

IME, and as the article also kind of says: paper mills target academics from developing countries. Academics who have trouble competing because of lack of funds, or lack of proper support from their universities, language barriers, cultural barriers, etc.

As you mention, being in academia is tough. Some people just can't compete, and then get lured into accepting offers like these. In a way, I think this is similar to how vulnerable government officials get bribed. It's not something you do unless you're rock bottom, and don't see anywhere else to go.

Using subwords makes your sequences shorter, which makes them cost less.

Besides that, for alphabetic languages, there exists almost no relation between form and meaning. I.e.: “ring” and “wing” differ by one letter but have no real common meaning. By picking the character or byte as your choice of representation, the model basically has to learn to distinguish ring and wing in context. This is a lot of work!

So, while working on the character or byte level saves you some embeddings and thus makes your model smaller, it puts all of the work of distinguishing similar sequences with divergent meanings on the model itself, which means you need a larger model.

By having subwords, a part of this distinguishing work already has been done by the vocabulary itself. As the article points out, this sometimes fails.

Hey, you are thinking about wordllama. We explicitly compare to wordllama in the results. Feel free to check it out.

Your point about BM25 is very interesting. We can run some comparisons! Thanks

Can you give more info?

There’s some old work [1] that conceptualized sentiment as an interplay between subjectivity and sentiment. The more subjective a statement, the more “range” sentiment gets. I think this is what you are getting at.

I don’t think it ever gained traction, probably because people aren’t interested in creating an actual theory of sentiment that matches the real world.

[1]: https://github.com/clips/pattern/wiki/pattern-en#sentiment

Hey, thanks for the question.

You are right, this can only be used to distill models for producing embeddings, although it's not restricted to encoder-only models. For example, you could use it on Llama, but you'd just get a bunch of embeddings out, not a model that can be used to do next-token prediction.

Hi Kyle, nice to see you on here. Happy customer here from Ecosia

That’s an interesting idea. I wonder whether it would actually lead to radically different solutions, bit with bigger codebooks.

This is very nice and important. We need more tools for small languages.