stephantul

I only just saw this, thanks! Will check it out

This hits really close to home!

I like thinking of types as a syntactical layer, just like in natural language. Without knowing the meaning of words, we can show that certain compositions of verbs, nouns, etc. can a priori never lead to an intelligible sentence. So trying to actually turn the sentence into a meaning or reading it (compiling/running) is useless.

But then again I was trained as a linguist so…

Name is correct, thanks a lot, I'll check it out!

Thanks a lot! Ordered a copy :)))

I don’t think that’s what this would look like. It’s more like: people won’t even know Wikipedia exists or that there is something to contribute to, or that the model gained its knowledge from this resource. That’s why this pattern is so disingenuous in my opinion. When I was young we were surprised that Wikipedia existed. Future generations might not know that Wikipedia exists, or that you can contribute to it.

Current frontier models couldn’t exist in this form or wouldn’t exist at all without people putting in the work of writing down what they knew for free. And the model creators not even paying lip service to this, and instead saying they will replace Wikipedia, is hubristic and a very clear example of the tragedy of the commons.

This so much. I love coding. I might be the person that still paints stuff by hand long after image generation has made actual paintings superfluous, but it is what it is.

Great post. I love the vampirism metaphor.

The internet as an open resource has been a tremendous boon to society as a whole. AI, likewise, acts as a force multiplier on top of this knowledge. You can learn anything. But AI also depletes the resources on which it builds.

An obvious way to counteract this is for the AI companies to give back generously through monetary donations or, at the very least, attributions. But unfortunately we see exactly the opposite.

As someone who works at Ecosia, thank you for this. I’m used to people being skeptical about Ecosia’s business model. But this is something else.

Usearch is a vector store afaik, not a vector db. At least that’s how I use it.

I haven’t compared it to lancedb, I reached for it here because the author mentioned Faiss being difficult to use and install. usearch is a great alternative to Faiss.

But thanks for the suggestion, I’ll check it out

Check out Unum’s usearch. It beats anything, and is super easy to use. It just does exactly what you need.

https://github.com/unum-cloud/usearch

This was interesting to see happen live on X

I think putting this on the llm is a bit generous. Their results were apparently 30x above the theoretical maximum, according to gpu master Tri Dao, so there was also a lack of understanding on what was possible with CUDA. See:

https://x.com/tri_dao/status/1892610951662153945

Also see this thread by Lucas Beyer: https://x.com/giffmana/status/1892510741242036468

One of the greatest skills in research is to remain skeptical of one’s own results, especially when they are exceptional. They chose to pull the trigger, and release too quickly.

This can happen in any setting, not just codegen, e.g. inadvertently training on the test set. Science is a slow ascent: if it looks too good to be true, you probably just have a bug.

Written by someone who is probably already disgustingly rich

I think this is interesting, but I think the way Python is described is slightly wrong? In Python, you don't need to define a protocol to support duck typing, you can just implement the same behavior. Protocols don't have a function outside of static typing (they don't influence any program behavior at run-time.)

I guess it depends on whether you consider duck typing to be "passes the static type checker" or "it works the same".

That's a good point! In many WordSim tasks, the words are pretty rare on purpose, so this is very likely. But it provides a good way of testing this hypothesis. Nice, we'll check that out.

Hello everyone,

here's some experiments we tried in distilling a ModernBERT model into a smaller model. Hope you enjoy!

Hello,

today we released Semhash! Here's a blogpost on how it works. We did a show HN yesterday, but that got deleted for some reason.

Let me know what you think!

Ada is deprecated: use the text embedding models instead.

It depends on whether you do full text search, or abstract only. If you do full text, I’d guess about 1k tokens per page, 10 pages per paper? So that would be 3B tokens, which would cost you 60$ if you use the cheapest embedder.

If you just do abstracts, the costs will be negligible.

Same! I’ve seen many proposals to use a GNN for some problem for which we used a “flat” model, e.g., taking into account HTML structure when predicting labels for pages. Even when it seemingly made a lot of sense to use them, it didn’t work.

Interesting, thanks! I’ve seen it for importable stuff but never for modules.

I guess I’m getting old

Just out of interest: why is every python file prefixed with an underscore? I’ve never seen it before. Is it to avoid collisions with package imports? e.g. “types”

They are not concatenated, but summed. I think concatenation wouldn’t work, as you indicate.

I think you mean the line in the original paper where they say compared the learned attention weights with the predefined encoding, and it made no difference.

Sorry for spreading false information. I edited the post above.

It is interesting that you’re not as up front about multilingualism compared to cohere. They seem to mention it a lot, which led to my confusion.

Yes, voyage models are API only.

There was a part here about multilingualism but that was wrong! Sorry!

FWIW: Voyage also has separate `law`, `code`, and `finance` models. See [1]

Really cool results, anyway.

[1]: https://docs.voyageai.com/docs/embeddings

I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient.

This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.

I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index.

Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

In short: their finding is that quantizing a model undoes various “unlearning” methods. An unlearning method is a specific update to model weights that make it forget specific facts. These are often meant to satisfy copyright claims, although I don’t know if these are ever used in practice.

I feel that this needs a good threat model analysis. Like, you possess an fp32 model, which someone has fine-tuned to forget some facts, which you can then quantize to recover those facts. When would this lead to a dangerous situation?

Unlearning is not necessarily “guard rails”, it is literally updating the model weights to forget certain facts, as you indicate. Guard rails is more like training the model to teach it what is acceptable and what isn’t.

I think it is good this post was written, I learned a lot, but it makes me sad that it was prompted by such an obvious trolling attempt.

It works because they use really large ngram values, up to 6. So most character-level information is in these subwords.