stephantul

No

The (relevant) segue into tires elevates this post so much. I don’t understand why, but it does

Agreed on all counts.

In many cases I’ve found directly using python primitives to be less confusing than pandas.

Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.

If you find a place where I can host a trillion parameter model without anyone finding out about it, let me know.

But how. Models don’t have access to their own weights.

Not the code: the weights. Are you going to host a trillion parameter model somewhere without someone noticing?

Ok but do any of these data centers have a copy of the models that attacked hf?

One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.

So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.

Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.

I think anyone who is even a little bit realistic knows that most technologies overclaim, or evaluate under very favorable conditions.

This is not a good thing of course, but I also feel that acting surprised that this is going on is a little unnecessary.

Having said that: most tools are not helpful

I’ve never liked that this was called “the platonic representation hypothesis”. Lots of weird baggage attached and seems like a waste of a good name.

This is a super long ai generated slab of text with very little actual info. It doesn’t include any analysis of results.

It does include the phrase “worth listing”.

Super charitable reading imo. This is like saying we can’t detect a speeding car because we can’t run as fast as a fast car.

It’s not like the humans were engaged in some kind of battle of wits with some super AI, it’s just some employee not monitoring the output of an experiment.

Cookie banners are made annoying on purpose. This has nothing to do with GDPR itself.

The entities forced to show them would rather not, and thus make it as annoying as possible for you. They then use this to weaken support for the GDPR.

Shame on the people making stuff like this.

Tragedy of the commons. The person that ruins a commons first takes all the supply.

This is why regulation before, not after, the commons are plundered, is important.

Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.

Of course having a holdout set is better than not having one. It’s just not a silver bullet.

It is perhaps ironic that I find this post very difficult to read. I'm super interested in the content, but it reads like it is generated.

My heart bleeds for Rovo’s product managers. They must also see that nobody actually wants or needs or even likes Rovo.

Anyone have info on whose regulatory restrictions they are referring to?

I think that some of these choices (as others have commented) show that the author has not investigated how tokenization works.

Tokenization is not some black box, you can run tokenizers and check them.

PCA is applied after the model, so there should be no difference in embedding throughput. Lookups in the index should be faster, but that speedup also applies equally to MRL.

So I guess the answer is: no

Ah I meant more to say that I was working on this as well. I haven’t published the results for this comparison specifically yet.

Nice! I’ve been working on something similar and found similar results.

In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though

I also wrote about this exact comparison when using PCA and MRL to quantize static models, see: https://stephantul.github.io/blog/mrl-pca/

LinkedIn recently added a “seems like AI slop” button. I.e.: independent of downvoting/not interested/flagging as spam/ToS violation, you can say “this is AI slop”. Maybe we need something like it here

That is true, I’ve seen people do biochemistry and geology work, and it did look very mind-numbing.

Then again, gassing rats and taking biopsies is not something you can do with AI.

I’ve always felt that the idea that science is bottlenecked and therefore needs more automation only works for a very narrow definition of what science is, and entails a very specific view on what it should be.

Love it. I especially thought the introspective “aside” section on related resources was great, and I wish more people would show this kind of reflection.

I’m in a similar boat. The one thing that seems to help is recognizing when the ask from leadership is genuine, and also sensible from a product point of view. Seize that moment to go fast, and give that your full attention.

The other project will either peter out, because they made no sense from a product point of view or because leadership lost interest. If they’re simple, can likely be done quickly with the help of AI.

It’s not a pretty answer, but AI has helped me cope with this kind of situation much better than in the past.

100% generated. I skimmed the paper, and came out with a feeling of still not knowing what this is about.

It is hilarious to see most comments are people peddling their own products more or less directly.

They sell their products using the credentials they gained.

I’d never heard of socket until they found and reported shai hulud hiding in pytorch lightning. It pays off.