spindump8930

Likely in this case the time vault was the collapse of Mt Gox, which has now recently been paying back holders.

Some combination of reporting bias given concerns about LLM security capabilities and actual new vulnerabilities found with LLM assistance. Even if exploits and outages are unrelated to LLMs, I'm certainly thinking about whether claude could build these things (or if actors already have).

It's very common if you improperly seed, as others in the thread brought up! Or in your framing, as rare as earth getting hit if it were surrounded by a sci-fi density asteroid field.

Sure, this is cute and interesting, but there's no validation or baselines and those examples are not particularly compelling. The o3 example just lists some terms!

Between the neo and the chances for privacy respecting local model inference, all the new apple hardware has me excited.

That artificial analysis page has some great references for this, thanks for sharing.

Remember that models on different inference platforms might not necessarily give exactly the same results, adding another axis of non-determinism to development. Things like quantization, custom model serving silicon, batching, or other inference optimizations might mean a model from the original provider performs differently from the hosted one :/

This paper isn't the exact same scenario, since it's an auditable open weight llama model, but shows the symptoms of this: https://arxiv.org/pdf/2410.20247

Any more context on the copilot training note? More pointers would be very interesting, but we'd need to keep in mind how many different underlying models were (are?) branded as copilot. I thought at some points the "copilot" model in autocomplete contexts was a finetuned GPT from OAI.

Re: GPL, there are other open access datasets of git repos that make some distinctions between copyleft licenses but those are older resources now.

> The researchers tested five LLMs: OpenAI’s GPT-4o (before the highly sycophantic and since-sunset GPT-5)

Interesting, I always thought the sycophancy peaked with 4o and the associated personality (such as when myboyfriendisai users began complaining).

Hopefully this money means more compute infrastructure to help Anthropic counter the efficiency changes that have created this perceived downtrend in claude quality.

Having known some folks who did recurse, I think places like this want to select for those who consider coding a type of craft or art or self-expression. You can use LLMs, but stand by what you do and have pride in construction.

Not clear that they even have any GPUs yet:

> Allbirds, which will be renamed “NewBird AI,” said it executed a $50 million deal with an unnamed institutional investor to acquire “high-performance GPU assets” to begin transitioning into a “fully integrated GPU-as-a-Service”

Serious folks know it's not straightforward to suddenly get any number of GPUs these days, even at that level of money

Yes, the paper itself tells a different story than the bullet points in this article.

The article seems quite editorialized, shifting between describing "large-scale AI models" and "neural network-based approaches".

The underlying paper itself is more precise, comparing against LUAR, a 2021 method based on bert-style embeddings (i.e. a model with 82M parameters, which is 0.2% the size of e.g. the recent OS Gemma models). I don't fault the authors of the paper at all for this, their method is interesting and more interpretable! But you can check the publication history, their paper was uploaded originally in 2024: https://arxiv.org/abs/2403.08462

A good example of why some folks are bearish on journals.

"AI bad" seems to sell in some circles, and while there are many level-headed criticisms to be made of current AI fads, I don't think this qualifies.

Yes, it's far more certain that meta released this, which is less convincing on evals, as a result of the mythos previews.

Re: changes, there's been enormous turnover in AI organizations, and in theory this one was developed by a "new" org. Whether that means less or more benchmaxxing is anyone's guess.

Spending tons of money on Claude and the recent token benchmarks came WELL after Meta's huge investments in compute infrastructure for AI as well as the long history of language model development inside science divisions at the company.

Only for poor quality systems. Unfortunately there are many systems that tried to make easy hype, but are the equivalent of an ML 101 classifier class project.

If one measures for perplexity (how likely text is under a certain language model), common text in a training set will be very likely. But you can easily create better models.

Pangram has time after time been shown as the only detector that mostly works. And that paper is pretty old now! There are recent papers from academics independently bench-marking and studying detectors e.g. https://arxiv.org/abs/2501.15654

Is the proxy here linkedin messaging/mail instead of direct email?

It was mentioned elsewhere in the thread but this article is relevant: https://www.cbssports.com/mlb/news/guardians-reliever-emmanu...

The ability to bet on short term individual events (such as a single pitch) means that even a single pitch, otherwise nearly inconsequential, can be abused.

For many of us a better Turing test is contextual to a topic we CARE about. Lots of LLMs sound better than a randomly sampled human on a topic I don't know too much about (e.g. opinions on new movies). They're decent on engineering topics I only vaguely know about, but still below the bar (though getting better!) on topics I really care about.

Fairly certain that AI (meaning an expensive llm type model) isn't needed to detect spam a large amount of the time. Classical classification methods could work while also being more privacy friendly (e.g. running on device).

> According to AT&T, a big difference with its product is that it is built into the network itself.

This doesn't seem like an asset...

I recall that there were similarly motivated lawsuits for the earlier answer boxes that used to appear (prior to direct genai injection into the SERP page). What ever happened with those? Finding it difficult to search for.

That's not what this is about.

"I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago" looks like you're using llama-cpp in that repo. This is about vllm serving many requests at once, at long sequence lengths.

> As it turns out, our request’s output does depend on the parallel user requests. Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass.

Your situation isn't really comparable.

This topic is interesting, but the repo and paper have a lot of inconsistencies that make me think this work is hiding behind lots of dense notation and language. For one, the repo states:

> This implementation follows the framework from the paper “Compression Failure in LLMs: Bayesian in Expectation, Not in Realization” (NeurIPS 2024 preprint) and related EDFL/ISR/B2T methodology.

There doesn't seem to be a paper by that title, preprint or actual neurips publication. There is https://arxiv.org/abs/2507.11768, with a different title, and contains lots of inconsistencies with regards to the model. For example, from the appendix:

> All experiments used the OpenAI API with the following configuration:

> • Model: *text-davinci-002*

> • Temperature: 0 (deterministic)

> • Max tokens: 0 (only compute next-token probabilities)

> • Logprobs: 1 (return top token log probability)

> • Rate limiting: 10 concurrent requests maximum

> • Retry logic: Exponential backoff with maximum 3 retries

That model is not remotely appropriate for these experiments and was deprecated in 2023.

I'd suggest anyone excited by this attempt to run the codebase on github and take a close look at the paper.

Don't forget that theoretical peak performance is (probably) half the performance listed on the nvidia datasheet because they used the "with sparsity" numbers! I've seen this bite folks who miss the * on the figure or aren't used to reading those spec sheets.

It's good to have this support in APIs but grammar constrained decoding has been around for quite a while, even before the contemporary LLM era (e.g. [1] is similar in spirit). Local vs global planning is a huge issue here though - if you enforce local constraints during decoding time, an LLM might be forced to make suboptimal token decisions. This could result in a "global" (i.e. all tokens) miss, where the probability of the constrained output is far lower than the probability of the optimal response (which may also conform to the grammar). Algorithms like beam search can alleviate this, but it's still difficult. This is one of the reasons that XML tags work better than JSON outputs - less constraints on "weird" tokens.

[1] https://aclanthology.org/P17-2012/

Great article, thanks for sharing. The tension between "SQL is declarative" and "Write the query like this or it will OOM" has always made me uncomfortable.

I've used closed, mature, systems with custom cost based optimization and still find that I sometimes need to override the optimizer on join orders. Interesting to see if any of the benchmarks in this paper have similar shapes.

Data is at least meant to be AGI given e.g. the measure of a man storyline, as a non-human but generally intelligent. Less clear for HAL, though the story has HAL undergoing a human-like learning process and the later storylines show that evolution.