niobe

Reminds me of how if you make a complaint against a lawyer or a judge it's evaluated by lawyers and judges.

If it was framed as "show us how you would break down this problem and think about it" then sure. If it's the gotcha quiz (much more common in my experience) then no.

But if that's what they were going for it should be something on a completely different and more abstract topic like "develop a method for emptying your swimming pool without electricity in under four hours"

I would hope anyone with the knowledge and interest to run OpenClaw would already be mostly aware of the risks and potential solutions canvassed in this article, but I'd probably be shocked and disappointed.

Having said that, there is nothing there that isn't public information. I guess the CIA's name added some weight but this could easily be published by any public institution interested in foreign affairs.

Guily by (lack of) association!

If I was asked that question in an interview to be a programmer I'd walk out. How many abstraction layers either side of your knowledge domain do you need to be an expert in? Further, being a good technologist of any kind is not about having arcane details at the tip of your frontal lobe, and a company worth working for would know that.

Is there a good technical breakdown of all these benchmarks that get used to market the latest greatest LLMs somewhere? Preferably impartial.

Hello, nice write up. I'm curious about your deal with the factory and downstream suppliers.. did all these iterations and fixes cost more every time? Or it was a fixed contract? How does that all work

isn't that what Simon Willison's `llm` does?

edit: [link](https://github.com/simonw/llm)

Good news everyone!

True but every science headline is misleading, loaded or exaggerated. My pet peeve "X found where it should not exist". What they mean is "scientists are pleased because there's some new evidence that is not explained by their current models and that means they get to improve their models which is the goal of science anyway, so pretty much just another day for science but glad to keep you updated"

This article taught me nothing except first-world tech consumerism is out of control. Why is that interesting? Because I think it's part of why the tech bros consistently over-estimate the relvance, pace and world-changing potential of technology like LLMs. If you're drowning in tech consumerism, you see it all around and think it's _really_ important. But this is living in a silicon bubble. 90% of the planet just doesn't care, is not even aware of these "toys" and wouldn't see them as relevant to a normal life if they did, and is extremely innovative about getting their actual needs met with far more basic tools and materials available. And they would be largely right. Technology does change the world eventually, but the latest pair of smart goggles or a slightly better camera on your personal surveillance device does not.

Also super unlikely, and only worth discussing except as a theoretical exercise. We just aren't undoing centuries of law. Same reason trusts are here to stay.

On the other hand, SV development culture is often obsessed with early release and adding features, traits I would identify with short-termism, rather than improving usability and minimising bloat (trends not rules). So a little old school conservatism might go a long way when it comes to software.. although the problems of too much silence in the face of seniority are also well known from the airline industry.

I say, what a splendid tale.

paywalled

could simply be 41 instances of the same server in 41 regions, not necessarily a cause for concern. Starlink is a global service after all. I'd be more concerned if 41 instances were sharing one key.

calling it "last" is defeating their own premise - that tests need to keep pace developments in ability

Wow, old google seemed to care about the quality of their data and the service they were providing to users, and then apply reasoning to achieve those aims.

Go outside, everyone's real .. at least for the time being!

Excel is in our DNA and will never die

I am a technical musician for 40 years and I couldn't understand the points he was trying to make... poorly explained

that's one good take away I guess

I knew you guys were all psychos/s

True, but of the same importance, the speaker cannot be measured independently of the room in a home setup. You need an anechoic chamber, or at least a large and very quiet space. Even a decent size living room will alter the measurements considerably and of course materials matter too.

I have spent a lot of time measuring rooms and you can't underestimate how they can mess with frequencies, but it's also true that good speakers sound pretty good in almost any room, and bad ones will sound bad.

But to some approximation we do know how an LLM plays chess.. based on all the games, sites, blogs, analysis in its training data. But it has a limited ability to tell a good move from a bad move since the training data has both, and some of it lacks context on move quality.

Here's an experiment: give an LLM a balanced middle game board position and ask it "play a new move that a creative grandmaster has discovered, never before played in chess and explain the tactics and strategy behind it". Repeat many times. Now analyse each move in an engine and look at the distribution of moves and responses. Hypothesis: It is going to come up with a bunch of moves all over the ratings map with some sound and some fallacious arguments.

I really don't think there's anything too mysterious going on here. It just synthesizes existing knowledge and gives answers that includes bit hits, big misses and everything in between. Creators chip away at the edges to change that distribution but the fundamental workings don't change.

I don't understand why educated people expect that an LLM would be able to play chess at a decent level.

It has no idea about the quality of it's data. "Act like x" prompts are no substitute for actual reasoning and deterministic computation which clearly chess requires.

> There is SO MUCH we haven't figured out about how to use them yet.

I mean, it's pretty clear to me they're a potentially great human-machine interface, but trying to make LLMs - in their current fundamental form - a reliable computational tool.. well, at best it's an expensive hack, but it's just not the right tool for the job.

I expect the next leap forward will require some orthogonal discovery and lead to a different kind of tool. But perhaps we'll continue to use LLMs as we knownthem now for what they're good at - language.

The algorithm makes several false assumptions AND misses some crucial points about harmony that affect chord naming. One of each below:

- Chords are definitely not always built on thirds. Mainly the traditional western sounds are.

- Even more importantly, chords are never named in isolation, they are named with respect to their musical context. An Am7 could more correctly be called a C/A depending on the overall harmonic picture - the melody, the chords before and after and even the emphasis created by the performing musicians. Voicings are basically ignored but could be said to come under this point.

When I teach harmony to students I always point out the key thing to understand is how the harmony is changing, and to look at it dynamically not statically. It's more about the relative motion of notes comprising chords than names although names are obviously a useful construct.

We'll all have uploaded to the Metaverse by then, running purely on beams of light and upvotes.