2,072 karma · joined February 18, 2023
The videos on the game (and all his other videos) with Irving Finkel, a curator at the British museum, are spellbinding. He has the looks, manners and enthusiasm of an eccentric museum curator from central casting!
I was searching for something like this recently but ended up rolling my own using a block-based OT to local server and diff sync to remote ones.
I’m reading the system guide and nodding along. It’s super validating to see the similarities and contrasts.
Can a country trust a trade deal with the US? Whereas when has China ever not honoured a trade deal?
On your last point abour “causing instability on the internet”, for example, I could point out we are in a discussion about America wanting to restrict LLMs into haves and have nots and if that isn’t destabilising the internet by preventing non-American companies from defending themselves or paying American companies to defend themselves against them protection racket wise then what is?
Why as a European be more afraid of a country that whilst you wouldn’t want to live in and which has supported Russia is quiet and selling you cheap stuff vs a country who used to be a friend who is now wanting you to grovel for a bad deal and is also supporting Russia?
Occam’s razor IBM didn’t invest in Fortran I because the internal political environment at the corporation didn’t have the incentives aligned to do so. This is completely orthogonal to whether they could have used a better compiler or not.
Now that wasn’t the most efficient and it has a whole bunch of perverse incentives too, but it’s what happened as companies got big enough to hire people to do the work that could be delegated because those delegatees got status from having people to themselves delegate to.
However, thinking back to the spat with the DoD and more generally how the administration is much more supportive - and supported by - OpenAI and XAI and it’s easy to imagine this is just another escalation in the fight between a “liberal leaning” company and its competitors and the administration.
There might have been something said by someone at Amazon or something but I’d guess Occam’s razor the administration just leapt at the chance after their supplier sanctions fell flat?
I do my own framework and spend a lot of time trying to debug this and it’s not so much the context size in hard numbers but rather the probability that there is debris or wrong directions in the window that are drowning out the things the user thinks are important.
This manifests in the llm that keeps going back to doing the thing that failed when they tried it just before the last approach etc. The frequency of things in the context window give weight even if they are the wrong things.
I have a lot of tricks like not giving the llm lots of tools but rather giving it a tool it can use to search for tools etc.
But the bigger solution is in process where you use something like superpowers to force the llm through stages and you control the context that carries forward.
Then of course Delphi came along and made all that true for windows apps too!
So somehow I chime with how your comment starts but have such different memories of how it ends :)
I’ve used Fable a lot. It’s a marginal improvement on Opus. It’s really not scary smart. I don’t want to go back to Opus because Fable was that little bit smarter, but it’s not like anything really changes at work and when we’re bumped back. So I’m really not buying any national security angle other than in the sense the administration can weaponise that to crown the AI race winners.
The idea that springs to my mind is to do Delaunay and Voronoi using spherical geometry. I think the article uses flat Euclidean geometry but if we tweak the fifth axiom we could do spherical or hyperbolic?
Pro tip they worked well for me with response truncation: in the truncated output, say that the full text is available in /tmp/whereever.txt - that way, the llm will be able to query and read more using built in tools without reissuing the big tool call.
But it’s a great short term business opportunity for AI vendors and it was Anthropic who went all in on being knowledge worker outsourcing in a big way first whilst OpenAI thought they’d replace Google in search.
I think Anthropic had the better business strategy.