spmurrayzzz

I just view it as a relative minor convenience, but it's not some game-changer IMO.

The tool use / function calling thing far predates Anthropic releasing the MCP specification and it really wasn't that onerous to do before either. You could provide a json schema spec and tell the model to generate compliant json to pass to the API in question. MCP doesn't inherently solve any of the problems that come up in that sort of workflow, but it does provide an idiomatic approach for it (so there's a non-zero value there, but not much).

Ah, got it. Yea, then I'd focus on learning how RoPE works first. That will at least help you understand how the retrieval in current long context implementations is so limited.

A colleague from a discord I spend time in threw together this video a year or so ago, might be helpful as a first watch before a deep dive: https://www.youtube.com/watch?v=IZYx2YFzVNc

Covers positional encoding as a general concept first, then goes into rotary embeddings.

Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that?

Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post.

RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864

Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209

The Impact of Positional Encoding on Length Generalization in Transformers - https://arxiv.org/abs/2305.19466

Scalable-Softmax Is Superior for Attention - https://arxiv.org/abs/2501.19399

The source file I linked in my initial comment is honestly the most succinct way to understand how this works, but the TL;DR is that there is a NoPE layer interval parameter passed to the transformer block implementation. That defines how frequent a "no positional encoding" layer is used. The NoPE layers use the global attention mask, which is a traditional application of attention (attends to all tokens in the context window). The other layers use RoPE (rotary positional encodings) and a chunked local attention mask, which only attends to a fixed set of tokens in each chunk.

There is a wealth of literature to catch up on to understand the performance motivations behind those choices, but you can think of it as essentially a balancing act. They want to extend the context length, which is limited by conventional attention compute scaling. RoPE on the other hand is a trick that helps you to scale attention to longer context, but at the cost of poor retrieval across the entire context window. This approach is a hybrid of those two things. The recent Cohere models employ a similar methodology.

I'd pump the brakes a bit on the 10M context expectations. Its just another linear attention mechanism with rope scaling [1]. They're doing something similar to what cohere did recently, using a global attention mask and a local chunked attention mask.

Notably the max sequence length in training was 256k, but the native short context is still just 8k. I'd expect the retrieval performance to be all over the place here. Looking forward to seeing some topic modeling benchmarks run against it (ill be doing so with some of my local/private datasets).

[1] https://github.com/meta-llama/llama-models/blob/eececc27d275...

EDIT: should be fair/complete and note they do claim perfect NIAH text retrieval performance across all 10M tokens for the Scout model on their blog post: https://ai.meta.com/blog/llama-4-multimodal-intelligence/. There are some serious limitations and caveats to that particular flavor of test though.

> This sounds like it is a third party (maybe just a small group of devs?) doing all the work

Just as a quantitative side note here — tinygrad has almost 400 contributors, pytorch has almost 4,000. This might seem small, but both projects have a larger people footprint than most tech companies' headcount that are operating at significant scale.

On top of that, consider that pytorch is a project with its origins at Meta, and Meta has internal teams that spend 100% of their time supporting the project. Coupled with the fact that Meta just purchased nearly 200k units worth of AMD inference gear (MI300X), there is a massive groundswell of tech effort being pushed in AMD's direction.

> wouldn't AMD just invest in doing this themselves, or at least provide much more support to these guys?

That was actually the point of George Hotz's "cultural test" (as he put it). He wanted to see if they were willing to part with some expensive gear in the spirit of enabling him to help them with more velocity. And they came through, so I think that's a win no matter which lens you analyze this through.

Since resources are finite, especially in terms of human capital, there's only so much to go around. AMD naturally can now focus more on the software closer to the metal as a result, namely the driver. They still have significant stability issues in that layer they need to overcome, so letting the greater ML community help them shore up the deltas in other areas is great.

> But that does not seem to be the strategy, which implies it is not so simple?

That is exactly what has been happening [1], and not just in pytorch. Geohot has been very dedicated in working with AMD to upgrade their station in this space [2]. If you hang out in the tinygrad discord, you can see this happening in real time.

> those I have talked to say they depend on a lot more than just one or two key libraries.

Theres a ton of libraries out there yes, but if we're talking about python and the libraries in question are talking to GPUs its going to be exceedingly rare that theyre not using one of these under the hood: pytorch, tensorflow, jax, keras, et al.

There are of course exceptions to this, particularly if you're not using python for your ML work (which is actually common for many companies running inference at scale and want better runtime performance, training is a different story). But ultimately the core ecosystem does work just fine with AMD GPUs, provided you're not doing any exotic custom kernel work.

(EDIT: just realized my initial comment unintentionally borrowed the "moat" commentary from geohot's blog. A happy accident in this case, but still very much rings true for my day to day ML dev experience)

[1] https://github.com/pytorch/pytorch/pulls?q=is%3Aopen+is%3Apr...

[2] https://geohot.github.io//blog/jekyll/update/2025/03/08/AMD-...

CUDA isn't the moat people think it is. NVIDIA absolutely has the best dev ergonomics for machine learning, there's no question about that. Their driver is also far more stable than AMD's. But AMD is also improving, they've made some significant strides over the last 12-18 months.

But I think more importantly, what is often missed in this analysis is that most programmers doing ML work aren't writing their own custom kernels. They're just using pytorch (or maybe something even more abstracted/multi-backend like keras 3.x) and let the library deal with implementation details related to their GPU.

That doesn't mean there aren't footguns in that particular land of abstraction, but the delta between the two providers is not nearly as stark as its often portrayed. At least not for the average programmer working with ML tooling.

(EDIT: also worth noting that the work being done in the MLIR project has a role to play in closing the gap as well for similar reasons)

I agree with you that the market exists and, as a result, solutions to this problem also exist in abundance. The most difficult part about a building a product like the one presented here is making something super generic that works for a wide swath of use cases. If you simplify the stack to more bespoke/custom approach, the build burden decreases exponentially.

For the folks who are already technical in this vertical, especially ones that leverage a low cardinality architecture (one or two models, small subset of tasks, etc), this type of thing is quite easy to build yourself first as a working prototype and then only slightly more difficult to productionize & automate.

I have some in-house infra that does similar work: monitors inputs and outputs from models, puts them in a UI for a human to score/rank, preps a DPO dataset for training, kicks off training run. The total amount of calendar time I spent from prototype to production was roughly two person weeks. Changing the human intervention mechanism to an automated reward function would be an hour or two worth of work. If I had to make this work for all types of users, tasks, and models — no shot I'd have the time personally to pull that off with any reasonable velocity.

With that said, having a nice UI with great observability into the whole process is a pretty big value-add to get out of the box as well.

(EDIT: for clarity, not affiliated all with the OP project/org)

GoBack was great. I gotta believe there's a way to manually instrument this using `tmutil` to create incremental APFS snapshots and some middleware code that knows when to wait for for the FS to be idle, but that's handwaving a ton of details.

Transformers are usually part of JEPA architectures. In I-JEPA's case, there is a ViT that is used in the context encoding phase.

> i.e there isn't a "I don't have enough information" option.

This is true in terms of default mode for LLMs, but there's a fair amount of research dedicated to the idea of training models to signal when they need grounding.

SelfRAG is an interesting, early example of this [1]. The basic idea is that the model is trained to first decide whether retrieval/grounding is necessary and then, if so, after retrieval it outputs certain "reflection" tokens to decide whether a passage is relevant to answer a user query, whether the passage is supported (or requires further grounding), and whether the passage is useful. A score is calculated from the reflection tokens.

The model then critiques itself further by generating a tree of candidate responses, and scoring them using a weighted sum of the score and the log probabilities of the generated candidate tokens.

We can probably quibble about the loaded terms used here like "self-reflection", but the idea that models can be trained to know when they don't have enough information isn't pure fantasy today.

[1] https://arxiv.org/abs/2310.11511

EDIT: I should also note that I generally do side with Lecun's stance on this, but not due to the "not enough information" canard. I think models learning from abstraction (i.e. JEPA, energy-based models) rather than memorization is the better path forward.

Definitely has been true for my work. LLMs have absolutely have been useful, I even forked an IDE (Zed) to add my own custom copilot to leverage a deeper integration for my work.

But even if we consider AI beyond just NLP, there's been so much ML you can apply to other more banal day to day tasks. In my org's case, one of the big ones was anomaly detection and fault localization in aggregate network telemetry data. Worked far better than conventional statistical modeling.

I usually assume there is a caricature of "AI Tools" that all of the detractors are working backwards from that are often nothing more than a canard for the folks that are actually using AI tooling successfully in their work.

At present it's not possible just via configuration, but you can configure a custom endpoint for both the assistant and inline assistant in settings.json.

To get custom tab completions working you need to mimic one of the completion provider apis (like copilot) [1] and direct the requests to your own endpoint. In my case, I am running llama.cpp and and MITM-proxying to its newer /infill endpoint [2]

That's why I mention the rust piece may be a blocker for some, you do have to hack apart the src a bit to get things working. At the time I started this, the only two completion providers available were supermaven [3] and copilot. You could mimic either.

[1] https://github.com/zed-industries/zed/tree/be830742439f531e8...

[2] https://github.com/ggerganov/llama.cpp/blob/master/examples/...

[3] https://github.com/zed-industries/zed/tree/be830742439f531e8...

For clarity, this does work with the mac os version of Zed. I use it frequently to work on my GPU nodes. The one tradeoff that is a bit of a smell for me is that the preview version of the feature requires you talk to a centralized broker on Zed's servers rather than fully p2p between your local IDE and your own server.

This is supposedly temporary though IIRC (may even be changed already in a dev branch, not sure).

Absolutely. This is already what I've been doing myself. Forked zed so I could use my local rig (4x 3090) to do FIM completions using a finetuned qwen coder 32B.

Only barrier for some folks will be if they're not familiar with rust or local LLMs in general, but it really wasn't that difficult looking back on it. Amounted to about an afternoon's worth of work.

Historically the problem with using LLMs for the super simple conventional NLP stuff is that they were hard to control in terms of output. If you wanted a one-word answer for a classification task, you'd often have to deal with it responding in a paragraph. This obviously hurts precision and accuracy quite a bit. There were tricks you could use (like using few-shot examples or GBNF grammars or training low-rank adapters or even re-asking the model) to constrain output a bit, but they weren't perfect.

Over the last 12-18 months though, the instruction-following capabilities of the models have improved substantially. This new mistral model in particular is fantastic at doing what you ask.

My approach to this personally and professionally is to just benchmark. If I have a classification task, I use a tiny model first, eval both, and see how much improvement I'd get using an LLM. Generally speaking though, the vram costs are so high for the latter that its often not worth it. It really is a case-by-case decision though. Sometimes you want one generic model to do a bunch of tasks rather than train/finetune a dozen small models that you manage in production instead.

Oh I totally agree that I think its not great for the game in terms of balance. As a dev, I do enjoy being able to make my own custom addons, especially as a healer. But I think a push towards less addons in the game will help its casual accessibility big time.

This is an issue even in Mythic+ where missing a single kick means you likely wipe your group and if its a pug they probably all leave the key afterwards (the latter is a separate community issue, but exists nonetheless). If Blizz in the future is forced to assume that most players dont have kick addons that tell them which party member has a kick up and a timer/alert to tell you when to kick the caster mob, then theyre also forced into improving the UI to help the players know that information without external assistance. Or even better improve the mechanics writ large so that arcane knowledge is less zero-sum when you dont have it.

I'd push back slightly on the idea that top players use minimal addons. During every RWF push, almost every top team has a dedicated resource solely for the purpose of making custom WeakAuras for different parts of each boss fight. During the last cycle, I think I saw Max call for no less than two or three dozen breaks just so folks could make sure their WAs were in sync and on the right version.

Granted I'm cherry-picking RWF where people are severely under-geared at first, but I think it does offer a relevant microcosmic example of the state of addons in retail WoW. They're more or less required for the competitive side of most "hard" content these days, even for the most coordinated of groups.

(EDIT: I wanted to add that in no way do I consider this "playing the game for you". It still took the aforementioned teams like 2 weeks to clear Nerub-ar Palace, and the largest chunk of that time wasn't even on the last boss, it was on Kyveza)

I was playing back then in 2006, you absolutely could. I don't remember any 10-man groups at that time, but people were absolutely doing this with mage-heavy comps with 20 people. And then after Stormcraft Sandbox when all the private servers began to explode in popularity, you had the ability to run back every encounter as much as you wished to speed up the learning curve, test/learn new strats, etc.

Totally agree with all of these points. I will roll a new toon on an era HC server every now and again when I'm bored with current games, just because it does feel like a new experience for me every time. Trying to figure out the safest routes, best quests to grab, non-death skips to get to easier leveling experiences (my favorite is one that gets you to darkshore by way of menethil harbor), etc.

On the other hand, MC is the most "solved" out of all the raids at this point though in terms of mechanics. The most experienced folks can low-man it with 10-20 people using mediocre gear (as long as they have some fire res). And its something that casuals can pug these days as well, I've hopped into a pug myself on an era realm with mostly blues as a resto shaman and was fine.

I tend to agree though that for "content" reasons this sounds impossible, someone will wipe the raid just for the laughs.

> People were in disbelief any of this was even possible.

Maybe my memory is that bad now in my old(er) age. But at that point in 2015, React had been a thing for about 2 years already. Ember had components even earlier than that in 2011/2012. Using components generally was a pretty well-accepted pattern from my recollection. The wars between the frameworks were all centered around render performance rather than APIs (i.e. which virtual DOM was best? do you even need a virtual DOM? etc.)

Not taking anything away from knockout per se, more just gut checking my own understanding/remembrance of that dark age.

I haven't seen this sort of work called AI Developer yet, but I may have missed the trend shift. Isn't the convention for this still to use the title of Machine Learning Engineer (MLE)?

They're still very useful on their own. But even more broadly, you can often use them in tandem with LLMs. A good example could be a classifier that's used as a "router" of sorts; could be for selecting a prompt template, directing to a specific model, or loading a LoRA or soft prompt vector to be used at inference-time.

I've had similar experiences, but I've noticed my Node.js applications which have few-to-no dependencies behave in the same way as my Go apps in that regard. I might get some deprecation logs from Node letting me know about future changes, but generally they do just work. The apps with a heavy dependency graph are a different story however.

This is still a feather in Go's cap given the fact that the standard library offers so much out of the box. I don't find myself reaching for dependencies that often.

IANAL but my guess that the output seems likely to be considered a "derivative work" (referring to US law in this case). So you'd need prior authorization from the copyright holder in the absence of a license.

"Only the owner of copyright in a work has the right to prepare, or to authorize someone else to create, an adaptation of that work. The owner of a copyright is generally the author or someone who has obtained the exclusive rights from the author... The unauthorized adaptation of a work may constitute copyright infringement." [1]

Some of the examples they give in the referenced document above are pretty close to what this tool outputs. e.g. "A new version of an existing computer program", "A revision of a website", "A drawing based on a photograph", etc

[1] https://www.copyright.gov/circs/circ14.pdf

Yes that is correct, but my position (which perhaps has been poorly-articulated) is that in the non-trivial instances, it is a distinction without difference in the greater context of the RAG stack and related pipelines.

Just allowing for a chunking function to be defined which is called at insertion time doesn't really alleviate the major pain points inherent to the process. Its a minor convenience, but in fact, as pointed out elsewhere in this thread by others, its a convenience you can afford to yourself in a handful of lines of code that you only ever have to write once.

This is how most modern vector dbs work, you usually can store much more than just the raw embeddings (full text, metadata fields, secondary/named vectors, geospatial data, relational fields, etc).

I didn't miss that detail, I just don't think chunking alone is where the complexity lies and that the pgai feature set isn't really differentiated at all from other offerings in that context. My commentary about full documents was responding directly to your comment here in this thread more so than I was the article (you claimed chroma can only insert chunks, which isn't accurate, and I expanded from there).