spmurrayzzz

I've done some testing on this idea of real-time training for memory purposes using prompt tuning, prefix tuning, and LoRAs. Its hard to pull off in many cases but I've seen it work in surprising situations. I got the original idea when I was testing out the concept of using LoRA swapping for loading different personalities/output styles per batched user prompts. Those were pre-trained, but it occurred to me that soft prompts could be trained much faster to potentially remember just a handful of facts.

The basic idea is that you summarize chunks of the convo over time and create tiny ephemeral datasets that are built for retrieval tasks. You can do this by asking another model to create Q&A pairs for you about the summarized convo context. Each sample in the dataset is an instruction-tuned format with convo context plus the Q&A pair.

The training piece is straightforward but is really where the hair is. Its simple to, in the soft prompt use case, to train a small tensor on that data and just concatenate it to future inputs. But you presumably will have some loss cutoff, and in my experience you very frequently don't hit that loss cutoff in a meaningfully short period of time. Even when you do, the recall/retrieval may still not work as expected (though I was surprised how often it does).

The biggest issue is obviously recall performance given that the weights remain frozen, but also the latency introduced in the real-time training can end up being a major bottleneck, even when you lazily do this work in the background.

Running them at the edge is definitely possible on most hardware, but not ideal by any means. You'll have to set latency and throughput expectations fairly low if you don't have a GPU to utilize. This is why I'd disagree with your statement re: viability — its really going to be most viable if you centralize the inference in a distributed cloud environment off-device.

Thankfully, between llama 3 8b [1] and mistral 7b [2] you have two really capable generic instruction models you can use out of the box that could run locally for many folks. And the base models are straightforward to finetune if you need different capabilities more specific to your game use cases.

CPU/sysmem offloading is an option with gguf-based models but will hinder your latency and throughput significantly.

The quantized versions of the above models do fit easily in many consumer grade gpus (4-5GB for the weights themselves quantized at 4bpw), but it really depends on how much of your vram overhead you want to dedicate to the model weights vs actually running your game.

[1] https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

[2] https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

> I work at a startup that leaned into AI and a use case is simply detecting mis-classified products (happens all the time at large retailers).

This tracks with my own experience for the last decade or so. I stopped being surprised at how many business problems can be, in part or in whole, reduced to classification issues.

Something that I've found worth pointing out to other builders is that you don't need to build an AI product outright to sell to consumers (re: your comment about your ideas being gobbled up by big tech). Building internal AI/ML tools to solve your own business/workflow/whatever problems is often the better route for folks to leverage the advances in that technology. The best AI you build is likely the one your users don't even know exists.

This also has a beneficial side effect of showing you first-hand just how bad this tooling is currently and how far away it is from replacing builders wholesale. And to be clear, I'm not just talking about leveraging closed proprietary APIs for these efforts, I'm actually more referring to building your own training and inference stacks from scratch. The pedagogical impact alone can make up most of the ROI for your time spent.

Its not impossible, but its definitely difficult. There is some overlap in the methods used to detect benchmark data contamination, though its not entirely the same thing. For the detection use case, you already know the text you're looking for and you are just trying to demonstrate that the model has "seen" the data in its training set. The challenge is proving that it is statistically improbable that the model could stochastically generate the same tokens without having seen them during training.

Some great research exists in this area [1] and I expect much of it may be repurposed for black box attribution in the future (in addition to all the work being done in the mechanistic interpretability field)

[1] https://arxiv.org/abs/2311.04850

There is an NLP section in Jeremy Howard's "Practical Deep Learning for Coders" course (free): https://course.fast.ai/Lessons/lesson4.html

The whole course is fantastic. I recommend it frequently to folks who want to start with DL basics and ramp up quickly to more advanced material.

> Fortunately for most apps there's a middle ground between “use a spyware” and “build your own”, and that's exactly why this tool is much needed for LLM in my opinion.

Sure I understand the motivation I think, the big tradeoff is performance. If your original commentary about people privileging convenience holds true across the end-to-end user experience here, I would say that single digit tokens per second rates probably qualify as inconvenient for many folks and thus cannibalize whatever ease-of-setup value you get at the outset.

There's a reason CUDA/ROCm is needed for the acceleration, there's a ton of work put into optimization via custom kernels to get the palatable throughput/latency consumers are used to when using frontier model APIs (or GPU-accelerated local stacks).

> I'm never gonna have my non-tech friend do any of this

Who was making that assertion? I certainly wasn't.

In the same way I am never going to tell my non-engineer friends to build their own todo app instead of just using something like Todoist. But if they told me they cared about data privacy/security, I'd walk them through the steps if they cared to hear them.

> having to install Ollama + Cuda to get locally working LLM didn't felt right to me when there's all what's needed in the browser

Was there something specifically about the install that didn't feel right? I ask because ollama is just a thin go wrapper around llama.cpp (its actually starting a modified version of the llama.cpp server in the background, not even going through the go ffi, likely for perf reasons). In that that sense, you could just install the CUDA toolkit via your package manager and calling `make LLAMA_CUDA=1; ./server` from the llama.cpp repo root to get effectively the same thing in two simple steps with no extra overhead.

> I'm seeing the same thing with LLMs, all people are focused on is: Prove to me AI isn't evil - people can see a valuable use case in a demo but it doesn't matter, I think like blockchain some are beyond convincing. They just aren't into technology anymore.

You might be shadowboxing a bit with a point I didn't make (or maybe your comment was intentionally orthogonal to what I raised, not sure). I work with this technology every day in a professional, commercial context. Not just LLMs, but many other ML/DL implementations that walk the gamut of downstream tasks from anomaly detection, time series forecasting, etc. I think its useful enough to be building real things with it to improve the way my business functions. In the efforts of building those inference and training stacks from scratch, I've also seen how spectacularly they can fail and how often.

I don't think AI is evil. I think autoregressive token prediction is stochastic enough to be considered unreliable in its current state. That doesn't mean I am going to stop building things with it, it just means that I've seen these systems implode regularly enough, even with grounding via RAG, that I tend to push caution first and foremost (as I did in my original message).

> I mention this because RAG is perfect for these kinds of use cases, where you really can't afford the hallucination - where you need its information to be based on specific cases - specific information.

I think it's worth cautioning here that even with attempted grounding via RAG, this does not completely prevent the model from hallucinating. RAG can and does help improve performance somewhat there, but fundamentally the model is still autoregressively predicting tokens and sampling from a distribution. And thus, it's going to predict incorrectly some of the time even if its less likely to do so.

I think its certainly a worthwhile engineering effort to address the myriad of issues involved, and I'd never say this is an impossible task, but currently I continue to push caution when I see the happy path socialized to the degree it is.

ElevenLabs[1] is actually very good, and may have gotten better since the last time I tried it, but its neither free nor is it open source/weights.

[1] https://elevenlabs.io/

My M2 Max significantly outperforms my 3090 Ti for training a Mistral-7B LoRA. Its sort of a case-by-case situation though, as it depends on how optimized the CUDA kernels happen to be for whatever workload you're doing (i.e. for inference, theres a big delta between standard transformers vs exllamav2, apple silicon may outperform the former, but certainly not the latter).

I don't think we can call it distillation, at least not in the conventional ML use of the word as you're not interacting with any of the actual model architecture, specifically — not computing the loss between the predictions of the parent model and the distilled target model.

This is an important distinction when it comes to assess model collapse risk, which is a risk I think has probably been overstated enough to this point where now its being understated.

Breaking the Spell is a book I still recommend to friends today. Sad to hear.

> I'm not sure if it is the same team that built Descent(s) but some of them were involved in level design of the originals

Not the same exact team head-for-head but pretty close. Matt Toschlog and Mike Kula were both game directors on the original (and founded Parallax Software). They were the game directors of Overload as well and they're the founders of Revival, the studio that made the game.

For future reference, you can always check the `config.json` file in the repo for this info. Specifically `max_position_embeddings` is the key you're looking for.

7B: 32k 8x22B: 65k

There are some interesting hacks you can do like replicating the K/V weights by some factor which allows them to be evenly divisible by whatever number of gpus you have. Obviously there is a memory cost there, but it does work.

I liked Steve Yegge's quote on the Latent Space podcast late last year: "I'll believe in agents when somebody shows me one that works."

That's really all I require, just show me an agentic workflow that doesn't routinely implode at various stages and I'll buy in to some of the broader claims about the future of agents.

Yea definitely. But in that case, you could train _much_ faster in pytorch, then convert to ONNX, and load in the browser for inference (as the transformers.js docs recommend)

EDIT: (I responded before your full edit with the bullet list). This next comment is orthogonal to the slow training performance topic I think, but the use cases you reference there don't seem to be well-suited at all for an autoregressive decoder-only model architecture.

In theory its definitely possible, but I suspect that maybe performance concerns are probably the reason its not implemented (yet). They have a webgpu embeddings benchmark in an HF space to give you a sense of the forward pass dynamics: https://huggingface.co/spaces/Xenova/webgpu-embedding-benchm...

Its impressive for what it is, but training would be painful at those latencies (fp16, batch 32, sequence length 512 generates a ~500ms forward pass with a 22M param model)

Anecdotally, I'm seeing many people get hired by just building things in the wild. There's a ton of open source work going on still in the early stages, everything from inference servers to training libraries to MLOps frameworks and beyond. Being a solid contributor to any of those projects, or starting an individual project is a good start I think.

(As an aside, I also think "job in AI" is a little to abstract to comment on in many ways since the term is so overloaded in 2024)

IMO I think the 3Blue1Brown video is a good place to start to build intuitions about how things work generally if you're new, and Andrew Ng's courses will help you dig into more detail, experiment, and implement things to build on those intuitions.

It also ships with an openai-compatible server implementation as well now that you could point your UI at (if you wanted to run leaner w/out ollama).

https://github.com/ggerganov/llama.cpp/blob/master/examples/...

h2o is definitely somewhat popular specifically for LLMs, but ray is certainly widely used for distributed training workloads

> You’ve implemented streaming and batched actions yourself have you?

Streaming and batching really aren't that onerous to build yourself. Especially if your design goal isn't to support every single LLM provider and related endpoints. And it's the kind of boilerplate that you build once and usually never touch again, so the front-loaded effort amortizes well over time.

With that said, I do think some of the langchain hate is definitely overstated. There's pieces of it that can be useful in isolation, like the document loaders, if you're trying to spin up a prototype quickly to test some ideas. But the pitch they make is that its the fastest/easiest way to build LLM-based application end to end. That pitch I find to be dubious, and thats being charitable.

In the case of Phind though, they also publish their models on HF with similar bold performance claims without publishing the datasets: https://huggingface.co/Phind/Phind-CodeLlama-34B-v2

Even I am to grant that their subscription product has some secret sauce they want to keep close to the chest (ignoring for a moment their paid product is GPT-4 based), not doing the same for all the models they release to the open source community free of charge with a commercially-permissible license seems suspect.

I realize this sort of open source contribution is mostly for marketing purposes, but being critical of the performance claims I think is still valid nonetheless.

With respect to the pretraining data, its true that we're probably SOL there in terms of verification. But for fine-tuning, they could still publish the dataset and see if others can reproduce their results as well as audit for contamination.

If we're comparing benchmark deltas between different fine-tuned variants that share the same base models, that seems like the bare minimum we should expect to come along with performance claims.

AFAIK they haven't released the dataset they fine-tuned on, so we can't be 100% there wasn't benchmark contamination. Agree that we definitely need more than N=1 to challenge the performance claims, but I still think its valid to call it out given how much benchmarking-gaming we've seen in this space.

While I agree that its an overstatement to say beam search is a minor/worthless detail, its not as cut and dry as "beam search > top-p". You will get wildly variant results with both sampling strategies depending on the objective function that any given model was trained on, in addition to the type of task domain you're expecting the model to perform.

Just to cherry pick one domain: if you're doing any creative work (storytelling, roleplaying, etc), top-p may be a better sampling strategy because you will almost always get more diverse output. But since it's much less predictable, you probably don't want that sort of strategy for summarization or sentiment analysis.