rhdunn

I was talking about running this on a server, hence my comments re 1xB200. Obviously, the more hardware/VRAM you have the better/faster you can run these large models. But if you are a small/medium sized company you could feasibly do it on just one B200. It all depends on how much hardware you can afford to run.

Yes, that's what I was saying w.r.t. expert offloading, i.e. ensuring that the GPU could fit the active parameters not all the parameters.

If it is a mixture of experts (MoE) model like the 2.x models, won't this reduce the hardware needed to run the model?

The Kimi-K2.6 model is 1.1T parameters with 32B active parameters. With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. On a single B200 you can have 5-6 experts loaded into VRAM at a time. Realistically that would be 3-4 to account for the context. [!]

[!] With this and other MoE models it looks like an interesting area for research would be to detect or predict which models would be needed ahead of time. That way you could schedule the load into VRAM step before the weights are needed. That way you shouldn't lose much/any performance from offloading the weights to RAM.

It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.

It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.

Firefox has had profiles for a long time (via about:profiles and a command-line argument). Unfortunately, the new profiles are not compatible with the old ones and cannot be migrated.

1. https://www.uea.ac.uk/about/news/article/fresh-evidence-of-c... -- (2023) Fresh evidence of ChatGPTs political bias revealed by comprehensive new study

2. https://www.ox.ac.uk/news/2026-01-20-new-study-finds-chatgpt... -- (2026) New study finds that ChatGPT amplifies global inequalities

3. https://stevepavlina.com/blog/2026/03/chatgpts-political-bia... -- (2026) ChatGPT’s Political Bias

4. https://pmc.ncbi.nlm.nih.gov/articles/PMC10623051/ -- (2023) Revisiting the political biases of ChatGPT

5. https://www.theverge.com/2024/2/21/24079371/google-ai-gemini... -- (2024) Google apologizes for ‘missing the mark’ after Gemini generated racially diverse Nazis

I think you're right with the current LLM/transformer architecture. There are several factors that affect model size:

- The number of token values supported by the model ("n_vocab").

- The number of parameters/features that are used to represent each token ("d_model").

- The number of attention layers there are ("n_layers").

such that the number of parameters is approximately:

   p ~= 12 * n_layers * d^2_model + n_vocab * d_model

Thus, the issue with the current architecture is that in order to scale the models (more token values, more attention blocks, more features, etc.) the model sizes increase exponentially. This is how you end up with billions or trillions of parameters.

It should be possible to keep the model size smaller by using better architectures, or making improvements to the existing model architecture.

For example, improving the token model by possibly using something similar to the image and audio data and getting the model to learn its own internal representation of the byte/character data instead of doing a tokenization pre-processing step. This way, instead of a separate model learning that several bytes/characters appear together, the transformer could learn things like language-specific prefices and suffices, character pairings (like in Japanese, Chinese, and Korean), and other syntactic morphology. It may also help with solving issues like "how many X characters are in the word/phrase Y". You could also experiment with using either 256 parameters (one per character in a byte) or using a single parameter per byte (that is 1/byte_value).

Does anyone know if they intend on releasing open source/weights variants for 3.8 or whether 3.6 was the last model they are/were doing that for?

A parachuting flamingo? An aardvark driving a bus? It should be easy to randomize the animal and the mode of transport (or vary it with animal playing a sport) to create images not in the training data.

Possibly. Telephone (电话) is electricity/electronic (电) + talking/speech (话).

In Japanese there's the Japanese possessive ('no') which can also be a modifier/qualifier in text like 男の子 (boy, literally "man of child") and 女の子 (girl, literally "woman of child"), so there are sequences of Chinese characters (possibly in combination with Japanese) that could be a single token like character sequences in the Latin script.

I've found https://digitalorientalist.com/2025/02/04/to-merge-or-not-to... with some information/analysis of this.

Have you tried the https://huggingface.co/LatitudeGames models? They are used by the https://play.aidungeon.com website, but can also be downloaded and used with llama-server in conjunction with something like SillyTavern.

But in general, I've experienced things similar to you. I've also found that LLMs are bad at subtext, e.g. hinting at an NPC being a werewolf or vampire.

It's not goblins, it's honest!

The other ingredients would be doing other things: making the pill/drug easer to swallow/consume, extending shelf-life, etc. You need enough of the drug for it to be effective, but not too much to overdose or exhibit side-effects.

Yes that clip [4] is making the rounds again in light of Mythos, but his stance (and that of others) hasn't changed ([3] is from a new interview).

[1] https://memeburn.com/amodei-says-open-source-ai-is-becoming-... [2 July 2026]

[2] https://fortune.com/2026/07/02/anthropic-fable-and-mythos-ar... [July 2, 2026]

[3] https://www.youtube.com/shorts/S72ZRBSNHZc [2 weeks ago] -- Dario: "Now what I do worry about with some of these laggard models is the risks of them, where we have Mythos-class cyber capabilities, 12 months from now we'll have much better cyber capabilities. But the Mythos-class cyber capabilities may just be available for anyone to download."

[4] https://digg.com/tech/zx1bqifo

LLMs grading the answers is relying on the LLM knowing the answer and not just hallucinating it. You also have issues if/when the model refuses to answer, or if it gets stuck in a loop (e.g. if running locally with a heavily quantized model).

I'm investigating/experimenting with using traditional NLP (stanza, spaCy, etc.) to try and grade the responses according to different metrics (is the response in first/second/third person?, is it written as poetry, prose, or drama? etc.). I'm also thinking about using information extraction and synonym detection to handle data queries and the like.

I'm not sure if they've fixed this, but older models have a tendency to ignore negation as `no`, `not`, etc. all occur frequently in the training data so are weighted less strongly than the verbs and nouns.

The advice I've heard is to emphasize the traits you want, not discourage the traits you don't. So rather than saying "make no mistakes" you can do something like you suggested with writing it as "check your work" or "ensure you answer correctly and concisely".

This approach is effectively seeding the context with how you want the LLM to behave/operate ("senior reviewer", i.e. the style of the responses you want) and the context/domain in which the LLM is operating in ("SWE-Bench").

This is common in system prompts and frames the responses.

For example, you'd get different responses saying:

1. you are a pirate writing sea shanties about programming;

2. you are a news reporter writing an article on physics;

3. you are a senior software engineer with complete knowledge of PostgreSQL.

For 1 you could get responses along the lines of the Wellerman sea shanty -- "There once was a program that was set to C ...".

The "make no mistakes" bit does look dubious. It would be interesting comparing the results with and without that bit and trying alternative ways of getting the same desired behavior.

You're not reading someone else's RP, though. Not with the applications that support this.

There are things like the character(s) you interact with, the initial setting, and some background information predefined, but the responses/evolution of the RP vary depending on things like the model used and the user's interactions.

It's somewhat like taking a D&D setting/scenario. Each group plays it differently.

Tell that to the LLM RP crowd.

A 27B model can fit easily on a 32GB VRAM card (e.g. 5090) or a 32GB computer in RAM at FP8/Q8 (unsloth have 28.6GB Q8 files).

For 24GB VRAM cards (e.g. 4090) you can use Q6_K (22.5GB) or Q5_K_M (19.5GB) quants, possibly offloading some of the weights to RAM.

A game company (https://www.zachtronics.com/) that have made a series of games where you either build machines or write instructions to solve a task. An example is Opus Magnum where you convert input elements to output elements.

The games track things like cycles taken to complete the task, size/area of the machine, and cost. Those scores are shown on separate leaderboards and optimizing for one can come at the cost of another (e.g. faster machines may be bigger and/or more expensive).

If you don't want the thinking, you can pass `enable_thinking: false` to the `chat_template_kwargs`. If using promptfoo, this can be done via:

    providers:
      - # llama-server
        id: openai:chat:qwen
        config:
          apiBaseUrl: http://localhost:7876
          apiKey: "..."
          passthrough:
            chat_template_kwargs:
              enable_thinking: false

The looping may be due to quantization -- I've seen it on locally quantized Q6_K Qwen 3.5/3.6 models. I recall seeing somewhere (here or r/LocalLlama) that Qwen models are sensitive to quantization of the keys, though I haven't yet experimented with/looked into fixing this. (I've been building up my promptfoo tests/infrastructure to detect looping, etc. on Qwen and other models.)

Dublin Core is effectively similar/related to schema.org's CreativeWork. If you have a creative work (audiobook, short story, news article, etc.) then Dublin Core is applicable, in addition to the corresponding CreativeWork subtype.

And yes, you should use whatever metadata is applicable to your site and test it against the search engines/etc. you want to support to make sure that they are reading the metadata correctly.

Sandi Toksvig, the current host of the BBC program QI (Quite Interesting), previously hosted by Stephen Fry. She's also been on a number of other BBC TV and radio shows.

It could be based on things like word frequency. I'd expect obfuscate/obfuscation to be less common outside of programming and RPGs (Vampire the Masquerade).

Norman French due to the Norman invasion of 1066 resulting in Old English evolving into Middle English. You can see that in the words for animals vs meats (cow and boef/beef, sheep and mutton, etc.) where the Germanic people raised the sheep and the Norman aristocracy ate them.

A lot of the more common and simpler words are Germanic, as is the grammar (e.g. compound words like cupboard).

I use promptfoo for evaluation. I'm experimenting with tests for my workflow/use cases.

I have a custom assert for loop/repeat detection that works well:

    def count_repeats(text: str, length: int) -> int:
        n = len(text)
        pattern = text[n - length : n]
        count = 1 # Include the end of the string as matching the substring.

        text = text[: -length]
        while text.endswith(pattern):
            text = text[: -length]
            count = count + 1

        return count


    def repeats(output: str, context: dict[str, any]) -> bool|float|dict[str, any]:
        threshold = context.get('config', {}).get('threshold', 3)
        count = 0
        length = 0

        for n in range(1, (len(output) // 2) + 1):
            n_count = count_repeats(output, n)
            if n_count > count:
                count = n_count
                length = n

        if count >= threshold:
            return { 'pass': True, 'score': 1.0, 'reason': f'Output repeats {count} times with length {length}.' }
        else:
            return { 'pass': False, 'score': 0.0, 'reason': f'Output doesn\'t repeat {threshold} or more times.' }


    def no_repeats(output: str, context) -> dict[str, any]:
        result = repeats(output, context)
        result['pass'] = not result['pass']
        result['score'] = 1.0 - result['score']
        return result

Just add it to your promptfooconfig.yaml:

    defaultTest:
      assert:
        - # ----- The output doesn't repeat/get stuck in a loop.
          type: python
          value: file://asserts/repeat.py:no_repeats

llama.cpp is great. However, Llama 4 was a misstep for them: it was too big, so was out of reach of the LocalLlama crowd and hard to train/customize into different variants like has happened with the smaller models on Hugging Face. 70B seems to be about the limit there, with smaller models being easier to run and customize.

It's all relative. For local use I'd classify it by hardware (VRAM size) using FP8 or Q6 quantization:

1. tiny <2-3B -- easily runnable on lower-spec hardware

2. small 4-8B -- runnable on 8GB GPUs

3. medium 9-12B -- runnable on 12GB GPUs

4. large 13-24B -- runnable on 16GB (for the lower end models) and 24GB GPUs

5. very large 25-32GB -- runnable on 32GB GPUs

6. huge >32GB -- not easily runnable on consumer GPUs without compromising performance (offloading layers to the CPU/RAM), quality (heavy quantization, esp. at <= Q4), or price (investing in multi-GPU setups and/or server-grade hardware).

You could possibly split huge down further, as 70GB models (e.g. llama 3) are easier to get working than >120GB models and 1TB models are completely intractable.

Yeah. I run LLM models locally and for me 22B-32B is the largest I'm willing to invest in trying out.

Even though Mistral 4 has 6B active parameters per token (allowing 3-3.5 per token parameters to be loaded on a 4090), the ~240GB download + storage is pushing the limits of being able to try this out locally, especially if you are downloading and evaluating multiple models.

It also makes it harder for other people to make downstream finetunes like with what happened with the older Mistral/Magistral models.