yunyu

There's nothing stopping you from identifying the code, running it, and passing the output back into the context window.

BERT can process 512 tokens. LLAMA and FLAN-UL2 can process 2048 tokens. GPT-4 can process 32768 tokens, and is much better at ignoring irrelevant context.

These general models can be fine tuned with domain specific data with a very small number of samples, and have surprisingly good transfer performance (beating classical models). New research like LORA/PEFT are making things like continuous finetuning possible. Statistical models also do a much better job at translating sentences to formal structure than the old ways ever did – so I wouldn't necessarily view those fields are disconnected.

I agree with the general sentiment, there are still major issues with the newer generation of models and things aren't fully cracked yet. But the scaling laws are saying there's still a lot of upside, even without new paradigms or architectural improvements.

LLaMa isn't instruction tuned

I assume you haven't tried Alpaca (which hasn't been released), only Llama. See the instruction fine tuning section in the article.

FLAN-UL2

Hellaswag is also a deeply flawed benchmark, I wouldn't read too much into it: https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this...

A simple bag-of-words classifier requires some feature engineering, parameter tuning, continuous retraining for generalization, and actual thinking on the engineering end. Simply tossing a small amount of data at DeBERTa v3 (for classification/regression tasks) will give the same or better results 99% of the time, with a slight increase in inference cost and significantly less human effort.

I assume most people needing fractional scaling on Mac are doing it with hidpi displays, where differences in text rendering aren’t visible and layout correctness is way more important. I have a 4k display scaled to 5k resolution to get more real estate, and there’s no difference in text clarity at that PPI.

Model parallelism is possible for inference and you can host extremely large models across multiple accelerators (see DeepSpeed ZeRO), but the inference speed is several times slower compared to just having all of it on one accelerator due to communication bottlenecks and overhead (parallelism refers to memory needed for parameters, not the actual computation in most cases).

Certain models may require storage formats like bfloat16 to run efficiently that may not be supported on older hardware.

Parallelism is also supported (and more necessary actually) for training, but since backprop is expensive it typically requires many multiples of memory requirements needed for inference.

FLAN-T5 11b is the most competitive public model that has undergone instruction finetuning. It has better few shot and instruction following performance than much bigger models like GPT-NeoX: https://huggingface.co/google/flan-t5-xxl, and you can deploy it on 24GB VRAM with bfloat16 inference

The claim that current LLMs are not that good at deep thinking might be true in general, but it is definitely not the best or correct explanation for the rot13 task being described in the article (or other simple character manipulation tasks that can’t be represented at the subword token level, including base64 or arithmetic)

This is obviously the case if you apply much narrower criteria, most benchmarks of existing large datasets aren’t for character level tasks. That said, the synthetic noise section should be extremely interesting if not fully representative of your criteria.

Agree that tokenization isn’t a weakness for most general applications, disagree that it isn’t a weakness for the specific string manipulation task that the blog post is referencing

The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like punctuation and spacing correction: https://arxiv.org/pdf/2105.13626.pdf

The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is not possible for LLMs without hacks like inserting spaces, which then suffers from poor performance due to this being rare in pretraining.

The reason byte level models aren’t as popular is because inference speed is significantly slower for a given completion length, and training for the same context window in characters is significantly more expensive

I say this as someone who’s deployed ByT5 vs normal T5 in prod, having found this to be true in real world use cases. No reasonable amount of fine tuning can save GPT from performing poorly on a space insertion task (“WHOLEFDSMKT” -> “Whole Foods Market”), while ByT5 just works.

There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process input and outputs on the level of subword units that compose multiple characters to support long context windows (i.e. "door" instead of "d", "o", "o", "r"). As a result, they do poorly on character manipulation tasks. You can get some insight here: https://beta.openai.com/tokenizer

This is a solved problem with models trained on byte-level objectives without tokenization like ByT5 (if you tried this task on one of those, it would probably work perfectly with a few samples). In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.

Most LLMs are crippled at tasks regarding character level manipulation because of how the inputs are tokenized (words and numbers are split into common substrings rather than the original characters, 101 -> “10” + “1” vs “1” + “0” + “1”).

If you tried this binary conversion task with a model that takes in raw bytes (for instance, ByT5) it would perform almost perfectly. In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.

Venmo? Cash app? Zelle? I use one of these every day

The edge cases are more common at elite institutions, which are the typical admissions mania targets. At the top ~10 university I attended, there were plenty of people (myself included) falling in the poor HS GPA bucket (mostly due to extracurriculars) with perfect or near perfect ACT/SAT scores, among other outliers.

Just fine tune the model? Prompt leakage and bypass is only a problem with zero/few shot inference, and is trivially solved by removing the need for a prompt.

Zero Asians have ever benefited from any affirmative action policies in US college admissions, regardless of their socioeconomic status

I have used very little algebra/calculus/pre-college content, but have used a lot of linear algebra/discrete mathematics. Would this be a common exception to the rule?

You can update the "graph edges" with content longer than the window by fine tuning: https://beta.openai.com/docs/guides/fine-tuning

How many people would buy a Tesla that costs 3x as much?

PyTorch

A quarter million dollars is what a typical new grad offer looks like nowadays

RCS doesn’t need a separate app either. You can have SMS and RCS conversations in the same app regardless of if you’re using Google Messages/Samsung Messages/etc

If you look closely at the profile photo that the post is "Posted by", the distorted background makes it clear that it's a deepfake

That Ryzen chip has 20% slower single thread performance, which is what really matters for desktop workloads

Sublime doesn't use native platform UI toolkits. They ship their own version of Skia (same as what's used in Chromium) and have their own UI toolkit for the editor rendering.

Typescript will cover this!

Facebook acquired osmeta in 2013, which built a UIKit implementation that allowed for cross-compilation of iOS apps