6,499 karma · joined November 17, 2015
I'm also more curious about tokenizers for LLMs than I've ever been before, both for Chinese and English. I feel like to understand I'll need to look at some concrete examples, since sometimes tokenization can be per word or per character or sometimes chunks that are in between.
--
https://github.com/agentclientprotocol/claude-agent-acp/rele...
Yeah, definitely. It lacks case and verb conjugations, plus whole classes of filler words, and words themselves are on average substantially shorter. If you listen to or read a hyper-literal transliteration of Chinese speech into English (you can find fun videos of this on Chinese social media), it even resembles "caveman speech" for those reasons.
If you look at translated texts and compare the English versions to the Chinese ones, the Chinese versions are substantially shorter. Same if you compare localization strings in your favorite open-source project.
It's also part of why Chinese apps are so information-dense, and why localizing to other languages often requires reorganizing the layout itself— languages like English just aren't as information-dense, pixel for pixel.
The difference is especially profound for vernacular Chinese, which is why Chinese people often note that text which "has a machine translation flavor" is over-specified and gratuitously prolix.
Maybe some of this washes out in LLMs due to tokenization differences. But Chinese texts are typically shorter than English texts and it extends to prose as well as poetry.
But yeah this is standard stuff: Chinese is more concise and more contextual/ambiguous. More semantic work is allocated in interpretation than with English, less is allocated in the writing/speaking.
Do you speak Chinese and experience the differences between Chinese and English differently? I'm a native English speaker and only a beginner in Chinese but I've formed these views in discussion with Chinese people who know some English as well.
> Another reason that a no-exceptions policy is important: If students with disabilities are permitted to use laptops and AI, a significant percentage of other students will most likely find a way to get the same allowances, rendering the ban useless. I witnessed this time and again when I was a professor—students without disabilities finding ways to use disability accommodations for their own benefit. Professors I know who are still in the classroom have told me that this remains a serious problem.
This would be a huge problem for students with severe and uncorrectable visual impairments. People with degenerative eye diseases already have to relearn how to do every single thing in their life over and over and over. What works for them today will inevitably fail, and they have to start over.
But physical impairments like this are also difficult to fake and easy to discern accurately. It's already the case that disability services at many universities only grants you accommodations that have something to do with your actual condition.
There are also some things that are just difficult to accommodate without technology. For instance, my sister physically cannot read paper. Paper is not capable of contrast ratios that work for her. The only things she can even sometimes read are OLED screens in dark mode, with absolutely black backgrounds; she requires an extremely high contrast ratio. She doesn't know braille (which most blind people don't, these days) because she was not blind as a little girl.
Committed cheaters will be able to cheat anyway; contemporary AI is great at OCR. You'll successfully punish honest disabled people with a policy like this but you won't stop serious cheaters.
Do LLMs generally perform better in verbose languages than they do in concise ones?
It's not all bad! It's also enormously fun. I've been able to work on things I'd been putting off forever. When I can use LLM agents, I less often feel paralyzed by perfectionism, which is probably the biggest productivity boost I get. My own code has not decreased in quality, and I think that for the truly important things, neither has that of my colleagues.
But LLMs don't make junior dev mistakes. They make "my brain has worms in it" mistakes.
I imagine how they treat these things will be contextual and maybe inconsistent. There aren't really hard lines between what they probably want editors that integrate with them to do and generic tools that try to sit a layer above the vendors' agent TUIs.
The worst breakage by far is protocol breakage; basically anything that uses HTTP as a basis for building some other protocol gets broken all the time. None of the people implementing it seem aware. They buy the vendor's claim that it's "transparent", when in fact even "inspect/trace-only" modes often break all kinds of shit.
I've seen Umbrella break:
- Git
- RubyGems
- `go mod`
- OrbStack
- Matrix
- Cargo
- all JDKs
- Nix
- Pkgsrc
- all VMs
and probably some other things I'm forgetting. When this breakage is reported, the first round of replies is typically "I visited that domain in my enterprise-managed browser and it's not blocked". That is, of course, a useless and irrelevant test.Often it takes hours to even fully diagnose the breakage with enough confidence to point the finger at that tool and not some other endpoint security tool.
I'm not sure if the people buying and deploying tools in this category don't know how much stuff it breaks or just don't care. But the breakage is everywhere and nobody seems prepared for it.
Hacky menu bar modification tools are basically an accessibility requirement for me, and my vision isn't even that bad. (Best corrected is 20/30 or 20/40 or so.) People with serious impairments are totally screwed by this on macOS, sometimes even with large external monitors.
But for me at least, a tool like Git seems pretty essential for inspecting changes and deciding which to keep, which to reroll, and which to rewrite. (I'm not particularly attached to Git but an interface like Magit and a nice CLI for inspecting and manipulating history seem important to me.)
What are you imagining VCS software doing differently that might play nicer with LLM agents?
My first memorable foray into Linux packaging was creating proper Ubuntu packages for builds of WINE that carried compatibility and performance patches for running Warcraft III and World of Warcraft.
Nowadays Proton is the distribution that includes such hacks where necessary, and there are lots of good options for managing per-game WINEPREFIXes including Wine itself. A lot of the UX around it has improved, and DirectX support has gotten really, really good.
But for me at least, WINE was genuinely useful as well as technically impressive even back then.
I think how to marry the Ruby formulas and a Rust frontend is something the Homebrew devs can figure out and I'm interested to see where it goes, but I don't really care whether Ruby "goes away" from Homebrew in the end or not. It's a lovely language, so if they can keep it for their DSL but improve client performance I think that's great.
I didn't know about the pending, official Rust frontend! That's very interesting.
Fwiw w/r/t Trivy in particular,I don't think Trivy is bad software and I use it at work. We're unaffected by this breach because we use Nix to provide our code scanning tools and we write our own Actions workflows. Our Trivy version is pinned by Nix and periodically updated manually, so we've skipped these bad releases.
It's not difficult to find first-hand accounts of this; go browse social media posts by teachers in mainland China if you're curious.
There are similar problems of "teaching to the test" in other contexts, too.
I'm not categorically opposed to standardized testing and I never said I was.
There may be some good things about the Gaokao but having spoken to some (Chinese) teachers in China, it's also a limiting factor for education prior to university in a lot of ways, limiting the freedom of teachers and driving up risk aversion in parents.
(It's also effectively graded on a regional curve, which might be a good thing but isn't meritocratic in the straightforward way you suggest.)
And that's a big part of what's so frustrating about Python generally: it seems to be a language used by lots of people who've never used anything else and have an attitude like "why would I ever try anything else"?
Python has a culture where nominal values of user-friendliness, pragmatism, and simplicity often turn into plain old philistinism.