Notes on OpenAI o3-mini
simonwillison.net
simonwillison.net
I ran the test on Perplexity Pro, which hosts DeepSeek R1 in the U.S. and which has just added o3-mini as well. The text was a speech I translated a month ago from Japanese to English, preceded by a long prompt specifying the speech’s purpose and audience and the sort of style I wanted. (I am a professional Japanese-English translator with nearly four decades of experience. I have been testing and using LLMs for translation since early 2023.)
An initial comparison of the output suggested that, while R1 didn’t seem bad, o3-mini produced a writing style closer to what I asked for in the prompt—smoother and more natural English.
But then I noticed that the output length was 5,855 characters for R1, 9,052 characters for o3-mini, and 11,021 characters for my own polished version. Comparing the three translations side-by-side with the original Japanese, I discovered that R1 had omitted entire paragraphs toward the end of the speech, and that o3-mini had switched to a strange abbreviated style (using slashes instead of “and” between noun phrases, for example) toward the end as well. The vanilla versions of ChatGPT, Claude, and Gemini that I ran the same prompt and text through a month ago had had none of those problems.
I’ve experienced the same thing with long-form translation. I would expect that chunking the translation (page or paragraph at a time) would fix the missing paragraph problem but potentially the loss of context would cause other problems.
This is such a common use case — would love to find resources.
I started using LLMs for translation after GPT-4 came out in March 2023—not that long ago! At first, the biggest problem was the context window: it wasn’t possible to translate more than a couple of pages at a time. Also, prompt writing was in its infancy, and a lot of techniques that have since emerged were not yet widely known. Even now, I still do a lot of trial and error with my prompts, and I cannot say with confidence that my current prompting methods are the best.
But, for what it’s worth, here are some strategies I currently use when translating with LLMs:
- In the prompt, I explain where the source text came from, how the translation will be used, and how I want it to be translated. Below is a (fictional) example, prepared through some metaprompting experiments with Claude:
https://www.gally.net/temp/20250201sampletranslationprompt.h...
- I run the prompt and source text through several LLMs and glance at the results. If they are generally in the style I want, I start compiling my own translation based on them, choosing the sentences and paragraphs I like most from each. As I go along, I also make my own adjustments to the translation as I see fit.
- After I have finished compiling my draft based on the LLM versions, I check it paragraph by paragraph against the original Japanese (since I can read Japanese) to make sure that nothing is missing or mistranslated. I also continue polishing the English.
- When I am unable to think of a good English version for a particular sentence, I give the Japanese and English versions of the paragraph it is contained in to an LLM (usually, these days, Claude) and ask for ten suggestions for translations of the problematic sentence. Usually one or two of the suggestions work fine; if not, I ask for ten more. (Using an LLM as a sentence-level thesaurus on steroids is particularly wonderful.)
- I give the full original Japanese text and my polished version to one of the LLMs and ask it to compare them sentence by sentence and suggest corrections and improvements to the translation. (I have a separate prompt for this step.) I don’t adopt most of the LLM’s suggestions, but there are usually some that I agree would make the translation better. I update the translation accordingly. I then repeat this step with the updated translation and another LLM, starting a new chat each time. Often I cycle through ChatGPT --> Claude --> Gemini several times before I stop getting suggestions that I feel are worth adopting.
- I then put my final translation through a TTS engine—usually OpenAI’s—and listen to it read aloud. I often catch minor awkwardnesses that I would overlook if reading silently.
This particular workflow works for me because I am using LLMs to translate in the same language direction I did manually for many years. If I had to translate to or from a language I don’t know, I would add extra steps to have LLMs check and double-check the accuracy of the translation and the naturalness of the output.
I was asked recently by some academics I work with about how to use LLMs to translate documents related to their research into Japanese, a language they don’t know. It’s an interesting problem, and I am planning to spend some time thinking about it soon.
Please note that my translation process above is focused on quality, not on speed. If I needed to translate a large volume of text more quickly, I would write a program to do the translation, checking, and rechecking through API calls, accepting the fact that I would not be able to check and polish the translation manually as I do now.
If anyone here would like to brainstorm together about how to use LLMs for translation, please feel free to email me. My website, with my email address on the Contact page, is linked from my HN profile page.
Would make a great article for your own site, otherwise I'm happy to link to it here instead.
I also use many LLMs to assist my translation tasks.
Recently, I have also been using Google AI Studio [1], and I find the its latest models to be smarter.
I have a large collection of Neo-Latin texts I’m trying to get translated.
My goal is to increase the accessibility of the works — not to create a perfect translation. I want to use LLMs to put text on the facing page of the source text. Errors present in the translation, I hope, can be addressed in a Wikimedia-style community editing system.
This approach could .01x lower the cost of translation—and train readers to question translations (something that is a very good thing to learn!)
Would you still expect this to produce a better result than Deepl or other purpose-built translation software?
One fundamental area of disagreement is how closely a translation should reflect the content and structure of the original text versus how smooth and natural it should sound in the target language. With languages like Japanese or Chinese translated into English, for example, the vocabulary, grammar, and rhetoric can be very different between the languages. A close literal translation will usually seem awkward or even strange in English. To make the English seem natural, often you have to depart from what the original text says.
Most translators will agree that where to aim on that spectrum should be based on the type of text and the reason for translating it, but they will still disagree about specific word choices. And there are genres for which there is no consensus at all about which approach is best. I have heard heated exchanges between literary scholars about whether or not translations of novels should reflect the original as closely as possible out of respect for the author and the author’s cultural context, even if that means the translation seems awkward and difficult to understand to a casual reader.
The ideal, of course, would be translations that are both accurate and natural, but it can be very hard to strike that balance. One way LLMs have been helping me is to suggest multiple rewordings of sentences and paragraphs. Many of their suggestions are no good, but often enough they include wordings that I recognize are better in both fidelity and naturalness compared to what I can come up with on my own.
“and” between noun phrases, for example)
it's just like.. the way it looks is so profound. maybe it's how it's formatted on my screen with the surrounding lines?
But then I noticed that the output length was 5,855 with the original Japanese, I discovered that R1 had “and” between noun phrases, for example) those problems.
idk, i'm just astonished. the amount of satisfaction that i get from reading your comment has given me enough dopamine to motivate me to create an account and write this entire comment itself, so thank you, your writing is appreciated, and i am very glad i came upon this and hope to find you again in the future somewhere. imagine???? xx
For more than a year, regular LLMs, when properly prompted, have been able to produce translations that would be indistinguishable from those of some professional translators for some types of translation.
General-purpose LLMs are best for translating straight expository prose without much technical or organization-specific vocabulary. Results are mixed for texts containing slang, dialogue, poetry, archaic language, etc.—partly because people’s tastes differ for how such texts should be translated.
Because most translators are freelancers, it’s hard to get a handle on what impact LLMs have been having on their workloads overall. I have heard reports from experienced translators who have seen work drop off precipitously and have had to change careers, while others report an increase in their workloads over the past two years.
Many translation jobs involve confidential material, and some translators may be hanging onto their jobs because their clients or employers do not allow the use of cloud-based LLMs. That safety net won’t be in place forever, though.
I suspect that those who work directly with translation clients and who are personally known and trusted by their clients will be able to keep working, using LLMs as appropriate to speed up and improve the quality of their work. That’s the position I am fortunate to be in now.
But translators who do piecework through translation agencies or online referrers like Fiverr will have a hard time competing with the much faster and cheaper—and often equally good—LLMs.
I made a few videos about LLMs and translation a couple of years ago. Parts of them are out of date, but my basic thinking hasn’t changed too much since then. If you’re interested:
“Translating with ChatGPT”
“Can GPT-4 translate literature?”
“What do translators think about GPT?”
https://www.youtube.com/watch?v=8JUepj7wIl0
I’m planning to make a few more videos on the topic soon, this time focusing on how I use LLMs in my own translation work.
Or another approach, feed it all into context but tell it to wait and not translate, and then feed it in an additionalt time part by part asking it to translate each part and with the translation style instructions repeated.
In general, though, I haven’t seen any sign yet that reasoning models are better at translation than nonreasoning ones even with shorter texts. LLMs in general have reached the level where it is difficult to assess the quality of their translations with A/B comparisons or on a single linear scale, as most of the variation is within the range where reasonable people disagree about translation quality.
Are there specific use cases where o1's higher cost is justified anymore?
gpt-3.5 -> gpt-4 (gpt-4-32k premium)
"omni" announced (multimodal fusion, initial promise of gpt-4o, but cost effectively distilled down with additional multimodal aspects)
gpt-4o-mini -> gpt-4o (multimodal, realtime)
gpt-4o + "reasoning" exposed via tools in ChatGPT (you can see it in export formats) -> "o" series
o1 -> o1 premium / o1-mini (equivalent of gpt-4 "god model" becoming basis for lots of other stuff)
o1-pro-mode, o1-premium, o1-mini, somewhere in that is the "o1-2024-12-17" model with not streaming, function calling, and structured outputs and vision
now, distilled o1-pro-mode probably is o3-mini and o3-mini-high-mode (the naming is becoming just as bad as android)
its the repeat, take model, scale it up, run evals, detect innefficiencies, retrain, scale, distill, see what's not working. when you find a good little zone in the efficiency frontier, release it with a cool name
Long tail stuff perhaps. Most stuff doesn't resemble a programming benchmark. A newer model thrives despite being small when there is a lot of training data, and with programming benchmarks, like with chess, there is a lot of training data, in part because high quality training data can be synthesized.
R1 has upped the ante so Im hoping we continue to get more updates rapidly... they are getting quite good
And it's currently free in aistudio.google.com and in the API.
And it handles a million tokens.
Original deleted post: https://old.reddit.com/r/MachineLearning/comments/1i83mhj/lm...
A copy of the content: https://old.reddit.com/r/MachineLearning/comments/1i83mhj/lm...
(evidence please, everyone)
[1] https://chatgpt.com/share/679d9579-5bb8-8008-ac4a-38cef65b45...
So finally ChatGPT catches up with Claude which has a 200,000 token input limit ever since.
Claude with its projects feature is my go to tool for working on projects that I have to work on for weeks and months. Now I see a possible alternative.
There’s no obvious answer of why one should switch to any of them!
There's a suite of code-related tasks -- covering a diversity of areas, including dev ops, media manipulation etc., derived from issues I have faced over the years -- I perform for every new release. No model has solved the set of issues solved in one go but Claude still remains the best.
An example of the sort of problems in the suite:
> I have a special problematically encoded mp4 file with a subtle issue (something I ran into a couple of years ago while fixing a bug in a computer vision pipeline). In the question prompt I also pass the output of ffprobe and ask for the ffmpeg command that'll fix it. Only Claude has figured the real underlying issue out (after 4 interactions).
Another thing it seems similar to is how Jeff Bezos registered relentless.com. There seems to be a gap between the ideal branding from the perspective of the creators and branding that makes sense to consumers.