It seems someone is trying to pass some less tasteful containers in the name of more reputable ones.
2,119 karma · joined September 1, 2018
More & contact: https://omarkama.li/about
I'm always happy to chat so hit me up!
The only one in this class to be even worse in my experience is Github Copilot.
But in this case the LLM is not exposed to explicit translation pairs between these two languages and rather by seeing enough examples in similar contexts, LLMs transfer some of their learnings in Python to Ruby (for better or worse results)
This kind of training data typically looks like ChatGPT style conversations where all the prompts are all templated like “Translate the following text from X to Y: [text]” and the LLM’s expected answer is the translated text.
LLMs can generalize through transfer learning (to a certain extent) from these translation pairs to some understanding (strong) and even answering (weak) in the target language. It also means that the LLM’s actual sweet spot is in translation itself since that’s what was trained in, not just a generalization.
LLMs process tokens sequentially, first in a prefilling stage, where it reads your input, then in the generation stage where it outputs response tokens. The attention mechanism is what allows the LLM as it is ingesting or producing tokens to "notice" that a token it has seen previously (your instruction) is related with a token it is now seeing (the code).
Of course this mechanism has limits (correlated with model size), and if the LLM needs to take the whole input in consideration to answer the question the results wouldn't be too good.
My take aways:
- This is the first model from OpenAI that feels relatively agentic to me (o3-mini sucks at tool use, 4o just sucks). It seems to be able to piece together several tools to reach the desired goal and follows a roughly coherent plan.
- There is still more work to do here. Despite OpenAI's cookbook[0] and some prompt engineering on my side, GPT-4.1 stops quickly to ask questions, getting into a quite useless "convo mode". Its tool calls fails way too often as well in my opinion.
- It's also able to handle significantly less complexity than Claude, resulting in some comical failures. Where Claude would create server endpoints, frontend components and routes and connect the two, GPT-4.1 creates simplistic UI that calls a mock API despite explicit instructions. When prompted to fix it, it went haywire and couldn't handle the multiple scopes involved in that test app.
- With that said, within all these parameters, it's much less unnerving than Claude and it sticks to the request, as long as the request is not too complex.
My conclusion: I like it, and totally see where it shines, narrow targeted work, adding to Claude 3.7 - for creative work, and Gemini 2.5 Pro for deep complex tasks. GPT-4.1 does feel like a smaller model compared to these last two, but maybe I just need to use it for longer.
0: https://cookbook.openai.com/examples/gpt4-1_prompting_guide
Re: Llama 4, please see the sibling comment.
RoPE (Rotary Positional Embeddings, think modulo or periodic arithmetics) scaling is key, whereby the model is trained on 16k tokens long content, and then scaled up to 100k+ [0]. Qwen 1M (who has near perfect recall over the complete window [1]) and Llama 4 10M pushed the limits of this technique, with Qwen reliably training with a much higher RoPE base, and Llama 4 coming up with iRoPE which claims scaling to extremely long contexts up to infinity.
[0]: https://arxiv.org/html/2310.05209v2
[1]: https://qwenlm.github.io/blog/qwen2.5-turbo/#passkey-retriev...
Marketing the API as OpenAI-compatible and then me getting 400s when I switch to Gemini leaves a sour taste in the mouth, and doesn’t make me confident about their MCP support.
It’s not just for AI, we created a custom package format[2] for automations that can run in CLI as a one-off, a REST API server, an MCP server with a single command..
Source: I train LLMs and push their limits.
Anything on your machine (such as a rogue browser extension or a malicious npm/pypi package) could scan for this and just get all your cookies - and that's only the beginning of your problems.
CDP can access any origin, any data stored (localStorage, indexedDB ...), any javascript heap, cross iframe and origin boundaries, run almost undetectable code that uses your sessions without you knowing, and the list is very long. CDP was never meant to expose a real browser in an untrusted context.
Use the pre-built Trails[1][2] as MCP servers or create and publish your own with a familiar puppeteer-like API, powered by your or your friends browsers.
A couple of ideas .. I would like to benchmark a remote headless server, as well as different methods to run the LLM (vllm vs tgi vs llama.cpp ...) on my local machine, and in this case llamafile is quite limiting. Connecting over an OpenAI-like API instead would be great!
It’s super easy, just a manifest.json file to add which you can generate here [0].
Also are you considering adding more drumkits? Or maybe allow a user to set their own samples (stored locally, just for themselves) like you can change the header color in HN.
It seems we have shared maker interests, feel free to hit me up on twitter!
Having trained quite a few LLMs by now, especially around the “uplift” from a text completion model to an instruct one, I noticed that the instruction following capabilities tend not to be uniform across all the tasks the LLM is able to perform.
In other words, it doesn’t know what is the implication of your request (time on the clock) in terms of change in output (the svg drawing of the clock), and all the abstractions and steps in between, this is why changing the time in the prompt might yield little difference.
Instruct datasets are quite small in retrospect and hardly cover the range of tasks instruct LLMs are able to perform, and mostly rely on extrapolation (interpolation?) capabilities for the LLM from its small training set. And it is this generalization that is not evenly distributed.
Just my two cents from my modest observations, sprinkle [citation needed] everywhere.