yberreby

> Because chatgpt-4o-latest already has all of those improvements

Does it, though? They said that "many" have already been incorporated. I simply don't buy their vague statements there. These are different models. They may share some training/post-training recipe improvements, but they are still different.

Good point, but how does that relate to, or explain, the decision not to release 4.1 in ChatGPT? If they have a nice post-training pipeline to make 4o "nicer" to talk to, why not use it to fine-tune the base 4.1 into e.g. chatgpt-4.1-latest?

I agree on both naming on stability. However, this wasn't my point.

They still have a mess of models in ChatGPT for now, and it doesn't look like this is going to get better immediately (even though for GPT-5, they ostensibly want to unify them). You have to choose among all of them anyway.

I'd like to be able to choose 4.1.

> Note that GPT‑4.1 will only be available via the API. In ChatGPT, many of the improvements in instruction following, coding, and intelligence have been gradually incorporated into the latest version (opens in a new window) of GPT‑4o, and we will continue to incorporate more with future releases.

The lack of availability in ChatGPT is disappointing, and they're playing on ambiguity here. They are framing this as if it were unnecessary to release 4.1 on ChatGPT, since 4o is apparently great, while simultaneously showing how much better 4.1 is relative to GPT-4o.

One wager is that the inference cost is significantly higher for 4.1 than for 4o, and that they expect most ChatGPT users not to notice a marginal difference in output quality. API users, however, will notice. Alternatively, 4o might have been aggressively tuned to be conversational while 4.1 is more "neutral"? I wonder.

Anecdotally, I used diffrax (and equinox) throughout last year after jumping between a few differential equation solvers in Python, for a project based on Dynamic Field Theory [1]. I only scratched the surface, but so far, it's been a pleasure to use, and it's quite fast. It also introduced me to equinox [2], by the same author, which I'm using to get the JAX-friendly equivalent of dataclasses.

`vmap`-able differential equation solving is really cool.

[1]: https://dynamicfieldtheory.org/ [2]: https://github.com/patrick-kidger/equinox

> the latest LLM models have almost as many parameters as we have neurons

I often see this take, but the apt comparison is between parameter and synapse count, not neuron count. You should be counting hidden units rather than weights if you want to compare to neuron counts.

> Actually your comment gives me hope that we will never have AI singularity, since how the brain works is flawed, and were trying to copy it.

As someone working in the field, the vast majority of AI research isn't concerned with copying the brain, simply with building solutions that work better than what came before. Biomimetism is actually quite limited in practice.

The idea of observing the world in motion in order to internalize some of its properties is a very general one. There are countless ways to concretize it; child development is but one of them.

> If you're tasked with it then it doesn't matter if you have a month or a decade (assume that time is.frozen while you do your research), its a too complex problem and simply cannot be solved, and the same goes for an AI.

I highly disagree.

Let's assume a superintelligent AI can break down a problem into subproblems recursively, find patterns and loopholes in absurd amounts of data, run simulations of the potential consequences of its actions while estimating the likelihood of various scenarios, and do so much faster than humans ever could.

To take your example of winning a war, the task is clearly not unsolvable. In some capacity, military commanders are tasked with it on a regular basis (with varying degrees of success).

With the capabilities described above, why couldn't the AI find and exploit weaknesses in the enemy's key infrastructure (digital and real-world) and people? Why couldn't it strategically sow dissent, confuse, corrupt, and efficiently acquire intelligence to update its model of the situation minute-by-minute?

I don't think it's reasonable to think of a would-be superintelligence as an oracle that gives you perfect solutions. It will still be bound by the constraints of reality, but it might be able to work within them with incredible efficiency.

> With 5 finetunes you need to host 5 copies or load and unload them.

If you use LoRA, which many do when fine-tuning nowadays, you don't need five full copies. You only need to store adapters, which can be in the tens of MBs range for a given finetune.

Indeed. But a point that is often omitted from comparisons with organic brains is how much "compute equivalent" we spent through evolution. The brain is not a blank slate; it has clear prior structure that is genetically encoded. You can see this as a form of pretraining through a RL process wherein reward ~= surviving and procreating. If you see things this way, data-efficiency comparisons are more appropriate in the context of learning a new task or piece of information, and foundation models tend to do this quite well.

Additionally, most of the energy cost comes from pretraining, but once we have the resulting weights, downstream fine-tuning or inference are comparatively quite cheap. So even if the energy cost is high, it may be worth it if we get powerful generalist models that we can specialize in many different ways.

> This is why we invented math and physics studies, to be able to accurately calculate, predict and reproduce those events.

We won't do away without those, but an intuitive understanding of the world can go a long way towards knowing when and how to use precise quantitative methods.

I'm not denying that there are obvious limitations. However, attributing them to being "pixel-based" seems misguided. First off, the model acts in latent space, not directly on pixels. Secondly, there is no fundamental limitation here. The model has already acquired limited-yet-impressive ability to understand movement, texture, social behavior, etc., just from watching videos.

I learned to understand reality by interpreting photons and various sensory inputs. Does that make my model of reality fundamentally flawed? In the sense that I only have a partial intuitive understanding of it, yes. But I don't need to know Maxwell's equations to get a sense of what happens when I open the blinds or turn on my phone.

I think many of the limitations we are seeing here - poor glass physics, flawed object permanence - will be overcome given enough training data and compute.

We will most likely need to incorporate exploration, but we can get really far with astute observation.

> It just computes next frame based on current one and what it learned before, it's a plausible continuation.

...

> Now it's just a "what is the next frame of this 2D image" model plus some textual context.

This is incorrect. Sora is not an autoregressive model like GPT, but a diffusion transformer. From the technical report[1], it is clear that it predicts the entire sequence of spatiotemporal patches at once.

[1]: https://openai.com/research/video-generation-models-as-world...

Using LaTeX does not necessarily imply rendering to PDF.

https://ar5iv.labs.arxiv.org/

I do not mean to discount your experience; I'm simply disproving the assertion that there isn't anywhere normal priced. Montreal is a major city with a lot of tech companies and research activity, so I think it is worth mentioning as more than a simple counterexample.

Hi. I live alone in downtown Montreal. I pay 999 CAD a month for a 1-bedroom apartment, sublet. This includes furniture, electricity, heat, Internet.

I have friends who are renting a shared flat, works out to ~650 CAD / person, in a nice area (Plateau Mont Royal).

I'm not claiming two data points are representative of the whole distribution, and things are not great at all in terms of inflation / the housing market is tough, but there's no need to resort to gross exaggerations.

I did not know about BGE, learned about it from your comment. Seems to be the new SoTA for semantic embeddings, even for small models. Very cool!

As someone who wrote a fair amount of Rust and OCaml code, I have to agree with the author.

While working at Routine (YC W21), I was tasked with porting our core library to iOS to minimize duplication of business logic. This was a lucky opportunity to write something resembling a compiler: it took in schemas described with our in-house data exchange library and generated C (for FFI) and Swift code (for the end users, i.e., iOS developers).

Since Routine uses OCaml for everything (which was a big motivator for joining the company—I wanted to see how that would work out), I wrote it in OCaml. The end result is a 3-5k LOC project. It's by no means a full compiler, but it was lots of fun to write. The language got in the way incredibly rarely. On average, it made my life a lot easier. We did encounter our fair share of issues, mostly due to the cross-compilation tooling[1], third-party libraries, and intricacies of FFI. Those do take their toll on sanity.

I tried my hand at writing small compilers / interpreters in Rust, and the experience was nowhere near as smooth. It was fun, and the runtime performance is definitely there, but the ergonomics aren't the same. I especially miss first-class modules whenever I code in something other than OCaml now.

  [1]: we initially used esy [2], flirted with Nix, and eventually switched to opam-cross-ios [3].
  [1]: https://github.com/esy/esy/
  [2]: https://github.com/ocaml-cross/opam-cross-ios

Love the concept. I hope there will eventually be an Emacs version of the extension - I couldn't imagine going back to VS Code.

GP didn't say to go for mainstream GC languages, though.

OCaml, F#, or Haskell would offer many of the benefits that come with Rust's type system.

I've used OCaml professionally, and it's been a real pleasure.

Cool project. I tried it last time this got posted, but it was still a bit buggy. Giving it another shot - I'm mainly interested in the local chat.

Could you elaborate on the incremental search feature? How did you implement it? Don't you need to re-encode the full query through a SBERT or such as each token is written (perhaps with debouncing)?

Also, having an easily-extended data connector interface would be awesome, to connect to custom data sources.

Indeed, but I'm not claiming it's always faster to check than to produce. Simply that when it is the case, using GPT-4 can be worth it. It is to me - I use it daily.

> The problem is distinguishing between these parts requires me to be be an expert in the area I’m inquiring about - and then why the heck do I need to ask some idiot bot for answers to questions that I already know an answer to?

Because it can be significantly faster to check something for correctness than to produce it?

More so when the correctness check can itself be automated to some extent.

I didn't pay much attention to recent evolutions of the language, so most of what follows is what motivated me to steer clear of Go in the past.

I think the prevalent attitude in the Go community towards complexity is harmful. Trying to get rid of over-abstraction is a laudable goal, but this can backfire when you're dealing with irreducible complexity.

For many years, the standard response when needing genericity was either `go generate`, `interface{}` or "copy-paste and adapt." In my opinion, generating code should be a last resort, because it is too powerful: generated code can do pretty much anything, whereas abstractions encoded in the type system have the benefit of being predictable. `interface{}` means giving up on type safety, and copying and pasting, while it can be justified in simple cases, does not scale. Go's lack of generics became something of a meme or a point of pride, depending on who you ask, and it has gotten more than enough discussion, so I'll leave it at that. N.B.: I'm aware that Go finally ended up implementing generics.

Additionally, after using sum types in other languages (Rust/OCaml/Haskell/Swift), I really miss them whenever I'm using a language that doesn't support them. Not being able to simply encode mutually-exclusive representations using the type system is a shame.

Add to this a general disdain of the community for anything that is perceived as "too abstract" - a sort of reverse-elitism, where reaching for advanced constructs means you're not "in." It's not a bad attitude to fight back against over-engineering, but I think it sometimes goes too far. This is subjective, of course.

What initially got me interested in Go was the promise of a simple, fast, modern language. I think it fulfills the first two criteria rather well, but falls short on the last one. And I don't mean "modernity" in the sense of "whatever's shiny nowadays" - I mean it in the sense of learning from common sources of bugs and giving the programmer tools to avoid them.

Remove too much complexity from the language, and you might just shift it to the programmer.

Good idea. Just did so.

Author here. You're correct. I started this when I was getting into formal languages / compiler design, but I couldn't dedicate much time to it after writing the lexer and parser. Later on, I didn't have much interest in finishing it, in large part due to a growing dislike for Go's design philosophy. It was a nice learning experience, though.

How does that test intuition?

I'm going to bet it's because this was written by Marc Andreessen.

I assume it means "neurotypical" here.

I reckon it is against DigitalOcean's terms of service to seed copyrighted content. They forward DMCA complaints and will threaten to deactivate your account if you disregard warnings.

Several comments here seem to imply that the answer to the title is 'yes'. Reading the article appears to imply either the opposite or no effect.

This is specific to asm!. It is a macro, and you can use macros to implement custom syntactic constructs.