ImageXav

This is fantastic. As the little joke I hope it is. Everyone gets their own small disfunctional group, and gets to figure out the challenges of management. You, the manager, are Michael. You know you have to produce something, and you do, but you have no real idea of how. Your diligent agents are Dwight. Overly literal sycophants that are ready to leap to action at your slightest command without any question.

I do think a lot of folk would benefit from the introspection this offers. We've all been given the opportunity to become middle (and middling) managers, and a lot of the challenges we face are those of people who direct. Setting direction is tough. But LLMs are awesome tools.

Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1].

[0] https://playground.roboflow.com/evals

[1] https://playground.roboflow.com/evals/object-detection

Me too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.

This feels extremely close in nature to being a generalisation of discrete distribution networks.

Paper: https://arxiv.org/abs/2401.00036 Project page: https://discrete-distribution-networks.github.io/

Given that these were published at ICML 2025, at which the author was a top reviewer as per their own website https://alexiglad.github.io/, I wonder how influenced they were to pursue this avenue of the back of it. They do have a related works section in appendix E, but it somehow seems to miss this. Which is odd, as the paper made a splash at the time at the conference and even made it close to the top of hacker news due to the novelty.

Of course, DDNs are a fundamentally new architecture, whereas this is more a generaliseable training strategy, but there is a very similar core.

I've been avidly using Fable since it was re-released and while it has been excellent at building the apps I want, the reasoning has been completely opaque.

Kim, however, has exposed the whole reasoning trace, or enough of it to matter. I'd almost forgotten how nice it is to see this. I've been able to see all of the weird twist and turns it takes and it is joyful. But also, far, far more informative and means I can debug ideas far more thoroughly. Also, at a first glance it seems to have gotten quite far on a niche hobby horse of mine that no LLM has been able to crack. I'll be testing this more for sure.

Ok, this is really cool. The fact that the robot can use pointing to decide where to go is a great design decision, and robotics really is the next frontier. Definitely cheering on Mistral here!

I think it depends on your workflow. I've had a great experience with the trial. I work in research, and have set up something similar to Kaparthy's auto research. I, with Fable, have managed to get an image generation model down from 80M parameters to 10M and keep the quality of the generated images on par (similar FID). And, importantly, every change was modular, explained by Fable, reviewed by myself, and understood and documented. If not understood, I read relevant docs until I did it didn't accept it as part of the plan. So it ended up being a simple composition of existing ideas which I had previously encountered, but stacked much more rapidly than I could have.

The structure of the code is easily readable as I enforce concenventions followed by good libraries. And I can easily plug in new datasets. It's pretty good frankly.

I read the Economist for over a decade growing up. It was a great way to learn about the world, who was in power where, and the challenges facing economies at the time. I found their exposition to be pretty good given the fact they were restricted to a few pages for important events. However, their proposed solutions were always the same. More market freedom, etc.

I did feel with the change in editorial direction a while back that they lost some of their edge. I've since mostly just stuck to the Financial Times. It feels less worldly, but the content of the articles feels better.

Agreed. Which is also odd, if you think about it. Surely with the amount of compute Anthropic and others have available, they could test each of the solutions in the SO data they surely have and rank them based on efficiency/elegance/other criteria and remove poor solutions from their training data.

It may feel that way due to the iterative nature of medical improvements, but over the past few decades there has been a consistent reduction in cancer mortality rates across most types of cancer [0]. Treatments really are getting better and more targeted. Immunotherapy has made huge breakthroughs. Combination treatments allow for significantly improved lifespans and better quality of life during treatments. There are a few cancers that remain hard to treat, but I have a lot of confidence that in the coming decades we will make strides in attacking them. That being said, I'm very sorry to hear about the pain you and your family must be going through. I've had a few close loved ones undergo cancer treatment and it was tough.

[0] https://acsjournals.onlinelibrary.wiley.com/doi/10.3322/caac...

This seems like such an easy way to create perverse incentives and profit off people who are already down on their luck. Imagine being told that the only way to get considered is to pay a fee. Then later on you get told to pay the gold fee for priority. Oh you're still not getting hired? Go for our platinum package that will definitely make the difference! Not enough money? No worries, we'll take 30% of your salary for the first few years. Or maybe we'll just give you some a fixed debt at a high interest rate. Aren't you glad you used us?

Agreed. One at a time testing (OAT) has been outdated for almost a century at this point. Factorial and fractional factorial experiments have been around for that long and give detailed insights into the effect of not just single changes but the interaction between changes, which means you can superpower your learnings as many variables in DL do in fact interact.

Or, more modern Bayesian methods if you're more interested in getting the best results for a given hyperparameter sweep.

However, that is not to detract from the excellent effort made here and the great science being investigated. Write ups like this offer so much gold to the community.

I guess that's the crux of it. From an individual perspective it makes sense to stay in a stable environment, especially if a family is involved. However, I think from a societal perspective it is desirable to have people who gamble on creating new products which can raise the bar in their given industries.

Also, just because the start up fails doesn't mean it was a waste of time. If you manage to provide employment for even just 3 or 4 people for a few years, help them and yourself develop, that is a valuable success.

I would add an aspect that is not covered here but is often ignored: the strong labour protection laws result in a mentality where if you get a good job you are much less likely to want to take risks e.g. start your own business. There was a post on the HENRY (high earner, not rich yet) UK subreddit the other day from someone who had a wealth of experience and had the opportunity to join a start up as a CTO. It honestly sounded like a great chance to initiate change. All of the comments were telling the poster that they had it good, that 99% of start ups fail, that the hours would be gruelling. I feel as though the conversation would have been quite different in a US subreddit.

A term they like to use is 'crabs in a bucket'.

This is an interesting point. I've been trying to think about something similar recently but don't have much of an idea how to proceed. I'm gathering periodic time series data and am wondering how to factor in the frequency of my sampling for the statistical tests. I'm not sure how to assess the difference between 50Hz and 100Hz on the outcome, given that my periods are significantly longer. Would you have an idea of how to proceed? The person I'm working with currently just bins everything in hour long buckets and uses the mean for comparison between time series but this seems flawed to me.

I've had the complete opposite experience, and feel the complete opposite way. What is there to learn from failing a leetcode? It feels like luck of the draw - I didn't study that specific problem type and so failed. Also, there is an up front cost of several months to cover and study a wide array of leetcode problems.

With a take home I can demonstrate how I would perform at work. I can sit on it, think things over in my head, come up with an attack plan and execute it. I can demonstrate how I think about problems and my own value more clearly. Using a take home as a test is indicative to me that a company cares a bit more about its hiring pipeline and is being careful not to put candidates under arbitrary pressures.

Yes, especially as models are known to have a preference towards outputs of models in the same family. I suspect this leaderboard would change dramatically with different models as the judge.

Thanks for sharing that. Interesting that the leaderboard is dominated by Anthropic, Google and DeepSeek. Openai doesn't even register.

How did you achieve that? I was looking into it and $0.006/min is quoted everywhere.

I feel as though it also represents the fact that contributors are less invested in the project. There was a small study done a few years back hypothesizing that the number of swear words related somewhat to code quality [0] due to emotional involvement of the codebase authors. I can imagine this to be somewhat true. I would love to see this study redone now that LLMs are widespread on pre chatgpt repos (as I suspect that repos created using LLMs are going to be very sanitised).

[0] https://cme.h-its.org/exelixis/pubs/JanThesis.pdf

Even better, python has named tuples [0]. So if you have a tuple that you are sure will always have the same inputs you can declare it:

``` Point = namedtuple('Point', 'x y') pt1 = Point(1.0, 5.0) ```

And then call the X or Y coordinates either by index: pt1[0], pt1[1], or coordinate name: pt1.x, pt1.y.

This can be a really handy way to help people understand your code as what you are calling becomes a lot more explicit.

[0] https://stackoverflow.com/questions/2970608/what-are-named-t...

Not necessarily. Interpretability of a system used to make decisions is more important in some contexts than others. For example, a black box AI used to make judiciary decisions would completely remove transparency from a system that requires careful oversight. It seems to me that the intent of the legislation is to avoid such cases from popping up, so that people can contest decisions made that would have a material impact on them, and that organisations can provide traceable reasoning.

Something that stuck out to me in the updated blog [0] is that Demon Adam performed much better than even AdamW, with very interesting learning curves. I'm wondering now why it didn't become the standard. Anyone here have insights into this?

[0] https://johnchenresearch.github.io/demon/

I've found that YOLOv7 [0] tends to perform better across the board than anything ultralytics has produced, without the horrendous licensing.

[0] https://github.com/WongKinYiu/yolov7

I think it is you who have misunderstood the Nyquist-Shannon theorem. Aliasing and noise are real concerns. Tim Wescott explains it very well [0] (Figures 3, 10 and 11). If your signal is below one half the sample rate but the noise isn't, you'll lose information about the signal. If your signal phase is shifted wrt. the sampling, you'll lose information. If your sampling period isn't representative, you'll lose information. These are not implementation details.

[0] https://www.wescottdesign.com/articles/Sampling/sampling.pdf

Actually, I was mislead by the video example. They do actually keep the background information they use a temporal encoding so that the information is propagated through. Very interesting and well thought out

As far as I can can tell though the core idea is the same, to focus on the differences, the implementation is different. Differential transformers 'calculates attention scores as the difference between two separate softmax attention maps'. So they must process the redundant areas. This removes them altogether, which would significantly reduce compute. Very neat idea.

However, I do think that background information can sometimes be important. I reckon a mild improvement on this model would be to leave the background in the first frame, and perhaps every x frames, so that the model gets better context cues. This would also more accurately replicate video compression.

I thought the same, but the description of the cat picture is pretty spot on. I wonder if this is a dataset issue. Cat pictures are far more prevalent than abstract art on the internet so might well be overrepresented. Can Vision LLMs deal with a long tail of underrepresented objects when small? Or can they only do so at scale?

Here's another one that I feel is often overlooked by traditional A/B testers: if you have multiple changes, don't simply test them independently. Learn about fractional factorial experiments and interactions, and design your experiment accordingly. You'll get a much more relevant result.

My impression is that companies like to add/test a lot of features separately - and individually these features are good, but together they form complex clutter and end up being a net negative.

I don't really see why a leap to indentured servitude is necessary here. There are obviously many other ways of implementing a training relationship - see the other comments re. how it is done in Germany.

The point I'm making is that society as a whole is worse off if companies are too risk averse to hire and train juniors up to a certain level of quality. This results in too few people capable of doing a specific, presumably valuable, job well. The consequences of this are a society that underperforms it's true potential in the long run. In monetary terms, as you raise the question, that means that less taxable value is generated. So society as a whole pays the consequence.