michalsustr

I made bunch of linters/formatters for Rust specifically that try to make everything "look the same" in the monorepo. Especially as it is being generated by AI. It makes it easier to review the code.

The kind of things I added: - Never use super:: or other relative references - always absolute. Within crate, always use the "private" paths, not the public API ones! - Make sure there are no "cycles" in the code. I.e. if you follow-click on definitions, you get somewhere deeper/sibling node, you never go around in a cycle (the only cycles that are allowed are in the same module). - Made a custom formatting tool, that always orders code in the same way: imports, constants, structs/enums, impl blocks, functions, macros, tests.

Things that would be nice: - All errors should propagate to the crate top-level errors.rs via thiserror - pub methods may not be marked as dead code, but they are dead within the monorepo (this annoys me now quite a lot now)

Probably more, just quickly from the top of my head.

Awesome! Looks way better than the janky things I’ve been developing on the side. Thank you for sharing!

The tenacity aspect makes me worried about the paper clip AI misalignment scenario more than before.

What are you using as a Rust debugger?

Yes exactly! Good idea to extend it to functions as well

Each struct and its referenced fields can be thought of as a graph which can be sorted. Ideally, it is a DAG, but sometimes you can have recursive structures so it can be a cyclic graph. By DAG-ordering a I meant a topological sorting such that you do it by layers of the graph.

https://en.wikipedia.org/wiki/Topological_sorting

https://en.wikipedia.org/wiki/Directed_acyclic_graph

As someone said: Custom lints are super useful.

What we do at https://minfx.ai (a Neptune/Wandb replacement) is we use TONS of custom lints. Anytime we see some undesireable repeatable agent behavior, we add it as a prompt modification and a lint. This is relatively easy to do in Rust. The kinds of things I did are:

- Specify maximum number of lines / tabs, otherwise code must be refactored.

- Do not use unsafe or RefCells.

- Do custom formatting, where all code looks the same: order by mods, uses, constants, structs/enums, impls, etc. In particular, I added topological ordering (DAG-ordering) of structs, so when I review code, I build up understanding of what the LLM actually did, which is faster than to read the intermediate outputs.

- Make sure there are no "depedency cycles": internal code does not use public re-exports, so whenever you click on definitions, you only go DEEPER in the code base or same file, you can't loop back.

- And more :-)

Generally I find that focusing on the code structure is super helpful for dev and for the LLM as well, it can find the relevant code to modify much faster.

Thank you! :)

I feel like I'm somewhere on that Venn diagram O:-)

The specific examples in the article are about UI.

I agree that UI ecosystem is a big and slow mess, because there is actually a LOT of complexity in UIs. I would even argue that there is often more complexity to be found in UIs than in backends (unless you are working on distributed systems, or writing your own database). On backend, you usually just need paralellism (95% of jobs is just parallel for, map-reduce kind of thing).

But in UI, you need concurrency! You have tons of mutable STATE flying around that you need to synchronize - within UI, across threads or with the backend. This is /hard/ - and to come back to the point of the article - the only low-level language that I'm familiar with that can do it well and reliably is Rust.

In Rust, my absolutely favorite UI framework is egui. It is based on immediate mode rendering (maybe you're familiar with dearimgui), rather than the old, familiar-but-complex retained mode. It's really interesting stuff, recommend studying on it! Dearimgui has a wonderful quote that summarizes this well:

> "Give someone state and they'll have a bug one day, but teach them how to represent state in two separate locations that have to be kept in sync and they'll have bugs for a lifetime." -ryg

We use egui in https://minfx.ai (Neptune/Wandb alternative) and working with it is just a joy. Emilk did such a fantastic job bringing egui about! Of course it has its flaws (most painful is layouting), but other than that it's great!

Same experience here! As an analogy, consider the model knows both about arabic or roman number representations. But in alternate universe, it has been trained so much on roman numbers ("Bad Code") that it won't give you the arabic ones ("Good Code") unless you prompt it directly, even when they are clearly superior.

I also believe that overall repository code quality is important for AI agents - the more "beautiful" it is, the more the agent can mimic the "beauty".

This article resonates exactly how I think about it as well. For example, at minfx.ai (a Neptune/wandb alternative), we cache time series that can contain millions of floats for fast access. Any engineer worth their title would never make a copy of these and would pass around pointers for access. Opus, when stuck in a place where passing the pointer was a bit more difficult (due to async and Rust lifetimes), would just make the copy, rather than rearchitect or at least stop and notify user. Many such examples of ‘lazy’ and thus bad design.

What are your favorite tools for profiling/optimizing C and Rust code?

What I think the author is hoping to get is some inspectable graph of the whys that can be a basis for further automation/analysis. That’s interesting, but the line to actual code then becomes blurry. For instance, what about self-consistency across time? If this would be just text, it would come out of sync (like all doc text does). If it's code, then maybe you just had wrong abstractions the whole time?

The way we solve the why/what separation (at minfx.ai) is by having a top-level PLAN.md document for why the commit was built, as well as regenerating README.md files on the paths to every touched file in the commit. Admittedly, this still leans more into the "what" rather than "why". I will need to think about this more, hmm.

This helps us to keep it well-documented and LLM-token efficient at the same time. What also helps is Rust forces you into a reasonable code structure with its pub/private modules, so things are naturally more encapsulated, which helps the documentation as well.

My experience (with minfx.ai) has been that it is very important to build a system which imposes lots of constraints on the code. The more constrained you can make it, the better. Rust helps a lot in this. Thanks to this, for the first time in my career, I feel like the bigger the system gets, /the easier/ it is to develop, because AI can discover and reuse common components. While human would struggle searching for these and how to use them in a large codebase. Very counter-intuitive!

We are planning to make an offline local version, see also https://minfx.ai/why.html and Web + Native

If you work in machine learning, we are building a much more powerful and faster Neptune/Wandb replacement.

https://minfx.ai/

It’s not an issue. When we run LTO optimizations, strip all symbols, we get 2MB compressed for decently complex GPU-accelerated rendering (minfx.ai)

300MB is nonsense, we are at 2MB compressed with https://minfx.ai

We love wasm! You can get pretty far with it. We’re building new machine learning experiment tracker using wasm on the front end. (If you know what Wandb or Neptune is, you should give us a try!)

As far as I know, we are the fastest on the market. The multithreaded support is a pain though.

https://minfx.ai

Hi, I’m building such one: https://minfx.ai/

Still early, but iterating really fast!

Very interesting! I suggest following up with this on a rust core devs forum, as there might be higher concentration of people capable giving feedback.

It actually gets worse. The GPUs are numerically non deterministic too. So your embeddings may not be fully reproducible either.

A very short notice... This is why we're building https://minfx.ai

We are building one! Check out https://minfx.ai

You may also want to migrate to https://minfx.ai

Check out https://minfx.ai

We are very serious about performance and scale, and have extensive experience with it. Check out https://minfx.ai

Great article, thank you for submitting it! Very useful and timely. Will likely use ideas from it for mocking server/client comms for faster unit tests - avoid dealing with tcp etc.

I'm not working on game-related topics lately, I'm in the industry now (algo-trading) and also little bit out of touch.

> Has there been any meaningful progress after that?

There are attempts [0] at making the algorithms work for exponentially large beliefs (=ranges). In poker, these are constant-sized (players receive 2 cards in the beginning), which is not the case in most games. In many games you repeatedly draw cards from a deck and the number of histories/infosets grows exponentially. But nothing works well for search yet, and it is still open problem. For just policy learning without search, RNAD [2] works okayish from what I heard, but it is finicky with hyperparameters to get it to converge.

Most of the research I saw is concerned about making regret minimization more efficient, most notably Predictive Regret Matching [1]

> I was thinking about developing a 5-max poker

Oh, sounds like lot of fun!

> I don't see why a LLM can't learn to play a mixed strategy. A LLM outputs a distribution over all tokens, which is then randomly sampled from.

I tend to agree, I wrote more in another comment. It's just not something an off-the-shelf LLM would do reliably today without lots of non-trivial modifications.

[0] https://arxiv.org/abs/2106.06068

[1] https://ojs.aaai.org/index.php/AAAI/article/view/16676

[2] https://arxiv.org/abs/2206.15378

I agree it is possible to build an LLM to play poker, with appropriate tool calling, in principle.

I think it's useful to distinguish what LLMs can do in a) theory, b) non-LLM approaches we know work and c) how to do it with LLMs.

In a) theory, LLMs with the "thinking" rollouts are equivalent to (finite-tape) Turing machine, so they can do anything a computer can, so a solution exists (given large-enough neural net/rollout). To do the sampling, I agree the LLM can use an external tool call. This a good start!

For b) to achieve strong performance in poker, we know you can do continual resolving (e.g. search + gadget)

For c) "Quantization" as you suggested is an interesting approach, but it goes against the spirit of "let's have a big neural net that can do any general task". You gave an example how to quantize for a state that has 2 actions. But what about 3? 4? Or N? So in practice, to achieve such generality, you need to output in the token space.

On top of that, for poker, you'd need LLM to somehow implement continual resolving/ReBeL (for equilibrium guarantees). To do all of this, you need either i) LLM call the CPU implementation of the resolver or ii) the LLM to execute instructions like a CPU.

I do believe i) is practically doable today, to e.g. finetune an LLM to incorporate value function in its weights and call a resolver tool, but not something ChatGPT and others can do (to come to my original parent post). Also, in such finetuning process, you will likely trade-off the LLM generality for specialization.

> you can do a k-NN or some other simple approximation. [..] You can say that about any other game then, no?

Yes, you can approximate value function with any model (k-NN, neural net, etc).

> In poker if you call 25% more or 35% more if the bet size is 20% smaller is unlikely to result in a huge blunder. Chess is more volatile and thus you need more "precision" telling patterns apart.

I see. The same applies for Chess however -- you can play mixed strategies there too, with similar property - you can linearly interpolate expected value between losing (-1) and winning (1).

Overall, I think being able to incorporate a value function within an LLM is super interesting research, there are some works there, e.g. Cicero [6], and certainly more should be done, e.g. have a neural net to be both a language model and be able to do AlphaZero-style search.

[6] https://www.science.org/doi/10.1126/science.ade9097