psadri

Even though I started using LLMs as soon as they became available, I was until recently very cautious around using them autonomously. I was of the opinion that I needed to provide heavy supervision around coding tasks. Granted, that was probably justified based on the model capabilities at the time.

However, lately, I have had several experiences that have completely changed my mind - the here is a high level goal (involving gathering production logs, setting up an eval, running tests, iterating, including diagnosis and coming up with new ideas, etc. and don’t bother me until it’s done. The experience has been incredible.

This has been possible because the agent has a way to iterate and hillclimb against a metric it can measure. Relatively easy in the world of software.

I’m now convinced that we will unlock the same gains once make other domains similarly “iterable”.

Eg Agent comes up with 100 new proteins, runs in lab autonomously, gathers results, iterates. One month (or whatever) later, you have your new protein designed.

Write tests first. Have agent iterate until they are satisfied.

The point is that it boils down to writing the tests correctly, regardless of who is implementing the actual code. Hand-written code without test coverage has the same problems as AI generated code.

Now that the models are smarter / know more than most of us, a new bottleneck is finding out what you don't know. If you don't know about logistic classifiers and how they could be applied to your problem, you will not ask for it.

These days, I tend to start my coding sessions by the high level problem I'm trying to solve vs the prescriptive, specific solution I may have in mind. It often surfaces ideas and approaches that I did not know about.

The point was that you can go further than "write me a contact that says x,y and z". I actually believe the process I went through with the AI was a more custom, tailored experience than any interaction I have had with actual lawyers, who'd usually apply a template. The AI asked me a bunch of questions, many of which were relevant, and led to follow up questions. I could ask my own questions, make corrections, etc. and all along, they made sense to me. This part of the experience was great.

There could still be more unknowns that I did not encounter and by definition don't know about.

And the final output could be wrong, wording, exact legal terminology etc.

Can we say working with an actual lawyer is guaranteed to be better, more correct? Lawyers are people and they also make mistakes, wake up on the wrong side of the bed, hate their job etc, just like the rest of us.

No admittedly not. I can try it and report back.

I was in a similar situation and tried a different approach.

I had start with asking for a contact with some details I provided. But then I ask the model to be an experienced corporate lawyer and ask me a series of questions to gather the details it needed and then write the contract. The result covered a lot of details that were highly relevant but were absent from the original attempt.

The key insight is that you can lean on the model to cover your unknown unknowns.

We have been working in this space for the past year. Based on our experience, I no longer trust any claims unless they are backed by benchmark results (yes, benchmarks are painful to run reliably and expensive).

It is possible to reduce token usage. It’s just much harder than the basic approach.

Thanks to coding agents, there is no reason not to

For anyone interested in private space flight: Wild Wild Space documentary. Also a commentary about startups in general.

Separately, I've been wondering how uneven human progress is. I think we have come a long way in electronics (relays to vacuum tubes to transistor to … to an M5 chip) but in some other domains, like moving objects through space, we are basically mostly reliant on thrust generated by burning fuel (EV's are a genuine advance, but mostly due to battery tech).

I have been wondering: more solar means we are making Earth less reflective. Solar energy that might have been reflected back to space is now absorbed by the PV to make electricity. But that delta has to go somewhere? Where does it go in terms of Earth’s temperature. Same with more nuclear… we’d be pumping more energy into the system.

Don’t take this as an argument against these technologies. It’s a huge win to not be increasing the CO2 concentration in the atmosphere by burning fossil fuels.

Looking forward to the series finale this Friday!

Luna is at a very compelling point on the price/performance curve.

I have found that sometimes a smaller model with max reasoning is actually more expensive than using the next tier model with a lower reasoning effort. It’s certainly faster.

In my experience, the thing that makes your better at something is actually _doing_ it, badly at first and then gradually improving by iteration.

But for writing, I usually see the advice that says you should read a lot instead of write a lot. What gives?

I looked at the OTel schema generated for recording a single numerical metric. It was like 12 or 13 meta fields in addition to the actual metric fields like timestamp, metric and value.

OTel looks like something designed by a committee of committees, funded by someone who is in the business of selling cloud storage/data warehousing services.

Who are OpenRouter’s competitors?

This is an awesome thread. I agree - the catalyst is joy, in a broad sense of the word. One is lucky to find joy in something that can take you deep.

To be honest I also just read about it. However everytime I pass through Heathrow I think someone really took the idea to heart.

Aside for the algorithms and how well they work, there is a psychological aspect to how people perceive "wait" times. It turns out just waiting is very annoying. But having people do anything during that same wait period → perceived as progress → not upset.

An example iirc was waiting for luggage at the airport. It takes min of x minutes for the first bags to show. In one version, passengers went directly from the gate to the luggage carousel and waited x mins for the first bag → unhappy. Then the airport decided to add a long, round-about "walk" path from the gate to the luggage area that burned some of those minutes → same total "wait" but happier passengers.

A version of this idea was mentioned in one of the Three Body Problem books. There, the bombs were pre-positioned along a path and detonated sequentially like dominos, with a vehicle riding the blast waves.

Sure. But I think they can do it more exhaustively than humanly possible.

I don’t think we have spent enough time on the creativity axis.

When we solve problems we usually follow a heuristically guided energy efficient path. We just prune a lot of possibilities based on our existing knowledge and experience.

Creativity happens when we consciously (or not) go off the beaten path and explore. Most of those explorations are dead ends. But some will yield unexpected connections, patterns etc that we call “creativity” .

An AI system could also go on those kinds of explorations. Today they aren’t it because we are not asking them to.

Be sure to visit the basement museum to see the upside-down string + tiny sandbag weights model that was used to simulate the structure. Pure genius.

Learning to code = understanding a problem, breaking it down into small, manageable pieces, putting all the pieces back together. Debugging. Iterating towards better metrics, etc.

All these are amazingly valuable skills/mindsets that can be highly portable to other "problem solving" domains.

This will cause an extra round trip to the LLM. Which means more $ spent.

I met Om once in SF over a meal. I could tell he was a kind soul. RIP.

The comment about memcached being ephemeral is orthogonal to whether people will use it as if it persistent. If the cache appears to get hit 99.9% of the time and is always there, sooner or later people will write code that relies on that behavior.

Maybe the client libraries can help by returning nulls 10% of time, in dev mode?

We are witnessing the birth of the term "pre-times" you often see in dystopian sci-fi.

It’s interesting that you have to be an A to tell the rest apart.

And good luck if you have a lot Bs that believe they are As.

Claude: configure my Emacs such that … has worked pretty well for me lately.

This is part of the plot in Murakamui's Hard Boiled Wonderland and the End of the World.