trjordan

It only spawns a single sub-agent (called Task iirc), which can do everything Claude Code can, except call Task().

This is different from a lot of the context-preserving sub-agents, which have fully different toolsets and prompts. It's much more general.

This is all true, and we've prototyped a number of these things at my current startup. You need to be pretty considered about implementing them.

For a counter-example, consider Claude Code:

- 1 long context window, with (at most) 1 sub-agent

- same tools available at all times, and to sub-agent (except: spawning a sub-sub-agent)

- Full context stays in conversation, until you hit the context window limit. Compaction is automatic but extremely expensive. Quality absolutely takes a dive until everything is re-established.

- Deterministic lookup of content. Claude reads files with tools, not includes random chunks from RAG cosine similarity.

I could go on. In my experience, if you're going to use these techniques 1) maybe don't and 2) turn up the determinism to 11. Get really specific about _how_ you're going to use, and why, in a specific case.

For example, we're working on code migrations [0]. We have a tool that reads changelogs, migration guides, and OSS source. Those can be verbose, so they blow the context window on even 200k models. But we're not just randomly deleting things out of the "plan my migration" context, we're exposing a tool that deliberately lets the model pull out the breaking changes. This is "Context Summarization," but before using it, we had to figure out that _those_ bits were breaking the context, _then_ summarizing them. All our attempts at generically pre-summarizing content just resulted in poor performance because we were hiding information from the agent.

[0] https://tern.sh

One of the funny things about LLMs and modern AI is that "the ability to recognize a cat" isn't a trained behavior anymore, as described here. It's an emergent property of training it to predict a lot of things, and cats happens to be present enough in the data such that they're one of the things you can ask a larger model and have it work.

My favorite work on digging into the models to explain this is Golden Gate Claude [0]. Basically, the folks at Anthropic went digging into the many-level, many-parameter model and found the neurons associated with the Golden Gate Bridge. Dialing it up to 11 made Claude bring up the bridge in response to literally everything.

I'm super curious to see how much of this "intuitive" model of neural networks can be backed out effectively, and what that does to how we use it.

[0] https://www.anthropic.com/news/golden-gate-claude

We've been building out our agent [0], and we've found this to be the case.

We actually dialed it back a bunch, because it feels _terrible_. Yes, you get more correct answers, but it's more akin to giving the agent anxiety. Especially with agents that have access to tools, they'll burn enormous amounts of time on tool calls, trying to get enough information to overcome a motivation that's essentially burned into its identity.

(We saw one conversation where it just browsed social media instead of looking at the code for like 5 minutes, which ... you know, I get it.)

It's been much more effective to make uncertainty or further exploration be part of the agents success criteria.

- BAD: "Critique your own thoughts" -> leads to the agent trying really hard to get it right, but still not willing to actually be wrong

- GOOD: "Expose where your thoughts are unsupported or could benefit from further information" -> leads to the agent producing high-quality results, with loose ends that the user can choose to incorporate, ignore, or correct.

That prompt, combined with dialing up the thinking (either with API or prompt tuning) works much better, because it's sidestepping the training and tuning that's implicitly encouraged it to sound correct at all times.

[0] https://tern.sh, code migration AI

No fun to have to do a panic migration!

Anyway my latest startup is trying to make AI do migrations, in no small part because of this problem 15 years ago.

Man, my first startup in 2010 used protovis, the charting library Mike Bostock built before deciding d3 was the better approach. It was rough to have an 8 month old startup with a core piece of tech that suddenly stopped improving.

My main takeaway from so much of this is that "just a chart" is one of the biggest sources of hidden complexity in displaying useful information to people. It's right up there with "a simple web form" and "a web page with some simple interactivity."

Everybody has a wildly different idea of what good looks like. Defaults will never be right. Personal and global taste changes annually. We clown react (rightly) for constantly reinventing the same 4 wheels, but customers gleefully use new stuff all the time.

It's kind of amazing that d3 has been so durable in the frontend world. It really is a wrapper over a pretty solid approach. And yeah, that approach is complex, but that's the reality of visualization. It's hard to imagine another one that's that good.

A few of these points are right, but a lot of it is so far lost from the reality of LLMs that it isn't even wrong.

Yes: structure beats vibes. Primacy/recency bias is real. Treating prompts as engineering artifacts is empirically helpful.

But:

- “Reads all tokens at once.” Not quite. Decoder LLMs do a parallel prefill over the prompt, then sequential token-by-token decode with causal masks. That nuance is why primacy/recency and KV-cache behavior matter, and why instruction position can swing results.

- Embeddings & “labels.” Embeddings are learned via self-supervised next-token prediction, not from a labeled thesaurus. “Feline ≈ cat” emerges statistically, not by annotation.

- "Structure >> content". Content is what actually matters. “Well-scaffolded wrong spec” will give you confidently wrong output.

- Personas are mostly style. Yes, users like words typed in their style better, but it'll actually hide certain information that a "senior software engineer" might not know.

I don't really get the Big-O analogy thing, either. Models are constantly exposing and shifting how they direct attention, which is exactly the opposite of the durably true nature of algorithmic complexity. Memorizing how current models like their system prompts written is hardly the same thing.

Of course there's no 0-risk version of any of this. But in my experience, bugs tend to get introduced with features, then slowly ironed out over patches and minor versions.

I want no security bugs, but as a heuristic, I'd strongly prefer the latest patch version of all libraries, even without perfect guarantees. Code rots, and most versioning schemes are designed with that in mind.

There is absolutely a good reason for version ranges: security updates.

When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).

But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.

I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.

I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.

I think the most interesting change Claude enables is letting AI try stuff. We do this all the time.

I have this sense this works best in small teams right now, because Claude wants to produce code changes and PRs. Puzzmo, where OP works, is <5 engineers.

In larger codebases, PRs don't feel like the right medium in every case for provocative AI explorations. If you're going to kick something off before a meeting and see what it might look like to solve it, it might be better to get back a plan, or a pile of regexps, or a list of teams that will care.

Having an AI produce a detailed plan for larger efforts, based on an idea, seems amazing.

I love the innovation. The basic idea here appears to be:

- Nobody wants this to be closed source. The code is freely available, and you may do with it as you want. The marginal cost to distribute the code is 0, after all.

- The maintainers, as people, don't want to do charity work for companies. Their time is limited, and if they're going to support revenue-generating activities, they want a cut of the revenue.

So even if this doesn't get perfectly enforced (and it won't), that's fine! The maintainers are now free to respond to complaints with "you need to pay us for us to care." Companies that pay get some level of support; hobbyists get the same experience. Only the companies that ignore this warning will see the consequences, and it's particularly effective for reports where the author leans on "but there are a huge number of important users [to me] that are affected." Pay up if it matters!

It strikes me as a pretty clean solution to a pretty common strain of open-source headache, _especially_ as AI-generated code/reports/etc. are on the rise.

The core argument here is: LLM docbots are wrong sometimes. Docs are not. That's not acceptable.

But that's not true! Docs are sometimes wrong, and even more so if you could errors of omission. From a users perspective, dense / poorly structured docs are wrong, because they lead users to think the docs don't have the answer. If they're confusing enough, they may even mislead users.

There's always an error rate. DocBots are almost certainly wrong more frequently, but they're also almost certainly much much faster than reading the docs. Given that the standard recommendation is to test your code before jamming it in production, that seems like a reasonable tradeoff.

YMMV!

(One level down: the feedback loop for getting docbots corrected is _far_ worse. You can complain to support that the docs are wrong, and most orgs will at least try to fix it. We, as an industry, are not fully confident in how to fix a wrong LLM response reliably in the same way.)

Seems like it could easily be training data set size as well.

I'd love to see some quantification of errors in q/kdb+ (or hebrew) vs. languages of similar size that are left-to-right.

ChatGPT has 400m weekly users. X / Twitter has 600m.

It's pretty popular!

If you like this kind of degeneracy, I tried AoC in Google Sheets this year.

I only made it to Day 6, and not even both stars every day. I'm pretty confident my Day 7 solution is correct, but I hit per-cell character limits on longer inputs.

Enjoy :) oh but don't open it on mobile, some sheets crash the app

https://docs.google.com/spreadsheets/d/10FY-89y19tnRM_EAAnCd...

If you don't want to scale the whole thing (why?), you can also deploy the monolith twice and route different API calls to different clusters. Slice and scale as much as you're willing, paying the price of deployment complexity and bin-packing the usage.

The servers are already bought; the electricity cost is negligible for actual accidents.

As mentioned in the post: the hosting providers don't actually pay marginal costs for transient mistakes, so neither should honest customers.

I don't think you're an outlier, but on the other side, "I watch Netflix on my laptop / iPad" is a _huge_ market. Yes, there's plenty of social watching experiences to be figured out, but solo watching is an entire industry unto itself.

I'm a bit like you. I don't watch TV by myself, and the idea of plugging into a VR device to do so seems weird.

But I carry my AirPods everywhere and mostly listen to music solo. It's not unreasonable to think that the same happens with TV/movies.

Friends, please, click on the link. There's a long and interesting investigation that answers the question, which is presumably why this was posted.

Legibility also increases the ability for a large organization to chase singular ambitious goals.

For example: software estimation is hard. It takes time, it's highly inaccurate (frequently by 2x or more), and rewarding good estimation incentives worse software.

But having deadlines helps functions like marketing and sales immensely. Marketing can plan effective launches, and sales can guide customers towards new products with minimal lag in uptake.

Lots of corporate life is like this. It's a net good if it's used to deliver a better experience to customers. It's a constant fight to make sure legibility efforts aren't being used to hoard power for personal gain.

The article you linked basically answers your question. The standard model for defense companies isn't VC-friendly -- you're either building hardware with a limited ceiling, or you're building a consulting business that bills by the hour. Those can be fine businesses, but they're not amenable to wiring $10m and getting back $100m 3 years later.

Defense buyers aren't inherently irrelevant, though. I'm working on a defense-related project at a VC-backed company, and plenty of our peers are. We're selling a standard SaaS model, and that continues to support our business model.

The real challenge is that most tech companies scale up before going after defense, so they can absorb the costs. If you're small, you're going to have to figure out how to scale quickly in the defense world. If you can crack that, I'm sure you'd find a VC or two.

As CTO, you probably don't have the time and space, structurally, to be a good developer. You have things to do that aren't a single project or feature! So of course other developers will be better at "being a developer" than you, by some measures.

I've seen two successful models of CTO that are "good developers:"

1. Knows the core tech of the company really well. There's some small bit of tech at your company that's legit an enabler of your growth. You should probably know it better than anybody. Maybe it's moved or changed or grown in the last 8 years, and you've lost touch a bit? It seems reasonable to carve out time to build something on your own, in the same space, using the new stuff. You probably have a better sense of what's important to get right than anybody else, so building a personal project with that tech on company time _is_ actually a great use of time.

2. Knows the system dependencies better than anybody else. This particularly applies if you have reliability woes or other systematic issues. You have a great position to ask other teams to educate you. Get a couple good devs from each team in a room, drop your ego, and have them teach you what they know in their domain. Do this across a bunch of teams. Some folks might find this irritating, but most folks will like the time to show off their work and share their worries.

Everybody is constrained by where they spend their time. If you want to go back to being a great IC ... you need a different job. Which it might be time for! But you can be technically great as a CTO, and I'm sure I've missed some other ways you can do it in the context of you "day job."

Contrast to Apple, which uses a fully functional model: https://hbr.org/2020/11/how-apple-is-organized-for-innovatio...

I find this evolution fascinating. It makes sense to me that the GM model worked for so long. If much of a company's advantages were cultural, you could put together teams that could out-compete startups within a large organization. Procter and Gamble needs to enter new markets with new products to grow -- there's only so big a market for soap or razors or drills.

But lots of tech companies today have a significant scale advantage. Google ads is most of their revenue; Netflix has a single product. Functional expertise that drives a 2% gain across the entire business is possible and valuable.

So ... why does Twitter want to go to a GM model? Are they watching Meta have success diversifying from Facebook? Or is there something else?

The interesting thing to me is that this indicates that investors largely believe founder-CEOs are the best CEOs for the company, long term. That’s new.

I suspect that part of this is due to extended timelines to go public. Bezos never got another grant after IPO, but he owned 41% of the company at IPO. Today, it’s more typical for even successful founders to end up with 5%-15%. Taking dilution in the C/D/E rounds and going public at $3b is a different beast than going public after the B (with a huge ownership stake) and growing a public company from $60m to $3b. It makes sense that the successful CEOs who gave more away in dilution end up with end power to demand it back post-IPO.

A database is the wrong solution to this problem. The problem is spam detection and blocking.

This solution is simple and good. A developer on the receiving end of a "add an item to this list" bug report has a clear, well-trodden way to do it: update the text file. They have a modern editor that handles big files just fine. They frequently deploy new code to production, including testing it.

Moving to a DB means that it needs an interface in production, because presumably Twitch doesn't let their devs run random commands on production boxes. It means ACLs and a sync between dev and prod. It means another moving piece when Twitch spins up a disaster recovery test.

The full-powered, industry-standard solution is an AI-based spam detector with some sort of lag on blocking (e.g. shadow banning). This requires inputs to train on, such as reports from Twitch moderates, which might need oversight. It requires ML engineers and quality heuristics. This stuff is getting cheaper, but it's a hell of a lot more expensive than an ugly text file.

I love this. It puts into (better) words my long-held belief that ideas do, in fact, matter.

Startups live and die by their ideas. The nuance is that ideas can be detailed or abstract, and those details can be some combination of right/wrong and relevant/irrelevant.

I've got back and looked at some of the early pitch decks of my current unicorn employer, and they're shockingly accurate. The founders / early team have been saying the same thing since before launch. It's unreal how fast they figured out the core value prop, and so much of the growth of the company is living up to the promises of that early idea.

Figuring out what really matters needs to happen early, and it's something that founders and early team need to internalize as much as possible. If you can get that deep understanding of what needs to be done, quickly, the company's odds of success go up so, so dramatically.

This is terrible, obviously. Where possible, all EV makers should protect against this.

Gas cars may not catch fire when they sit in a garage, but they're not perfectly safe. My parents' friend left the car on one night, by accident. The carbon monoxide killed half the family.

There's low-probability tragedies around us, each a little different. There's no silver bullet.

The biggest simplification that I disagree with in this post is that impact rises monotonically.

It's true that long-tenured engineers (if they're good) can do amazing things. Things that other engineers don't have the implicit knowledge or political capital to do. The problem is that once that project is over, it's not guaranteed that the next project will be as impactful.

So ... do you pay them for their impact? And promote them? Knowing you're guaranteed nothing the next quarter?

Or do you put up barriers to promotion? "Sustained performance" or "multiple successful projects?"

I'd love to see an approach to IC comp that's variable. Land a huge project; get a bonus. Everything past Senior Engineer is temporary; if you want to be paid like Senior Staff or Principal, do the work in that year. I suspect orgs could justify being more generous with bonuses because it's not recurring.

It's at odds to a lot of the way engineers look at comp, but it would align money with what individual businesses need in a more interesting way than the current "plan on your RSUs going up by 50% next year" comp plans at FAANG.

This line of thinking ends with "companies are evil." It's not totally wrong, but it's perhaps not a useful framing.

Businesses make money in all sorts of ways. Most big companies blend these together. NYTimes charges users AND sells ads. Enterprise SaaS sells licenses AND charges for installation and configuration services. App store widgets charge for download AND sell aggregate data.

Brand gets undersold in a lot of tech/startup communities, because those communities are frequently organized around doing new things, which necessarily don't have brands. But the answer to a lot of these problems IS brands, because brands are a proxy for trust.

At some point, if you want a service, you have to decide which company you trust the most to deliver it without misleading you or doing things you disagree with. Most businesses run without constant lawsuits keeping them in line, because trust and word of mouth and reputation matters.

If you can't trust anybody, then don't use computers, I guess. But the world is shades of gray, and I do think it's worthwhile to figure out which companies you do trust.

The opportunity cost is staggering. Not just of the people that do the work, but the cost of focus and top-level priorities over the years it takes to do the work.

Everybody loves to cite DropBox here. The greater arc of DropBox is their product stalled, they lost the enterprise of their market to Box, and they found themselves a commodity in a commodity market. Heck, maybe they'll go back to Amazon if they keep doing things like this: https://aws.amazon.com/solutions/case-studies/dropbox-s3/

But that's not all. It takes a top-level directive to repatriate an entire SaaS. That's at the cost of other top-level projects. It's wild to me that any company that has significant fuel left in the tank would buy back single-digit COGS percentages instead of investing in product that could add double-digit growth for several years at scale.