lumost

No, reusing segments of the kv cache for different purposes in an order independent manner is an active research area.

The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache.

There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.

Is this the inevitable outcome of frontier labs who own their hardware? the GPUs and datacenters are the major cost. The inference and training a higher tier value proposition, if the company gets nervous that the investment in hardware won't pay off - renting it becomes a major topic of conversation.

A frontier model team having to fight their board on whether to monetize the datacenters directly or continue to invest in AI work is going to have a hard time.

Is it really "wear and tear?" or is it an evolved mechanism to keep genetic drift and natural selection alive? Alternatively, it could be an evolved mechanism to avoid genetic bottlenecks caused by highly reproductive individuals over long periods of time.

If John and Mary were first, how long until everyone is a descendent of john and Mary?

Why do we think frontier model vendors are high margin?

TSLA has been around for many years, whether I agree with its value or not. It has been able to retain its valuation in the public market.

An IPO with massive insider selling counterbalanced by a flood of index fund rebalancing is entirely different.

It's quite clear that there is an effort to engineer mega financial vehicles that index tracking funds are forced to buy. The incentive to do so is massive, and there is nothing illegal about it.

As a holder of index funds such as the S&P, I'd much prefer that these vehicles are excluded for at least some period of time to ensure that the greater fool isn't simply my index portfolio.

the math is odd.

Paying 2k for something that you use 100 hours of is quite expensive. Having the capability built into your existing silicon which you would buy for 1k is cheap. Paying 200 dollars a month for 2 years give a present value of $4200 dollars. Meaning that that paying 2k upfront cuts your overall spend in half.

I spent 6k on codex last month which, if repeated, implies a present value of ~144k.

The current theme is that agi may not be definable, and an ai which matches humans on all economically relevant tasks is close enough for business purposes.

Billions spent on RL may be good enough to beat human performance.

I can't put a 10 page narrative on how my specific motor should work into a hacker news post ;) you can also imagine the above where the goal is to have the ai exceed the performance of stock motors.

Laptops/desktops are cheaper per flop than any datacenter hardware by a good order of magnitude.

The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin.

You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x the same hardware sitting in the back of an office somewhere. While all of these premiums matter for 24/7 systems under active development, they don't really matter for ephemeral small scale workloads.

"Design me a 3d printable rocket engine for a hobby rocket project. Verify it's design in a full simulation. Iterate until it works reliably in simulation based on a verified printable design on a consumer laser sintering device (or substitute contract manufacture for under 1000 dollars)."

This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics studies, material analysis, operating systems etc.

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription.

There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than sufficient.

Taste, architecture, new innovations. These are all streams of tokens which are subject to the same scaling laws as code, language, and basic classification.

We are going to see a new generation of models which effectively “solves” these problems for most businesses. Likely within the next two years- then we’ll talk about some other problems which limit adoption.

The author’s implied argument is that capital control’s the entrepreneurship game already.

There is only 1 top law firm, financiers of law firms have no interest in starting a race to the bottom. Foundation model labs will take a significant portion of the value, the remainder will be captured by entrenched monopolies.

They are likely also starting to realize that the end result of their anthropic contract is that nobody but anthropic knows how to run their business. Why would anthropic not treat their business like a utility in the future?

The critical view is that these IPOs are bumping up against physics. How many trillion dollar companies can the economy support? The US GDP is roughly 32 Trillion. A company with 100 billion dollars in revenue and 10x annual growth would be expected to increase the size of the economy by 3.2% in its first year, and about 30% in it's second year post IPO existing.

While we could claim that such a company can grow by consuming a larger share of the GDP ... this would not bode well for future political stability, and nationalization would be a major topic.

So your left with a fast take off scenario, a job apocalypse, or a massively reduced growth rate.

Which also leaves OpenAI vulnerable to NVidia's aggressive pricing. To my knowledge Anthropic is relatively well positioned across multiple compute vendors/hardware providers.

I find it to be a fundamental hinderance as they use a cheap model. The cheap model gets confused based on conflicting reddit threads and then I get a wrong answer.

It's better to just ask codex to do the search for me, but this is much slower - but increasingly my go to. I wish there was a fast search api codex could hook into to answer internet questions faster.

The main story of agent use cases is in enterprise so far. An enterprise will only pay for a model capable of handling the task and no more. Most enterprise's see no need to hire PhDs as factory line workers.

Coding is an interesting case as [1] the pace of progress has been absurd and [2] it's hard to put an upper bound on required capability. However hard to put a bound on and will are different, it's quite possible that the average engineer will cease to see the benefit of rapid progress - or that their employer will be satisfied with lower tier models.

How smart of a model do you need to build a high quality CRUD app for internal users? Or build a scalable web service?

Pretty much, Kimi K2.6 is opus 4.6 quality for coding. If you include discounts due to more efficient input caching it is around 1/10th of opus4.6.

https://openrouter.ai/moonshotai/kimi-k2.6

The march of cost efficiency moves on.

For equal capability tokens, there has been about a 10x drop in cost every 6 months.

We are still chasing the best because the best is moving rapidly, but it’s a simple thought experiment to work out what the cost to serve an 8B model from 2 years ago is in a world of 2T models.

Note: parameter counts are illustrative. Concretely, qwen3.6 27B delivers opus 4.5 capability at 1/27th the cost on openrouter. Single chip llama3 8b performance can exceed 17k tokens/sec.

but then you would need to keep running it, what if you don't want to do that anymore? how do you incentivize someone to keep it going to pay off the debt you just borrowed outside of the llc.

It provides liquidity to business owners.

As a business owner, if you want cash today because you are done with a business. You could go to a bank and get a loan to pay dividends. This is a bad deal for the bank as you have no incentive to operate the business after you cash out the loan. A private equity firm comes in and operates the business on the model that they still keep some of the profits after the loan value.

The crappy side comes in as a customer, the PE firm can do this to an arbitrary number of firms in the area and raise prices on each/cut services. PE firms can trivially build out monopolies. Many of these monopolies will be invisible as they leave the existing branding etc. in place.

How many of those wtf moments are simply from not “being in the room when it happened?” Most enterprise software is riddled with wtf moments demanded as one compromise or another.

The great fear of our current incarnation in AI is that it will create a permanent class divide between those who own the AI and those the AI commands. There may not be a "go to college and get a better job" option in the future if AI eats the majority of knowledge work. The traditional "invisible hand" only matters if we still live in a capitalist society.

This will probably lead to some balancing act like ye olden days of big data etc. Companies want AI native engineers who will use AI to do their work, but don't want AI quality outputs and don't want to drop 200k per year per employee.

AI quality outputs are fine for backoffice work now, but they are awful to read and reason about. Hallucinated features are also difficult to work with.

The economics of tokens are still favorable for reasonably well defined tasks at the top tier. I can build out a detailed design and then spend a few k turning it into reality. Compared to spreading such work across multiple engineers the economics are simple.

The more nuanced question is what are the economics for tasks with ambiguous roi. I spent 100 dollars optimizing a build/ci system on Friday because it made me “feel” more productive. The result of about 24 hours of gpt5.5 in codex was a build time reduction of 75%.

I have a family split along classic ideological lines between the northeast and southeast of the US. If you are unfamiliar with conservative's hatred toward immigration, I suggest you travel more.

Maxed out the 200 dollar per month plan then incurred the overage. This was over the span of 1 month or so.