angusturner

You assume that for small steps (I.e taking some noisy code and slightly denoising) you can make an independence assumption. (All tokens conditionally independent, given the current state).

Once you chain many steps you get a very flexible distribution that can model all the interdependencies.

A stats person could probably provide more nuance, although two interesting connection I’ve seen: There is some sense in which diffusion generalises autoregression, because you don’t have to pick an ordering when you factor the dependency graph.

(Or put otherwise, for some definitions of diffusion you can show autoregression to be a special case).

Hm suppose for argument sake that feeding a batch of data through some moderately large FF architectures takes on the order of 100ms (I realise this depends on a lot parameters - but this seems reasonable for many tasks / networks).

Now suppose instead you have an CTM that allocates 10ms on the standard FF axes, and then multiplies it out by 10 internal “ticks” / recurrent steps?

The exact numbers are contrived, but my point is : couldn’t we conceivably search over that second arch just as easily?

It just boils down to whether the inductive bias of building in some explicit time axis is actually worthwhile, right ?

I’m really glad to see people converging on this view because I feel a bit insane for not understanding all the hype.

Like, yeah, we need a standard way to connect LLMs with tools etc, but MCP in its current state is not a solution.

I think this is article is too generous about the use of stdio - I have found this extremely buggy so far, especially in the python sdk.

Also if you want to wrap any existing code that logs or prints to stdout then it causes heaps of ugly messages and warnings as it interferes with the comms between client and server.

I just want a way to integrate tools with Claude Desktop that doesn’t make a tonne of convoluted and weird design choices.

Yeah this is what I don’t really get about the stable coin hype.

In Australia for example, we are almost cashless now and most bank transfers are instant and cheap - no blockchain required.

Digital currency is a good idea - unclear what value the blockchain part actually provides?

I would actually say that scams and financial crimes seem to be the only concrete use case of the technology at the moment.

(Unless you count speculation/gambling).

The thing is though, once you regulate crypto then what’s the point? You are left with a highly inefficient/expensive and immutable database.

Bitcoin solves (or attempts to solve) for exchange in absence of trust and regulation. But this is a stupid thing to solve for, because without trust and regulation you can’t even have a functioning society.

The thing this really hits home for me is how Apple is totally asleep at the wheel.

Today I asked Siri “call the last person that texted me”, to try and respond to someone while driving.

Am I surprised it couldn’t do it? Not really at this point, but it is disappointing that there’s such a wide gulf between Siri and even the least capable LLMs.

I have been developing with MCP for a few weeks now, making some small python and javascript servers to integrate with Claude Desktop.

I am yet to see a use case that wouldn't be better served with an HTTP API. I understand the need to standardize some conventions around this, but at the heart of it, all "tool" use boils down to: 1. an API endpoint to expose capabilities / report the API schema 2. other endpoints ("tools") to expose functionality

Want state? ("resources") - put a database or some random in-memory data structure behind an API endpoint. Want "prompts"? This is just a special case of a tool.

Fundamentally (like most everyone else experimenting with this tech), I need an API that returns some text and maybe images. So why did I just lose two days trying to debug the Python MCP SDK, and the fact that its stdio transport can't send more than a few KB without crashing the server?

If only there was a stateless way to communicate data between a client and a server, that could easily recover from and handle errors...

I mean, it seems possible that with greater absolute wealth people might tolerate greater relative inequality but like … why run the experiment?

Anyway, that’s the path we are on - Piketty doesn’t hazard a guess about if/when society collapse, IIRC the point is just that things will only continue to get more unequal unless there’s a major crisis or some political intervention (e.g wealth or inheritance taxes - imagine trying to get the US to agree to that lol).

Also you can’t ignore relative inequality because wealth is strongly coupled to political power/influence.

Look at the head of DOGE - he spent $300m to get himself a top gov role, in charge of regulating his own business and restructuring US gov spending to suit his own whims.

In terms of percentages / orders of magnitude it’d be like your avg citizen spending $100 for a top gov role.

I personally think we should resist the temptation to separate out values from the discussion. Like, even if it was the case that some extremely high-level of inequality turned out to be "optimal" in some GDP-maximizing way, the question still ought to hinge on some notion of fairness or justice (in my view).

(Although, I guess if your metric is "citizen satisfaction" maybe that's not as terrible).

In any case, there are lots of case studies showing what happens at the extreme inequality end of the spectrum. As I mentioned in another comment, my favorites come from Piketty "Capital in the 21st Century" and Acemoglu & Robinson "Why Nations Fail".

In the latter, the case study on the rise and fall of Venice is particularly fascinating - huge economic growth due to inclusive economic institutions that promoted social mobility, followed by a downturn once the aristocracy moved to entrench their own interests at society's expense. This seems to be the central thesis of the book, although I'm only a few chapters in.

The parallels with modern US politics are pretty hard to ignore though.

When I was at uni I read Thomas Piketty's "Capital in the 21st Century", and parts of his newer "Capital and Ideology" (although I never quite got through that one).

The big takeaway for me was that wealth inequality never improves without some major catastrophe (war, revolution, plague etc). The proposed model is really intuitive and compelling (tldr; return on capital has historically always been higher than real growth, which guarantees indefinite concentration of wealth until there's a crisis).

Last year's Nobel Economics Prize winners, Acemoglu and Robinson, tell a similar story in "Why Nations Fail", which I am working through at the moment. Although in their case, they seem be suggesting a more causal link between erosion of political and economic institutions and the collapse of empires.

I wish these ideas were more broadly accessible and understood. The real risk of total societal collapse should transcend any partisan fighting about ideal tax rates, government inflation/unemployment targets etc. Everyone has a common interest in there not being a violent upheaval (arguably the rich most of all).

Last I checked the numbers, current wealth inequality seems about as bad as it was before the great depression. And its not enough to just say "well, absolute wealth is more important". As others have pointed out, its not stable to have such huge relative wealth disparities. And that's before you even consider corruption.

I have no idea what you are basing this claim on, although it seems in stark contrast to the evidence provided by other commenters in this thread.

AI is not a replacement for engineering expertise and deep understanding of the actual user problems that you are trying to solve.

Any 19yo at DOGE can have Cursor pump out shitty web portals without regard for the problem context or actual end user.

But that’s not what engineering is about and it’s depressing to see people celebrate and advocate for that kind of mediocrity.

Credit to the engineers that built this, but it fills me with rage that Elon has this sort of unchecked power.

How long before this starts getting deployed in safety critical applications or government decision making processes?

With no oversight because Elon seems to have the power to dismiss the people responsible for investigating him.

Anyone not scared by this concentration of power needs to pick up a book.

I am so so sorry you have to deal with this. As an Australian I have been watching on in horror this week at the way trans persons are being demonized and oppressed in your country. I know HN is meant to be an apolitical space, but I hope that the mods here have the sense to allow a certain amount of push back again this fascist nonsense.

I think the tools to solve the challenges of waste, environmental damage etc. already exist within the framework of capitalism. Mostly they are just unpopular and seen by many as a government overreach.

1. taxes that force corporations and individuals to pay for the negative externalities / social costs of their actions 2. regulation (e.g. stop allowing planned obsolence, mandate the right to repair etc.) 3. government spending into R&D, incentives and subsidies for renewables etc.

Anyway, my point is that the issue is basically one of co-ordination and political will. It obviously doesn't help that many Americans (and Australians too for that matter, where I live) don't accept the basic facts of the situation (before we can even discuss solutions).

Not sure why this is down-voted so heavily.

In Australia, the property market has been artificially inflated for years with favorable tax incentives that encourage investors to hoard properties and sit on them, while writing off any losses as tax deductions.

Meanwhile, renters are being squeezed because landlords are trying to pass on the cost of rising interest rates to their tenants, who are forced to choose between groceries and making rent.

And yet, I would bet my kidney that when interest rates fall we won't see landlords pass on the savings to their tenants.

As the OP's point, the landlord renter system HAS "divided our society into haves and have nots", and it causing serious bitterness in division in our country. And the issue is politically intractable, because the propertied class are not going to accept any reform that lowers the value of their house.

Fascinating - thanks for the detailed reply. I can’t believe I failed to think of lighting but this makes so much sense.

It’s almost like for a pure ML solution you would need a nerf (or similar) which is conditioned on the entire (dynamically changing) scene geometry and lighting positions?

Graphics isn’t my area of ML though, so I’m sure there’s a lot of nuance that I don’t appreciate.

Can anyone familiar with 3d graphics speculate what would be required to implement this into a game engine?

I'm guessing that adding physics, collision-detection etc. on top of this is non-trivial compared to using a mesh?

But I feel like for stuff like tree foliage (where maybe you don't care about collisions?), this would be really awesome, given the limitations of polygons. + also just any like background scenery, stuff out of the player's reach.

Are beyond and impossible really failed? In Australia they seem quite popular. Just basing this off the fact that the supermarket section for plant based meat/dairy alternatives seems to be exploding in recent years. Also a few popular burger chains here offer impossible meat as one of the main vege options.

Its worse than that. I've seen some people conflating the term "AI" with "LLM".

Great write up. The Github mirror is also quite convenient.

I would love for a non-python based deep learning framework to gain traction.

My initial impression though is that the scope is very broad. Trying to be both sci-kit learn and numpy and torch seems like a recipe for doing none of these things very well.

Its interesting to contrast this with the visions/aspirations of other new-ish deep learning frameworks. Starting with my favorite, Jax offers "composable function transformations + autodiff". Obviously there is still a tonne of work to do this well, support multiple accelerators etc. etc. But notably I think they made the right call to leave high level abstractions (like fully-fledged NN libraries or optimisation libraries) out of the Jax core. It does what it says on the box. And it does it really really well.

TinyGrad seems like another interesting case study, in the sense that it is aggressively pushing to reduce complexity and LOC while still providing the relevant abstractions to do ML on multiple accelerators. It is quite young still, and I have my doubts about how much traction it will gain. Still a cool project though, and I like to see people pushing in this direction.

PyTorch obviously still has a tonne of mind-share (and I love it), but it is interesting to see the complexity of that project grow beyond what it is arguably necessary. (e.g. having a "MultiHeadAttention" implementation in PyTorch is a mistake in my opinion).

One area that I would dive into (if I had more time) is "geometric deep learning". i.e) How to design models in a principled way to respect known symmetries in the data. ConvNets are the famous/obvious example for their translation equivariance, but there are many recent examples that extend the same logic to other symmetry groups. And then there is also a question of whether certain symmetries can be discovered or identified automatically.

Sorry, but I cannot get past the fact that so many Americans openly refer to undocumented migrants as "illegals".

What is really dystopian is focusing blame onto immigrants instead of the economic and political elite.

I think this “wisdom” is not really agreed upon, and in fact I’ve seen a few works successfully leverage synthetic data (including a cool one from Sergey Levine et al recently).

I’m an Aussie, and after living 3 months in LA I think it was the most poorly designed city I’ve ever been to for this exact reason - I felt unable to walk practically anywhere!

It felt like my options were drive or taxi. And we know what LA traffic is like.

I can’t speak to other US cities, and it is possible that certain areas of LA are less terrible than where I stayed. (But I will say, I was in quite an affluent area which had no business being unwalkable and without public transport).

But it really opened my eyes to how good we have it in Australian cities (which themselves are still far behind many European cities).

HN still seems to value constructive criticism though, to an extent that is simply not possible on YouTube.

YouTube comment section is basically a cesspool of toxic positivity now, with absolutely no informational value at all.

I feel like this has been the case for a couple of years now, where nothing even slightly critical will be surfaced by whatever automated algorithm they are using to filter/boost content.

Anyone else think its an abuse of terminology to refer to speaker conditioning as "in-context learning" and "prompting" now?

Like, using a reference encoder to condition on an unseen target speaker has been around for 4 or 5 years now. These results are already cool without mystifying the results by calling this "prompting" or "icl"

Unless there is some subtle difference that warrants the new terminology?

I guess the challenge here is that the human mind is not a blank slate, and has been optimized first by billions of years of evolution.

If it takes all the data on the internet (or more) to bootstrap AGI, but that system is then capable of leveraging its knowledge to solve new out-of-distribution tasks, that seems like a fair test to me.

I agree with the article that we see "sparks" of this generality with GPT4.