singingfish

I'm only concerned with the input efficiency, but the output energy efficiency is quite bad too. I can not see a pathway to generating economies of scale for this stuff.

Maybe I shouldn't have been pissed of by your commient. In which case - the excess costs of the training, output, feedback, and training cycle of artificial neural networks seem to prohibit the nightly training consolidation cycle (i.e. sleep) that is an important part of natural neural networks.

I can't make sense of your comment. Firstly because of the obvious massive over-build of GPU infrastructure the AI companies are engaging in. Secondly because the instance of the LLM in the data centre that users interact with is only a small part of the story. The training phase is clearly prohibitively expensive, thus the fact that these things have no way to learn from experience except by smoke and mirrors.

Also your comment feels like the classic climate denial discourse - say something a bit complicated and a bit difficult to follow that looks at a very small out of context part of the story to cast doubt.

I can't see a pathway for these things to be able to learn from experience in any meaningful way - the energy budget seems to prohibit it. Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks. Despite this, large nervous systems are very energy intensive as well. For example in humans 20% of the energy budget goes to the brain which is 5% of the body weight.

So I refrer to LLMs as language extrusion confabulation machines. Language extrusion was a term I heard the linguist Emily Bender use. Confabulation because my observation is that talking to an LLM is very much similar to my experience of interacting with Korsakov syndrome patients some years ago.

I look forward to the hype settling down to see what we end up with.

Meta's bots are spectatularly awful, and if your organisation needs to use facebook advertising, they're also unblockable. There are ways around it that doesn't exactly block them but does clearly communicate they should GTFO, but it just shows you what an awful company Meta is.

It was brought to my attention that the phrase "terminal, degenerative, sexually transmitted condition" returned no results from my search engine. So I tried to fix it.

as a rule the interviewee should be assessing for goodness of fit, as well as the interviewer. If my simplified nuanced answer to why I limit my usage of LLMs was poorly received, I wouldn't want to take a job somewhere likely to make me miserable.

aka LLMs can not learn from experience - this is a fundamental limitation. c.f - individuals with Korsakov's syndrome - who also confabulate in a similar manner.

Both Yamaha and Roland keep a long back catalogue of reasonably priced parts for at least their mid-range pianos and up as well. I should check Korg too I guess.

I've had some impressive customer service from Yamaha concerning decades old saxophones that they have zero prospect of generating revenue from in the future, for an unrelated (musical) data point.

Of course they do - it's the same as bookmakers or other gambling syndicates, it only makes sense to operate at a certain scale otherwise the rounding errors and stochasticity will kill you.

If you want to just stop yourself from shooting yourself in the foot in your current checkout and maintain privacy around your own stupidity, you can add patterns to `.git/info/exclude`

and long may he idle

dumb, sick, stupid, other adjectives - generally not negative although could be in the right context - both in the Australian and North West England (where Matt was from) context. Just watch out for being called a fucking cunt. In almost all cases that one is pretty unambiguous. I do remember at one point having a conversation with Matt about how handling East Coast and West Coast North Americans was quite a different proposition from a cultural perspective, also related to this thread.

Matt would have liked this discussion. And given that his and my mutual friend has some actual legitimate serious expertise on obscenity in the English language even more so.

and child prodigies are notorious "arseholes" QED.

I totally get it, some people found his mess very difficult and it could easily lead into a death spiral. Others had a very different experience. He was certainly someone needed to be managed by those who knew him well from time to time, when possible.

Matt taught me everything I know about how to make commercial programming creative, engaging, artistic/craftsman type activity, aligned with my desire to keep everything open source to the maximum extent practical.

Another former colleague who is way more talented than I am emailed me privately to express a similar sentiment.

You'll find Matt's indirect influence in things like SQLAlchemy, and chunks of the enduring parts of the javascript ecosystem as well. He was known in the perl community, but his unparalleled thinking skills have a much wider indirect influence

Matt was a child prodigy, and child prodigies have it notoriously tough. He and I worked closely for a while. There is someone else important in my life that has somewhat close to mst's intellectual gifts, and similarly to mst they also have difficulty controlling their reaction to other people. However, unlike this other person in my life, mst did know how to express accountability and had been on a learning process to deal with his limitations. Matt and I never had beef, perhaps because we recognised that our respective strengths and weaknesses were complementary.

Yeah, there's useful, but not that useful. It help to think of them as "language extrusion confabulation machines" to understand the actual limitations.

In other news, water is wet and the sun rises every morning.

What about extra feed - grain or silage or whatever? Or do they keep stocking density low?

Tyre is clearly the correct spelling. I quickly tire of anyone claiming otherwise. Gaol is an archaic spelling which I've found incredibly confusing ever since I was a small kid a long long time ago.

All that fertiliser that gets put into Cattle grazing lands increases the methane and other emissions, so your read on this is to a significant extent incorrect.

Was just talking to a Cambodian wildlife biologist about this very thing last night. In a South East Asian context rather than North American that is.

me? No. Ex trainee neruopsychologist and failed academic who was in the right place at the right time back in the mid 90s who didn't pick up computers for professional interest until the mid-late 2000s after getting excited by Neil Stephenson's Cryptonomicon when I was looking for a career change. These days I identify as an international computer hacker, but mainly to take the piss (due to the tiny element of truth sitting underneath)

Yeah the internal representation of organic neural networks are also weird - check out the signal processing that occurs between the retina and the various parts of the visual cortex before any decent information can emerge from the signal - David Marr's 1980s book Vision is a mathematically chewy treatise on this. This leads me to start thinking that human intuition may well caused by different neural network subsystems feeding processed data into other subsystems where consciousness and thus intuition and explanation emerges.

Organic neural networks are pretty energy efficient in comparison- although still decently inefficient compared to other body systems - so there is the capacity to build things out to the scale required, assuming my read on what's going on there is correct, that is. So it's not clear to me that the energy inefficiency of ANNs can be sufficiently resolved to enable these multiple quasi-independent subsystems to be built at the scale required. Not even if these interesting looking trinomial neural nets which are matrix addition based rather than multiplication come to dominate the ANN scene.

While I was thinking this comment through I realised there's a possible interpretation wherin human activity induced climate change is an emergent property of the relative energy inefficiency of neural architecture.

The reasoning is quite subtle, and because I'm not a very coherent guy I have problems expressing it. In the LLM space there are a whole bunch of pitfalls around overfit (largely solvable with pretty standard statistical methods) and inherent bias in training material which is a much harder to problem to solve. The fact that the internal representation gives you zero information on how to handle this bias means the tool can itself not be used to detect or resolve the problem.

I found this episode of the nature podcast - "How AI works is often a mystery — that's a problem": https://www.nature.com/articles/d41586-023-04154-4 - very useful in a 'thank goodness someone else has done the work of being coherent so I don't have to' way.

Yeah, give 'em small constrained jobs where the lack of coherent internal representation is not a problem.

I was involved in ANN and equivalent based face recognition (not on the computational side, on the psychophysics side) briefly. Face recognition is one of these bigger more difficult jobs, but still more constrained than the things ANNs are useful for.

As far as I understand none of the face recognition algorithms in use these days are ANN based, but are instead computationally efficient versions of the brute force the maths implementations instead.

I've been following the whole thing low key since the 2nd wave of neural networks in the mid 90s - and made a very very minor contribution to the field which has applications these days back then too.

My observation is that every wave of neural networks has resulted in a dead end. In my view, this is in large part caused by the (inevitable) brute force mathematical approach used and the fact that this can not map to any kind of mechanistic explanation of what the ANN is doing in a way that can facilitate intuition. Or as put in the article "Current AI systems have no internal structure that relates meaningfully to their functionality". This is the most important thing. Maybe layers of indirection can fix that, but I kind of doubt it.

I am however quite excited about what LLMs can do to make semantic search much easier, and impressed at how much better they've made the tooling around natural language processing. Nonetheless, I feel I can already see the dead end pretty close ahead.

The extra boilerplate noise with excessive repetition doesn't help one little bit.