wokwokwok

This is super big news if it’s real.

Basically, given an agent with an initial set of predefined actions and goal, they’re saying “decompose this into steps and pick and action to achieve each step”. Pretty standard stuff.

Then they say, hey, if you can’t solve the problem with those actions (ie. failed repeatedly when attempting to solve), write some arbitrary generic python code and use that as your action for the next step.

Then save that as a new generic action, and slowly build up a library of actions to augment the initial set.

The thing is, there’s no meaningful difference between the task “write code to solve this task” and “write code to solve this action”; if you can meaningfully generate code that can, without error, perform arbitrary tasks, you’ve basically solved programming.

So… that would be quite a big deal.

That would be a real “Devon” that would actually be able to write arbitrary code to solve arbitrary problems.

…which makes me a bit sceptical.

Still, this seems to have at least worked reasonably well (as shown by being a leader on the GAIA leaderboard) so they seem to have done something that works, but I’m left wondering…

If you’ve figured out how to get an agent to write error free deterministic code to perform arbitrary actions in a chain of thought process, why are you pissing around with worrying about accumulating a library of agent actions?

That’s all entirely irrelevant and unnecessary.

Just generate code for each step.

So… something seems a bit strange around this.

I’d love to see a log of the actual problem / action / code sequences.

Many sites have a two tier application with a CMS SEO “landing page” abd conversion funnel run by the marketing department and a user-login that takes you to a separate website that is SEO irrelevant.

So… a site that doesnt care about SEO and is not indexed and is purely for logged in users is also very common.

You’ve taken literally one word from my comment, out of context.

Yes, you’re right, they rushed it through. …and that sucks.

Now what? Is that really the only meaningful complaint you can come up with about it?

I don’t support it because I think it’s overreach by a government that is already trying to make a nanny state.

…but I’m not naive enough to think it’s not going to work. It will work.

It probably has unintended consequences; that’s the problem.

I’m not for this legislation, but your argument doesn’t hold up.

If people are hiding it, that in itself normalises that people are not using social media.

That’s the point; taking someone’s phone away makes them an outcast. If no one at school is “allowed” to have a phone, then it’s normalised, it’s normal to not have one.

It is pandering to the minority, but your argument hides behind insults to avoid facing the reality which is that it will normalise people not being on social media, which will make it easier for parents to ban their kids from using social media.

This isn’t some black market crap where the kids are nipping down to the servo to buy some vapes and get a slap on the wrist if their caught; if they’re banned and they’re caught, their account will get suspended.

That's instant death on SM, because of early adopter network effects.

It really will do something. You don’t understand the dynamics on SM if you think otherwise.

Whether you agree it’s good or not, is certainly open to debate. What it will do is open to debate; hidden group chats and an escape to platforms that are “technically not SM”, who knows… but the current landscape of SM & TikTok follower accumulation will change, dramatically.

Purego is ffi and is slower than cgo.

That library (like this) claim “no cgo” when they use ffi is just ridiculous.

This one is particularly egregious in even supporting cgo!

Pure go my ass. You’re invoking c functions. It’s not pure go.

The distinction is in the tool chain; cgo is a better, first party solution.

Purego is an unstable (it is, read the repo!) workaround for people who a) don’t like the word “cgo” or b) dont like having a c tool chain.

…but realistically you have to interop with c code in many domains.

Sure, a pure world is great, but that’s not always possible.

People who don’t like cgo and choose purego are doing it for, mostly, optics (cgo is bad! This isn’t cgo… it’s ffi… totally different) and appear to fail to appreciate the technical distinction more complicated than they imagine, and they are still using c.

It’s not a technical thing.

It’s not bad.

It’s a misguided community thing.

Cgo is the correct choice for many domains (ui, gamedev, systems stuff) where you can’t use a network boundary as your “interacts with other languages”.

The language is opinionated; it’s not good at playing with others. It’s good at other things (network services, etc)… but you can and it’s not bad.

…it’s just hard, and people don’t like hard things.

That sounds like 'running on CPU considered to be competitive to running on GPU' and my BS alarms are going off...

...buuuuut, to be fair, I don't either a) particularly care, or b) really have that much knowledge about the Ryzen AI 9 HX 37 CPU.

Anyone care to point me vaguely in the direction of something meaningful that would suggest that CPU based inference would, in general, be even remotely comparable to dedicated GPU performance for AI?

I remain pretty skeptical.

(That reddit thread kicks off with someone saying 'LLM isn't particularly compute intensive workload', which is obviously a) false, and b) gives me no confidence in their review. If LLMs were just memory intensive, we would all just be getting machines with more RAM and no one would be going crazy with 4 parallel 24GB GPUs just to eek out the performance for inference; they'd just slap a 64GB sim in their machine and be done with it).

LLM inference it is memory intensive, but it IS compute intensive too.

Being able to run on a CPU at 1 token per century is not what anyone wants.

> I guess I feel bad for people who are particularly sensitive to the areas that Go does not succeed in, because they are probably going to be complaining about it for the rest of their lives.

Well, that’s a stellar endorsement of the article, because that’s literally the point they’re making.

You’ll use go.

…and then regret it.

…but by then it’ll be too late, and you’re stuck with it.

I think the author makes a compelling argument, which is very difficult to counter, that it is a bad choice and you will regret having it in production in many of the explicitly listed cases, and in many professional situations where companies that are not technically competent use unsuitable tech.

Companies should stick to boring tools.

…but, for personal projects? Sure, go for it.

The parent said “normal people”; that’s the majority, by definition. That’s what “most people” is.

The technical details of the reverse engineering are from last year (1) (or here (2) directly), this post is just regarding the announcement of a big fine, and some rambling.

[1] - https://reversing.works/posts/2023/12/mobile-reverse-enginee...

[2] - https://media.ccc.de/v/37c3-12133-mobile_reverse_engineering...

Most. Most normal people.

Sure I know people who pay for it too; but I know a lot of people who like free things and don’t or can’t pay for subscriptions.

Do you think most people have a spare $30 to spend every month on something they already get for free?

At the moment? I don’t.

> I really find it hard to believe

What's true and what's not true is not related to what you personally believe.

It is factually and unambiguously false to state that generated code is, in general, not similar to other code from the corpus it is trained on.

> And none of it appears anywhere else; I've checked.

^ Even if this statement, is not false (I'm skeptical, but whatever), in general, it would be false for most users of copilot.

None of it appears anywhere else? None of it? Really?

That's not true of the no-AI code base I'm working on.

That's very difficult to believe it would be true on a code base heavily written by copilot and the like.

It's probably not true, in general, for AI generated code bases.

We can have a different conversation about verbatim copied code, where an AI model generates a large body of verbatim copy from a training source. That's very unusual.

...but to say the generated code wouldn't even be similar? Come on.

That's literally what LLMs do.

> even though they're not actually true for anything but the most trivial hello-world types of problems.

Um.

All the parent post said was:

> then try to find similar code on the web, you usually will.

Not identical code. Similar code.

I think you're really stretching the domain of plausibility to suggest that any code you write is novel enough that you can't find 'similar' code on the internet.

To suggest that code generated from a corpus that is not going to be 'similar' to the code from the corpus is just factually and unambiguously false.

Of course, it depends on what you interpret 'similar' to mean; but I think it's not unfair to say a lot of code is composed of smaller parts of code that is extremely similar to other examples of code on the internet.

Obviously you're not going to find an example similar to your entire code base; but if you're using, for example, copilot where you generate many small snippets of code... welll....

You can be reasonably confident that unless there’s been a significant breakthrough (there hasn’t) if a smaller model beats a larger model it’s either fine tuned for a specific purpose or trained on the test data somehow (ie. fine tuned to have good metrics).

To be less snarky, they claim:

> These models are built on a meticulously curated high-quality training corpus

Ie. Good training data plus a small model beats a bigger model.

…but I’m skeptical, when I read:

> We observed that performance continues to improve with longer training, even beyond the Chinchilla optimal point. Therefore, we decided to train the 1.7B model on 1 trillion tokens and the 135M and 360M models on 600B tokens, as the performance gains after 400B tokens begin to slow on some benchmarks for these smaller models.

So they’re evaluating their models against various benchmarks as they train them and picking the practice that gives the best benchmarks?

I dunno.

The claim is basically good data > more parameters, but it’s just an observation of “this happened to work for us” rather than something you can usefully take (as far as I can see) and apply to larger models.

The claims they actaully make about performance are far more modest than people are making out.

The 1.7B model performs better than any other 2B models in their evaluation.

Seems nice. Not ground breaking. Not convinced it’s real rather than polluted training data personally.

> Also, it's not objectively easier to make a mistake when you type one command, than when you type three.

It is; because you can pause between doing those three commands. Adding a review step objectively lowers the error rate in activities. That's why we do PRs.

> I would say my error rate goes up as I type more, so I feel it's objectively easier to make a mistake with your method.

The word you're looking for is 'subjectively', not 'objectively'.

> Also, you still haven't discussed that your method does the (in my opinion) clearly wrong thing of building whatever the current state of the master branch is, rather than a release.

Use `git clone -b`... or, use a a proper package manager with support for features like parallel binary dependencies and prebuilt binaries.

What you’re written it literally false.

You have no idea what that command does. It fetches something off the internet and builds it and installs it.

It is objectively less explicit than having a local folder that you have explicitly fetched, and run a build on.

It is also objectively easier to make a mistake when you type a single command that fetches and installs as one step, than it is to clone, and review then build it.

So, apart from being objectively wrong, I do agree with you; for now, it is easy, convenient and honestly, good.

However, I think you’re being extremely short sighted.

As I said, and as you can see, yourself, from the “command-install” related issues on the cargo issue tracker, the status quo isn’t gong to last. More people will use it. More people will want to distribute binary dependencies and prebuilt binaries.

We are walking blindly into a future where a “cargo install marmelaide” (whoops, was it marmite? Did I spell it right?) installs what you didn’t expect it to, and where people increasingly pressure for more features to be added to cargo to support, the obvious lack of features cargo install currently has (again, read the issue tracker if you doubt me or for some reason think I'm making this up).

I don't support that; but sure, that's just my opinion.

There are other fully featured better designed alternatives. Cargo does not need to be, or should be, a dumpster fire of features in order to be a generic cross platform application distribution tool.

I don't see that as being controversial. /shrug

bininstall is a third party project.

There’s nothing wrong with building an app store in rust; I’m just saying that it shouldn’t be part of cargo.

> brew install cargo-binstall

…and it is not. It’s a great project, and if you want to use it, you can.

Using rust does not opt you into using it, just like it doesn’t opt you into using brew.

Crates.io is a flat namespace.

If you have to lookup the exact crate name (and you do, because otherwise you’re installing who-knows-what) what difference does it make?

It’s just a lazy tool for lazy installs that only works when there are a) not that many people using it (easy memorable names that aren’t easy to mess up still exist), and b) no bad actors squatting on typo names.

You can’t be more explicit than “build and install this folder I have”.

/shrug

> cargo install marmite

I'm not the only one (1) that believes this is wrong direction for cargo, and is abusing it's intent and purpose.

If you want to distribute an application, there are lots of ways to do that. If it's a developer tools, for developers, `cargo install` might be suitable, sure... but I think there's a fair argument to be made that if you want a tool:

    git clone foo
    cd foo
    cargo install --path .

Is an explicit, safe and less error prone way of doing it.

...and one that doesn't walk us down the road of (see the linked thread) the obvious desire people are going to have sooner or later to cache binary builds instead of building locally, and turn `cargo install` into some kind of binary application distribution application or app store.

If you don't believe me, read that thread, and the linked thread.

[1] - https://github.com/rust-lang/cargo/issues/13994#issuecomment...

While some of that might be true, explain to me why an organisation like macdonalds would allow up to 30% of such machinery to be out of working order for years and years in high value areas.

Your analysis posits what, that they’re just blatantly incompetent?

I don’t believe that.

If the economics of a working ice cream machine were that good, it would be a solved problem.

You don’t run an organisation like MacDonalds with a basic inability to maintain infrastructure. Imagine if that was true of fryers? Or microwaves? That almost never happens. It’s clearly not a technical problem.

The only way that you could justify the continued maintenance failure you see is if the benefits of a working machine were negligible at best.

Perhaps what you say might be true in general but in this specific circumstance I think you’re applying an “in general…” kind of logic (like “in general more unit tests are better…”) which doesn’t ring true here, to me.

Yeah, but the point is that MacDonalds obviously doesn’t want to sell icecream.

Maybe it’s low margin.

Maybe cleaning is expensive.

Maybe it helps having the promise of cheap ice cream but “sorry not right now” leading customers to buy other (more expensive) items.

It’s not an incompetent organisation.

If they wanted the machines to work, they would work.

They don’t.

So… if anything changes out of this, it will be to keep the status quo; people coming in and not buying cheap ice creams.

Well, perhaps anyone who’s thinking about it can do their own research in /r/vscode and read about how much people love c# dev kit.

TLDR; it’s not just me.

I’m glad you like it and have found a workflow that works for you. I think you’re crazy.

What fan boy nonsense is this? The VSC support for c# is miles behind visual studio. Rider isn’t as good either, but it’s certainly better than VSC.

Are you heavily using typescript with a bit of c# or a really tiny code base?

This comment is incomprehensible to me. Do you never refactor code? There are a lot of sophisticated things you can’t do with VSC.

It’s a great editor; but not for c#.

The benefit of using it is absolutely zero unless you’re heavily leaning into the other parts of the VSC ecosystem (like a big typescript code base).

> it’s come a really long way

So has visual studio; and it started off better, and still is.

Rider is too.

I’m happy to die on this hill; if you’re using VSC for c#, it’s because it’s free, and perhaps good enough for some things; not because it’s better than the alternatives.

Even if you’re stuck on a Mac, I can't believe you honestly find VSC an acceptable editor after using rider.

All I can say is I certainly do not agree.

All I can say is my experience is that this is the difference between wanting something to be true, and it actually being true.

> 8B models are extremely fast and cheap to run

yes.

> Combined with good RAG they can do very well.

This is simply not true. They perform at a level which is useful for simple, trivial tasks.

If you consider that 'doing well', then sure.

However, if, like the parent post, you want to be writing scripts, which is specifically what they asked... then: heck, what 8B are you using, because llama 3.1 is shit at it out of the box.

¯\_(ツ)_/¯

A working unit test can take 6 or 7 iterations with a good prompt. Forget writing logic. Creating classes? Using RAG to execute functions from a spec? Forget it.

That's not not the level that I need for an assistant.

> So 8b is really smart enough to write scripts for you?

Depends on the model, but in general, no.

...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase".

> How often does it fail?

Immediately and constantly if you ask anything hard.

An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt.

The capability compared to sonnet/4o is like a potato and a car.

Search for 'LLM Leaderboard' and you can see for yourself. The 8b models do not even rank. They're generally not capable enough to use as a self hosted assistant.

> date specified by Sandia after the VWAP Price (as defined in the Forward Purchase Agreement) for 20 trading days during a 30 consecutive trading day-period has fallen below $1.00 per share (a “VWAP Trigger Event”).

> Because a VWAP Trigger Event has occurred, Sandia has the right, but not the obligation, to accelerate the term of the Forward Purchase Agreement at any time

ie. The share price tanked below $1 and that was the condition.

Lookup ZPTA; the price has been below $1 for 6 months.

Presumably the announcements recently did not excite investors and they pulled the plug.

I mean, shoot me down here, but is it that bad of an idea?

If you're going to have an assistant or a taxi driver, and you start off at the base position of "AI is totally unreliable", then having a fully remote gig-worker remotely piloting your robot...

I mean, it doesn't seem like a massive stretch from what Uber does.

...and heck, having a 'remote robot body' is pretty cool tech.

I guess. As long as you don't use it to pretend its just AI for the meaningless purposes of generating hype about your AI that really isn't actually any good.

Everyone has a pet theory about what’s wrong with SO; but here’s the truth:

Whatever they’re doing, it isn’t working.

Blame mods. Blame AI. Blame askers… whatever man.

That is a sinking ship.

If you don’t see people complain about SO, it’s because they aren’t using it, not because they’re using the search.

Pretty hard to argue at this point that the problem is with the users being too shit to use the platform.

That’s some high level BS.

What the article actually says:

> If we go back to thinking about our Application Server models; this allows us to have a fresh process but without paying the startup costs of a new process. Essentially giving us CGI without the downsides of CGI. Or in more recent terms, serverless without cold starts. This is how Wasm is the new CGI.

^ It's not a frivolous claim.

> Wasm improves performance, makes process level security much easier, and lowers the cost of building and executing serverless functions. It can run almost any language and with module linking and interface types it lowers the latency between functions incredibly.

^ Not unreasonable.

I don't agree that its necessarily totally 'game changing', but if you read this article and you get to the end and you dont agree with:

> When you change the constraints in a system you enable things that were impossible before.

Then I'm left scratching my head what it was you actually read, or what the heck you're talking about.

> Serverless is mostly there to make money for Amazon and Azures of the world and will eventually go the way of the CGI.

There's... just no possible future, in which AWS and Azure just go away and stop selling something which is making them money, when a new technology comes along and makes it easier, safer and cheaper to it.

> I kind of like this variety of headline for it's ability to stimulate discussion but it's also nonsense. CGI can be any type of code responding to an individual web request, represented as a set of parameters. It has basically nothing to do with wasm

*shakes head sadly...*

...well, time will tell, but for alllll the naysayers, WASM is here to stay and more and more people are using it for more and more things.

Good? Bad? Dunno. ...but it certainly isn't some pointless niche tech that no one cares about is about to disappear.

CGI enabled a lot of things. WASM does too. The comparison isn't totally outrageous. It'll be fun to see where it ends up. :)

?

It’s like putting an android apk file on your website for an app.

If you’re giving it away for free to friends, sure.

If not, how do you sell it? How do you stop people stealing it? How do people find out about it?

Obviously, you the answer is usually “you write your own installer / updater” and do your own marketing.

Maybe that’s fun, and there’s something to be said for owning the entire stack… but it’s a loooot of work and probably not a reasonable financial decision to make.