crustycoder

Heartening to see so many "This is an apple, that is an orange" comments. Spot on folks!

Yes, exactly. The issue detection step isn't particularly well-suited to a LLM, as it will tell you itself. The issues can be established deterministically by examining the Cloud resource data with a set of relatively simple rules, and Prolog is ideal for that. Where a LLM comes into the picture is analysis of the symptoms - resource relationships, ages, ownership, hypothesising root causes, generating cleanup plans, management reporting and so on. The interface between Prolog and the LLM is a JSON file containing the detected symptoms which is described to the LLM by a Skill. It all works pretty well.

It's not that sophisticated ;-) There's command editing using readline, result pagination using less and I output links to the reports and the Cloud console using ANSI HTTP link escapes. Primitive but sufficient.

I think in many real-life cases it's a mix - for example impure code that deals with the outside world to set up the data needed for the pure "core" of the app to run over.

You could always read the docs?

https://www.swi-prolog.org/pldoc/man?section=clpfd-integer-a...

https://www.swi-prolog.org/pldoc/doc_for?object=!/0

I've just rolled out an internal SWI Prolog app that is similar to one linked to elsewhere in the thread [1]. We have a large Cloud estate with 10s of thousands of resources in it. Detecting unused or misconfigured resources manually isn't practical, and there are significant cost savings to be had by cleaning things up. The Prolog app reads in JSON resource snapshots, creates an in-memory database of facts from it and then applies rules to detect issues. Most of the rules are simple and the ones for detecting unused resources (3 LOC) or resources that reference other non-existent resources (7 LOC) are entirely generic. There's also link metadata that models the possibility of links existing between resource types, even if they aren't always there in practice.

There's TUI that allows querying of issues and the resource hierarchy. Issues can also be output as JSON which is fed into a LLM to produce cleanup actions and management reporting.

The Prolog app is very fast, considering what it's doing, largely because it makes heavy use of tabling so once an issue has been detected it's not recomputed when queries are made.

[1] https://web.archive.org/web/20190525163234/https://dev.to/da...

I get what he's saying but I think it's overstated. I'd categorise his list as "Things to be careful with" not "Coding horrors". For example, "The primary means to make your programs defective in this way is to use predicates like assertz/1 and retract/1" is an unqualified statement that makes it sound like you should never ever use them, and that's not the case. I have a real-life Prolog app that applies rules to facts read from JSON data files. I could do that two ways:

1) Read the JSON with Prolog (there's a library) and assertz() the facts from that, building an immutable database in the first phase before applying the rules in the second phase.

2) Externally transform the JSON into Prolog facts, load that into the app on startup and apply the same rules to it.

I agree that mutating the database in the second phase is probably a bad idea, but that's not the same as saying "assertz() always bad". I'd read his site before it appeared on HN and whilst there a lot of very good stuff on it, some of it reminds me of FP purist edicts - fine if you want to go that way and it's appropriate to your problem, but that isn't always going to be the case. That was the basis of my earlier (downvoted) "Mostly overblown" comment.

But nice to see Prolog mentioned at all on HN :-)

Mostly overblown.

Name me a new bit of tech that hasn't been hyped beyond reasonable bounds. And yes, this is one of the worst examples. But saying it doesn't have its uses isn't reasonable either.

Go figure it out, it will be a useful challenge for you.

I'd tell you to read it again, but you seem to be struggling.

I'm not going to repeat myself, I've already explained the context to you - funny how you seem to have ignored that. If you want to find out, do the experiment yourself.

You clearly missed the "The truth is somewhere in between" bit.

Picking phrases from what I said and deliberately misquoting them out of context does not make you right.

"exactly this inspection" != "what does it exactly do"

Nicely put. I haven't seen anyone say that the introspection abilities of LLMs are up to much, but claiming that it's completely impossible to get a glimpse behind the curtain is untrue.

Yes of course there's a risk it may still be incorrect but querying the LLM with the limited facilities it provides for introspection is more likely to have at least some connection with facts than the alternative that some people use, which is to simply guess as to why it produced the output it did.

If you have an alternative approach, please share.

I don't owe you anything. If you want to go find out, go do it yourself.

You could even ask a LLM to help you if you,like...

People have applied "think" to the actions of software for decades. Of course it LLM's don't "think" in the human sense, but "What the output of the model indicates in an approximate way about its current internal state" is a bit long winded...

"Not in the slightest" is an overreach, the paper the second level down from that link doesn't really support the conclusion in the blog post - the paper is much more nuanced.

Are they going to fib to you sometimes? Yes of course, but that doesn't mean there's no value in behavioural metaqueries.

Like most new tech, the discussion tends to polarise into "Best thing evah!" and "Utter shite!" The truth is somewhere in between.

> It would be interesting to see one of these evals and how it generated the score, to work out whether it is in fact arbitrary or based on some scale of points.

So go repeat the exercise yourself. I've already said this was a short-enough-to-post rollup of a much longer LLM assessment of the skills and that while most of the points were fair, some were questionable. If you were doing this "for real" you'd need to assess the full response point-by-point and decide which ones were valid.

> If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.

What on earth are you on about? The whole point of of the sentence you were replying to was that you can't blindly trust what comes out of them.

Is that based on your "deep understanding" of how LLMs work or have you actually tried it? If you watch the execution trace of a Skill in action, you can see that it's doing exactly this inspection when the skill runs - how could it possibly work any other way?

Skills are just textual instructions, LLMs are perfectly capable of spotting inconsistencies, gaps and contradictions in them. Is that sufficient to create a good skill? No, of course not, you need to actually test them. To use an analogy, asking a LLM to critique a skill is like running lint on C code first to pick up egregious problems, running testcases is vital.

They aren't arbitrary, as I said earlier I got the LLM to de a detailed analysis first, then summarise. If I was doing this "properly" for something I was doing myself I'd go through the LLM summary point by point and challenge anything I didn't think was right and fix things in the skill where I thought it was correct.

You aren't going to have much success with LLMs if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).

And yes, Skills *do* make a significant difference to performance, in exactly the same way that well written prompts do - because that's all they really are. If you just throw something at a LLM and tell it "do something with this" it will, but it probably won't be what you want and it will probably be different each time you ask.

https://agentskills.io/home

No of course you wouldn't because LLMs are nondeterministic. But the scores would likely be in the same ballpark. The scores I posted are the result of a much more detailed analysis done by the LLM, which was far too long to post. I eyeballed it, most of the points seemed fair so I asked it to summarise and convert into scores.

I wouldn't go that far but the only way I've found so far of getting a reasonable insight into why a LLM has chosen to do something is to ask it.

Because he's asking the LLM to interpret those instructions to drive his process. If the skills are poorly defined or incomplete then the process will be as well, and the LLM may misinterpret, choose to ignore, or add its own parts.

Skills are just another kind of programming, albeit at a pretty abstract level. A good initial review process for a Skill is to ask the LLM what it thinks the Skill means and where it thinks there are holes. Just writing it and then running it isn't sufficient.

Another tip is to give the Skill the same input in multiple new sessions - to stop state carryover - collect the output from each session and then feed it back into the LLM and ask it to assess where and why the output was different.

LLMs work OK for "Mostly iterative and mostly one-off" tasks like codegen, where you can effectively "review the result into existence", and that's where most of the buzz is at the moment.

Where they don't work at all well is for hands-off repeatable tasks that have to be correct each time. If you ask a LLM for advice, it will tell you that you need to bound such tasks with deterministic input contract and a deterministic output contract, and then externally validate the output for correctness. if need to do that you can probably do the whole thing old-skool with not much more effort, especially if you use a LLM to help gen the code, as above. That's not a criticism of LLMs, it's just a consequence of the way they work.

They are also prone to the most massive brain farts even in areas like coding - I asked a LLM to look for issues in some heavily multithreaded code. Its "High priority fix" for a infrequently used slow path that checked for uniqueness under a lock before creating an object was to replace that and take out a read lock, copy the entire data structure under the lock, drop the lock, check for uniqueness outside of any lock, then take a write lock and insert the new object. Of course as soon as I told it it was a dumbass it instantly agreed, but if I'd told it to JFDI its suggestions it would have changed correct code into badly broken code.

Like anything else that's new in the IT world, a useful tool that's over-hyped as sweeping awsy everything that came before it and that's gleefully jumped on by PHBs as a reason to get rid of those annoying humans. Things will settle down eventually and it will find its place. I'm just thankful I'm in the run up (down?) to retirement ;-)

He's also missed a major step, which is to feed your skill into the LLM and ask it to critique it - after all, it's the LLM that's going to act on it, so asking it to assess first is kinda important. I've done that for his skills, here's the assessment:

==========

  Bottom line
  Against the agentskills.io guidance, they look more like workflow specs than polished agent skills.
  The largest gap is not correctness. It is skill design discipline:

  # stronger descriptions,
  # lighter defaults,
  # less mandatory process,
  # better degraded-mode handling,
  # clearer evidence that the skills were refined through trigger/output evals.

  Skill           Score/10
  write-a-prd          5.4
  prd-to-issues        6.8
  issues-to-tasks      6.0
  code-review          7.6
  final-audit          6.3

==========

LLM metaprogramming is extremely important, I've just finished a LLM-assisted design doc authoring session where the recommendations of the LLM are "Don't use a LLM for that part, it won't be reliable enough".

So it's the relative and not the absolute diff that matters - thanks.

A timely link - I've just spent the last week failing to get a ChatGPT Skill to produce a reproducible management reporting workflow. I've figured out why and this article pretty much confirms my conclusions about the strengths & weaknesses of "pure" LLMS, and how to work around them. This article is for a slightly different problem domain, but the general problems and architecture needed to address them seem very similar.