Show HN: QUALITY.md – open format/specification, agent skill, and CLI | Hacker News Reader

Show HN: QUALITY.md – open format/specification, agent skill, and CLI

getquality.md

29 pointscraigsmitham29 comments

Hello all, I created QUALITY.md to help build a holistic quality evaluation process for my projects. Turns out it's also ideal for loop engineering. I'm hoping this provides a valuable contribution to the conversation around quality and craft and having AI help us in the effort. I hope to shift the mindset from a reactive/review/repair mindset to a proactive care mindset.

Give it a go. I look forward to your thoughts/comments/feedback!

Website: https://getquality.md GitHub: https://github.com/qualitymd/quality.md

Open articleView on HN

Is this really where we've landed? I refuse to believe that any of this markdown insanity will continue indefinitely.

It's insane to me that the "fix" for AI errors is adding more "PLEASE PLEASE DO BETTER" to the prompt

If it makes you feel any better, the Markdown part is optional (and has no semantics). Somehow it feels about right that the Markdown file can actually just be a YAML file with the wrong extension.

(Actually, to be more specific, a YAML file with no directives, explicitly-signalled start-of-document-content, and followed by a second null document. I will note that frontmatter syntax is not specified; the non-normative Appendix B is the only place that suggests it means prefix and suffix --- lines. And no, frontmatter is not part of Markdown, or CommonMark, and is in fact incompatible with both. And it’s invalid YAML too, the end-of-frontmatter line should be ... to indicate end of document without starting a new document.)

it's looking like llms are interpreters, and markdown plus english text is the language of choice to run non deterministic programs on it

> it's looking like llms are interpreters, and markdown plus english text is the language of choice to run non deterministic programs on it

That's actually a pretty good clear way of putting it for the typical nerdy "programmer minded" individual.

QUALITY.md feels similar to CONSTITUTION.md

Looks like unless something better comes up, we'll be stuck with it for a while.

I find markdown useful for repo-specific conventions, especially skills.

> I find text useful for repo-specific conventions, especially skills.

sure, but structured text like markdown is even more useful, since humans can parse and understand it as easily as skills can

> sure, but structured text like markdown is even more useful, since humans can parse and understand it as easily as skills can

And there are a number of nice viewers / editors either already installed or easily installable on most operating systems to view / edit Markdown in a "beautified" fully rendered form, on both CLI and GUI interfaces; and since most (all these days?) LLMs also "understand" Markdown formatting pretty-much natively, you can easily emphasize certain points to add "weight" to them in the LLMs' "mind" / "thinking" (calculation of statistical token probabilities) process. Plaintext without Markdown is just ... well ... plain. :)

I thought the same about Yaml and Kubernetes/Helm…

it is until we define real consistent deterministic gates and protocols. It really is a symptom of the lack of concerted effort. Everyone has a personal preference on how to shove the context and most of them are just "here's some good text I've found to work in my context"

> define real consistent deterministic gates and protocols

I've been experimenting with doing kinda exactly that with the "routing layer" / "harness" level of things, before the "main" LLM itself ever receives the user's input, by getting "user intent" (as a little JSON packet) really quickly from an ultra-lightweight model first and deciding from there in deterministic code what "context" to inject into the user message template, which system prompt to use, and which model to route the assembled context "packet" to for the final response. These LLMs really are fun to play with once you get a feel for which ones do what well, and where each falls short so you can use them each around their individual strengths. :)

It already refuses to read the AGENT.md/CLAUDE.md files, what's the point of giving it even more markdown it won't read until you yell at it.

The one thing I do not understand is that here you say:

"Ensure stakeholders are aligned on what matters most and why"

But it is instructions for LLMs, right? A way to describe something that the humans know and the LLMs don't.

LLMs literally cannot be stakeholders, by definition.

THe problem is that humans often don't know - this is as much about encouraging getting the humans aligned as the agents. Completely agree agents really aren't stakeholders. Fine point. I'll update description to clarify ... thank you!

Not OP, but it seems to me the idea is that stakeholders can collaborate and come to consensus on the contents of QUALITY.md.

Here's the question I ask about every project that claims to make a LLMs output so much better: If it works so well then why would the model provider not just put it in the system prompt? Or in the case of interactive skills, why would Claude Code/Codex not make it a core part of the product?

On top of that, if your magic markdown file really does work then where's the evidence showing that? These projects never include even basic benchmarks. At best they're entirely vibe based, however more often they're completely untested. Give us a proper benchmark, even a single prompt and it's output with and without your skill in use would be better than every other project out there.

No magic. QUALITY.md describes what is unique and valueble to your proejct context that model providers won't have insight into.

You can already put these into the AGENTS.md(or CLAUDE.md) or if it’s too big, you can put it into a SKILL — no need to reinvent the wheel.

I'm less interested in this than in what people are willing to aggressively trade off against in order to get the stuff they truly care about.

For example, readability. Where are the developers out there saying "I am very willing to sacrifice a lot of readability to get even a small improvement on e.g. abstraction cleanliness", and sticking with it?

Or "performance can take a huge hit at the cost of being dead easy to read and reason about". Coming up with a list of abstractly good-sounding qualities is just prosocial signaling without knowing what you're willing to sacrifice. There should be a FUCKIT.md that enumerates these.

OP here. You're spot on. Trade-offs matter. The trade-offs are implied by the selection of what quality factors/attributes are selected and their requirements. A statement like "performance can take a huge hit at the cost of being dead easy to read and reason about" can sit right there in the QUALITY.md as a comment or in the markdown body.

Whats the revenue model for this NBPaaS? (No Bugs Please As A Service)

No commercial offering associated with this effort. But a lot of potential for others to incorporate the QUALITY.md standard into products that assess/evaluate quality at varying levels of a loop stack.

The agent skill/CLI that's provided generate a quality evaluation report with recommendations for handoff (ideal for loop engineering) is just one example of how the QUALITY.md file can be used.

It's easy to imagine a SaaS that does the same that provides better eval, reporting, and integration capabilities.

This is perfectly encapsulated in xkcd's "Standards" strip [https://xkcd.com/927/].

I'd really like feedback on the standard/specification. In short, it defines a quality model of quality factors/characteristics (which you can define as security, reliability, etc), requirements (how those qualities are assessed), a customizable rating scale, and "areas" to have different attributes/requirements for different areas of your project (e.g. frontend/backend, tests, specs, etc). That's basically it - and it follows a consistent pattern of how quality models have been practiced for decades. They are simple and powerful, but - until AI - kind of a pain/toil to get started with. Simple but not easy - until now. QUALITY.md + AI makes it easy. However, you still have to put in the work/care/attention to what goes into your QUALITY.md so you can get maximum leverage from it.

So is every proposal to standardize new things, and eventually the cream rises to the top, even though some people are perfectly happy sticking with milk.

Useful Nice

Pure slop.

What?

Reply on news.ycombinator.com