Hacker News Reader

Comment by JimDabell | Hacker News Reader

This is not how these kinds of things should be designed for the web.

Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785:

https://datatracker.ietf.org/doc/html/rfc5785

Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for.

I’m not sure making it easier to consume content is something that is needed. I think it might be more useful to define script type=llm that would expose function calling to LLMs embedded in browsers.

View on HN

I think the whole idea of extra instructions required for LLMs is unnecessary. A decent LLM should be able to handle browsing the site, if needed it can use the sitemap. It can hopefully also figure out what the various sections are about.

I agree. There are enough standard places to put metadata in a website.

But why would we waste that many tokens on objects that LLms dont care about?

Humans appreciate beauty. LLMs do not. Why are we wasting effort?

> Humans appreciate beauty

Some humans do for websites; I personally couldn't care less and I find it often just annoying / in the way. I wish all sites where just black on white or the reverse and with clear interaction elements (including for saas sites). I welcome the near future where I can say; 'show me all important sentry issues, ah yes, make an issue in github to to fix this one and just make the rest resolved' instead of having to click through a myriad of useless and often confusing 'UX' and 'beauty' just to do things.

Non saas sites I just visit to read so I immediately ram the reader-mode button which shows the site indeed as I want.

I wish that the website would provide the content, and my user agent would provide my preferred styling e.g. fonts, colours, line spacing, etc.

Anyone else remember when websites had sitemaps? I miss those days.

You mean sitemaps for humans?

  > A decent LLM should be able to handle browsing the site

A decent web dev would design the site such that it is easy enough to parse for humans, that a machine could do it too.

If only this RFC was well-known among the people who actually put stuff out on the Web.

How about an LLM agent which automatically finds inconsistencies in RFCs.

If only that RFC didn't make it a hidden directory.

I can think of a dozen reasons why hiding that folder is a horrible idea, and not a single one for why it would be a good thing to do.

The reason is because it's supposed to be a standard folder that isn't in use accidentally for other purposes.

It's exceedingly unlikely that a website is going to just happen to make content available in a hidden directory path without it being created by automated tooling (which would likely be aware of such standards).

The entire point is to avoid adopting a path that people already publicly use for something else. A hidden directory is the best way to do that.

Also "well-known" was always such an awkward name to me.

It's a joke about things being described as needing to be put at "well known" paths.

> If only that RFC didn't make it a hidden directory.

There's zero guidance on configuring how URIs under `/.well-known/` should be served at all, is there? They just reserve/sandbox the initial path component for the URI schemes which support it. That's it. It's the developers' choice to implement it as a directory - hidden or otherwise - on a filesystem; neither RFC says they SHOULD or MUST be served in such a way.

(The updated RFC says "e.g., on a filesystem" in section 4.1, and mentions directories in section 4.4 in a way that, to my eyes, pretty much recommends against making it hidden)

A URL path is not a directory path; there is no reason to assume that a path must be served by a directory by the same name. I mean, do you assume that there somewhere exists an actual machine with its Unix hostname set to “news.ycombinator.com”?

> there is no reason to assume that a path must be served by a directory by the same name

Other than static sites, sure.

Even in static sites, rewrite rules are a thing.

Even static sites might be served out of a simple hash table in RAM. A URL path is not a file path: <https://www.w3.org/Provider/Style/URI>

OpenAI was good about using well known for plugins

Any example?

Look in the URL field in all the files in https://github.com/cfortuner/wellknown/tree/main/plugins

Eg: https://www.wolframalpha.com/.well-known/apispec.json

they did a good attempt but I think they just realized (at same pace as people like me) that you can just grab all this shit from OpenAPI and not worry about defining custom format

I think it is. Lots of actually used standards use it. I see it in the wild all the time.

Certainly more well used than content negotiation which grandparent also mentioned.

It's very broadly used.

For example:

- Apple uses it for their app to website association files

- OpenID Connect uses it for connection discovery

- security.txt is usually served from .well-known

- JSON Web Tokens uses it for Web Key Sets to verify public keys

- LetsEncrypt uses it for its ACME HTTPS verification protocol.

There's probably more, but these are ones I've personally used.

Fwiw the active RFC is 8615 as RFC5785 is obsolete.

https://datatracker.ietf.org/doc/html/rfc8615

Having two different RFCs on this is clearly causing confusion!

I think I'll create a new RFC to supersede both to clear up the situation.

OP replied to this complaint in a top-level comment:

https://news.ycombinator.com/item?id=41442092

They're correct that the RFC technically requires registration, and looking through the existing list of contact information for registrations I'd be likewise intimidated to attempt to register something that's experimental.

It's based on the series of robots.txt, humans.txt, security.txt, etc

Yes, and robots.txt was reasonable because it was created in the early days of the web. But the others don’t have any excuse (and they were warned about it as soon as they were announced and ignored people).

FWIW, security.txt is placed in .well-known/

What if each LLM had to register and tell you when/where/what it scraped/ingested from your site/page/url? And you could look at whatever your LLM trigger log looks like, and have a DELETE_ME link. (right_to_be_un_vectorized)

Reply on news.ycombinator.com