Show HN: DNS-based alternative to Semantic Web – 23M data points now in DNS | Hacker News Reader

Show HN: DNS-based alternative to Semantic Web – 23M data points now in DNS

25 pointselliottinvent18 comments

We’ve just launched NUM [1] in the UK, it’s an alternative to the semantic web [2] but based in DNS. We’ve published 23 million data points about 4.8m UK businesses.

We had some really interesting HN feedback when we previously posted about one of our example apps [3] so keen to hear what the community has to say this time round.

1. https://www.num.uk

2. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-23m-data-points-to-dns

3. https://news.ycombinator.com/item?id=27598164

View on HN

Is a semantic web that failed because it was too complicated for businesses to setup established by letting them setup DNS records?

I’m working on making the web semantic (on your search query) by using large language models which is more equipped for reality since businesses can keep doing their websites as they want

Thanks for taking a look. In my opinion, the methods of adoption of the semantic web (html tags) weren’t that complicated. The different variations and possibilities of how even basic data could be described is what made it complicated for those storing data and developers building apps on top of it.

With NUM, we’re not asking people to store this data in DNS because it’s simpler (although it does separate it out from your website which has benefits in my opinion), the point of storing it in DNS is that we can pre-populate it at scale and offer basic tools to manage this data safely in DNS.

Your project sounds cool. Is it searchable through a centralised database or some other way? Google have done the centralised model and that’s what we’re trying to offer an alternative to.

It’s agent based - while currently we store the results of a search query (to present them to you), with our tool an agent will go and browse the web for you, looking up what you want.

The project is focused on repetitive tasks on the internet. It’s not intended to replace one off searches but rather to build datasets/tables/graphs on your query.

One off search: best pizza in Berlin

Our usecase: show me the best rated salami pizzas sorted by price in Berlin

Whereas you’d visit each website with a one off search and go through their menu (and TripAdvisor ratings) we’d serve you a table/graph/written summary…

Where does it get a list of domains to crawl that match a given query?

Thanks! You may signup on [1] but you’ll need to wait still to use it.

On NUM: I like the idea of building open graphs, but to move the needle it gotta be easier for business owners to provide that information. Lots of orgs use let’s encrypt some of them update their dns records every 90 days with a dns challenge. If there was to be some way to use a process like this (with a companion tool) to get business info into NUM records it would help adoption. I find the idea of calling a domain, sending money to a domain compelling I have to say.

[1] https://get-sentinel.io

Thanks for the feedback.

It works for 90% of UK businesses already, because we’ve pre-populated the DNS with their phone numbers, addresses, social media.

So they don’t need to adopt it, for it to work. They can if they want to override the pre-populated data but they can also do that with our simple tools.

See https://companydirectory.uk for an example app powered by NUM.

I was totally aware that you’ll be asking this question and it was intentionally not said, sorry :)

EDIT:

it’s due to the workings of the agent that we do not need to index the internet to use it

Ok, good luck! Hope to hear about it on here some day.

Looking at the NUM protocol, this looks a lot like X500 type addressing format glued into a DNS record.

Conceptually the idea had legs, but I can't see wholesale adoption of it. Whilst all (most?) of the records are hosted at Num.net it doesn't meet it's promise of decentralised.

It feels like you need a few big-name companies to stand up and endorse you, and set their own NUM entries on their DNS.

Thanks for taking a look.

We don't promote NUM as decentralised, but it has that potential with enough independent adoption.

We talk about NUM as an alternative to centralised APIs for two reasons: (a) it can be adopted independently; (b) there are no rate limits or restrictions, which we see as the big problems with APIs.

I agree, some big endorsements would be good but I don't think that'll drive adoption on the domain side. The point of num.net is that we don't need wholesale adoption on the domain side, we need adoption on the developer side (and by extension users) to drive adoption on the domain side – either by people claiming their records on num.net or publishing independent records.

I think this comment highlighted the issue best (and it's not possible to address really): https://news.ycombinator.com/item?id=27632195

I was initially curious, then cautious, then a bit disappointed as I dug into how you're doing what you're doing.

What would make this data truly decentralized and "DNS-based" is if the data was published to the companies' respective domains as extra standardized records (and even thinking about it for 30 seconds I can see the problems with that). As it stands, everything is hosted on num.net.

It's literally just a JSON API with extra steps.

Now don't get me wrong; you've opened my mind to the fact that yes, hosting certain things over DNS has use cases. I think you built something neat, but I too am concerned that you're directly contributing to a tragedy of the commons effect. You're piggy-backing off decentralized infrastructure that gets to be free because its costs are globally minimal.

In other words, you're paying for your infrastructure by fishing for pennies in the Trevi fountain.

Thanks for the feedback. There's some replies on that comment that I won't repeat here.

I find the disappointment fascinating and I guess it's maybe a network traffic purist view point?

> What would make this data truly decentralized and "DNS-based" is if the data was published to the companies' respective domains as extra standardized records (and even thinking about it for 30 seconds I can see the problems with that). As it stands, everything is hosted on num.net.

Surely you can see that there's zero chance of getting to this truly decentralised way of doing things without some kind of fallback (See Semantic Web vs Google). NUM enables organisations to do exactly what you want – publish this data to their own DNS and that overrides data in num.net

We package it up as a JSON API-like library because that's what developers are used to, but under the hood it's nothing like an API – requests aren't tracked, rate limited or restricted.

> In other words, you're paying for your infrastructure by fishing for pennies in the Trevi fountain.

This is a colourful description!

So you're saying that you reinvented the WHOIS protocol?

Why didn't you go for RDAP, the new protocol that AfriNIC, APNIC, RIPE and even LACNIC are adopting?

[1] https://www.icann.org/rdap

No, not saying that.

As I’m sure you know, WHOIS and RDAP are about domain registrant data supplied by registrars and registries.

We didn’t do this in RDAP because: a) we’re not a registrar or registry; b) RDAP has strict limits designed specifically to prevent exactly what we’re trying to enable - unlimited and unrestricted access to data for developers; c) NUM is much broader than just contact data

> unlimited and unrestricted access to data for developers

I'm not sure what you're trying to argument with here. Why is RDAP limited if anybody can just use a client and any server on the internet that fulfills the same spec? [1]

The argument works both ways, because as of now, your solution is literally "n (servers) = 1", therefore it's the maximum restricted access possible. Arguing that an IETF- and ICANN-backed RFC is limited and restricted; whereas you are using a proprietary NUM protocol on top of DNS, an RFC by the IETF, is ridiculous.

Don't you realize that you're trying to push a proprietary protocol here?

RDAP is an IETF-RFC, so I'd argue that adoption is much more likely; as it is also backed by ICANN, RIPE and all internet registries.

You're just using DNS as a generic database here, and I'd argue that the DNS protocol wasn't designed for the purpose that the NUM protocol seems to have as its goals; given the overhead of schema validation that each client has to do for themselves. [2]

> NUM is much broader than just contact data

RDAP supports different schemas for entries, objects and models; just as any other database. Add a new rtype="whatever" and define its properties, and you're set to go.

Honestly in times of GraphQL making predictable queries as easy and as built-in as possible; the NUM protocol feels like a huge setback in terms of transport encryption, integrity, predictability of state complexity and confidentiality.

[1] https://datatracker.ietf.org/doc/html/rfc7482

[2] https://www.numprotocol.com/specification

Ah, I might have misunderstood your previous comment about RDAP - I thought you were asking why we didn’t fetch this data from registrar RDAP implementations, so my answer was in response to that.

You may have been suggesting that we serve out our data from num.net using RDAP instead of DNS. In answer to that: NUM is a DNS-based protocol because every online organisation has access to DNS - it’s a standardised way for them to store and serve out data, they’ve already got it and already pay for it.

Thanks for the detailed technical feedback, it's why I'm here.

> Why is RDAP limited if anybody can just use a client and any server on the internet that fulfils the same spec?

RDAP as a technology and RFC isn't limited or restricted of course, just like restful APIs as a technology are not limited. However, the implementation of the technology is limited. The servers offered by registrars and registries will be limited and the access and use of data restricted.

We promote NUM as unlimited and unrestricted because NUM records (whether in authoritative nameservers or under num.net) are hosted in DNS and requests can not easily be tracked or rate limited.

> I'd argue that the DNS protocol wasn't designed for the purpose that the NUM protocol seems to have as its goals

I disagree, we're using vanilla DNS TXT records which have existed in DNS since '94 [1]: "This paper proposes the use of the DNS TXT resource record [...] to contain new types of information."

Most NUM Records can be transported over UDP, even under the original 512b limit. A NUM record containing let's say 20 typical key value pairs is smaller than a DNSSECd A record and smaller than millions of DNS TXT records at a domain apex (dig target.com TXT).

> given the overhead of schema validation that each client has to do for themselves

Validation is minimal and is dealt with at a library level, as it will be with RDAP or any other API. The fact that mainstream data serialisation formats (e.g. JSON, YAML, etc) are not designed for DNS, does not mean that DNS is not designed to store serialised data. We created MODL [2] to solve this problem.

To demonstrate low overhead, we have a heavy example implementation that fetches 62 NUM records, validates them and displays them in 1.6 seconds [2].

> Honestly in times of GraphQL making predictable queries as easy and as built-in as possible...

This data is already centralised in graphs by Facebook, Google and more. There was an effort to decentralise it with the Semantic Web. We're offering an alternative to both, where users can retain their privacy, developers can actually build scalable, profitable products (as opposed to using Google / Facebook APIs) and the data is actually available (as opposed to SemWeb).

> ... the NUM protocol feels like a huge setback in terms of transport encryption, integrity, predictability of state complexity and confidentiality.

These seem like complaints about DNS which are addressed by DNSSEC, DNS over TLS (DOT), DNS over HTTPS (DOH), DPRIVE and Oblivious DNS-over-HTTPS (ODOH).

Thanks again for your thoughts, happy to dive deeper into any of this.

1. https://datatracker.ietf.org/doc/html/rfc1464

2. https://companydirectory.uk/barclays.co.uk/contact-informati...

3. https://www.modl.uk

Hey elliot I like what you're doing and would like to share a frw thoughts. Feel free to each out, email at my bio.

Reply on news.ycombinator.com