Interoperability Can Save the Open Web (2023)
spectrum.ieee.org
spectrum.ieee.org
Anthropic is now racing to close this gap because they realize there's no lock-in. If the product is just .md files with hierarchy, you can drop any harness and intelligence on top of it. It is interoperable by default, possibly not even by intention.
We should do everything possible to stop the great lock-in that they'll attempt in the next 18 months.
That might mean leaning into SQLite or some other open format. My own thought is that a graph-like structure of documents will be ultimately more valuable than either a tree-like structure or a database-like structure.
But if a proprietary implementation of whatever usurps structured text in a file-system becomes popular, that company will have significant leverage.
But I also agree that this has limitations. If I were to challenge you and say that Obsidian solves this, or gets close to solving it, what gaps would you say were left unfulfilled?
I am because I'm working on something in this space and your comment touches on my three main ingredients: text files, SQLite, and graphs.
Let's imagine a .zip file format that combines:
- A directory of .md and/or .xml files (there is more to structured text than markdown)
- A sqlite db in the root for some metadata
- A graph representation (RDF, triples, whatever) of the files
That is directionally what I am talking about, but with a standard spec so that someone receiving a project bundle knows how to parse it all.
You can still have all of the Obsidian stuff over the contents of the directory. But you can also store much more interesting meta-data in the db that may not be appropriate to put in any individual file (or in some kind of pure data .md file). It is something that can be sent over the network a bit more conveniently than a directory. And if you standardize the db schema and the graph you can have immediate interoperability between systems.
In fact, given advanced enough LLMs, you could just have an README.md or something in the root that sketches out the directory structure and files, the db schema for the metadata, and then just expect that the agent, knowing how to load and query a SQLite db, will be able to work with it. But I think there is value in having some loose standards around the db schema, some YAML frontmatter conventions, some well known graph representation, etc.
> When Facebook was born, it gave people who were already using MySpace a tool that would pretend to be you and log into MySpace, collect all the messages that your friends had left for you, and put them in your Facebook inbox. You could reply to them there, and it would send them back to your MySpace outbox, so your friends would see them. And that was what allowed Facebook to take so many users from MySpace so quickly. This is what interoperability is about. But if you tried to do that today, Facebook would use laws that were either enforced differently or did not even exist at the time of MySpace, to ruin you.
> If we were to restore this “noble ancient art” of technological interoperability, the users who are so obviously discontent with the platforms they use would consider the costs low enough to leave and join better spaces. In turn, the companies would be smaller, would pay more attention to user satisfaction, and could not push around the governments that tried to hold them to account.
https://www.greeneuropeanjournal.eu/cory-doctorow-how-to-tak...
It's mostly about leverage to create a stronger position for negotiations.
You might also define "exists" in some sort of way that makes sense. And you can also realize that payers are encroaching on every aspect of interoperability data exchange.
But there is a real health element to it. Although I perfectly agree that standards are good for the consumer, the incentives here are not as strong.
This is an implementation of something called MUMPS, which is apparently some US system that is very arcane but widely used.
Again, I'm not an expert on this topic, but it indeed seems like standards, API's, file formats and whatnot would be keys to a system where decoupled components can be evolved step-by-step over time instead of the current system which seems to be a humongous monolith.
Even if you don't care about this stuff, FHIR is definitely worth investigating.
I am deeply in self-host. For the self-host to succeed it needs to be better, unregulated, and free. It needs to be easily distributed. The data should be easily distributed. Import and export should be fast and easy.
That is why most of my programs use JSONs that are human readable, or use SQLite tables that are just copy-paste away.
I am from Poland. My ancestors were able to survive by hiding, and by fighting small partisan battles. My idea of software is "partisan". It battles big tech in small, distributed ways.
I am not sure, but I think what I said is similar to interoperability.
If you're only talking specifically about your program that no one else has access to, I don't think there is any battle? Do whatever you want, no one cares nor would even know about it.
If you're talking about making software available for others, for free and open source, I also don't think there is any battles to be won here.
When people talk about the web not being open, or "age checks" and "backdoors" and so on, they're mainly talking about for-profit platforms, that let users "use" their platform in exchange for something. These probably shouldn't be "do whatever you want, consequences be damned" but instead have some sort of checks against them, so we don't end up letting the business-people rush towards building torment nexuses.
Even if platforms has to have age checks, encryption backdoors and a whole slew of other "bad stuff" or just "annoying stuff", I don't think the self-hosted ecosystem has much to worry about, we all run software "without warranties" already, and plenty of the stuff I'm running at home I've written myself, of course I won't care about age checks or whatever, even if it was regulated to be forced.
If you host friends over for dinner at your house a lot, nobody will ever say you are subject to the same rules as a restaurant. You start letting other people host dinners at your house, and things could change. You start letting people solicit your place for paid dinners, similar outcome. Do it once, nobody will probably know or care. Continue to do it at scale, though, and I don't know why you would expect to not be subject to regulations.
As has been pointed out elsewhere, if you want to take the effort to connect and verify the different parties that are going to communicate with your server, you are almost certainly going to remain free to do so.
Do I think there are probably some concerning ways those burdens can be placed on folks? Certainly. But we already require inspections and other similar activities for things that individuals can do at home without an inspection. See the food industry.
This is the fraud every would-be censor perpetrates to establish their chokepoint. First, invent allegedly "neutral" rules that only large entities can comply with, causing only large entities to remain. Then lean on the large entities to censor whatever you want in exchange for political favors or lack of enforcement of other laws.
> You couldn't claim "free speech" and build a radio tower that transmits long distances, as an easy example.
Which is another great example of them doing the thing. The government couldn't spare a single frequency for unlicensed long-distance directional radio communications?
Moreover, the excuse for censoring the airwaves is that there is finite capacity in a broadcast medium, so how is that supposed to apply to a unicast service whose transfer capacity can be increased without bound by running more fiber?
> For that matter, you can't claim free speech to allow concerts at your house.
So if the government wants to declare that you meeting with two other people for the purpose of conveying information to them is a "concert" and prohibit it from any place that isn't a "concert hall" (which is prohibitively expensive for you to own), that seems fine to you?
> As has been pointed out elsewhere, if you want to take the effort to connect and verify the different parties that are going to communicate with your server, you are almost certainly going to remain free to do so.
All you have to do is the thing which is morally and economically unsound.
> But we already require inspections and other similar activities for things that individuals can do at home without an inspection.
Except that now you want to do it even when they are doing it at home.
What am I supposed to do other than point that out?
You ask why we couldn't have a single frequency for long-distance radio communication. Which is either willfully ignoring all of the frequencies that we can use for directional communication. Or shockingly ignorant over how many people could use a "single frequency" for anything long distance. (Indeed, this was the wishcasting that I found off putting.)
You then reduce to absurd the idea that you could have a concert between two other people. Which, I feel fairly confident saying nobody cares if you are running small two person concerts at your house. Unless you are violating noise concerns of your direct neighbors.
And then, your entire "they are doing this in their home" argument ignores that the problem here is open solicitation across the internet. If you are setting up a LAN equivalent where only direct friends ever gain access, this is just not relevant here.
I was talking about creating/running software for yourself, in a self-hosted scenario, not just "I run the software, but it's for others" but really "I run software and it's for myself and/or my family, no one else"./
What I'm saying in the previous comment is that regulations requiring "Age checks, encryption backdoors and other bad/annoying stuff" also apply to small hosts and can be abused like DMCA (unless you are hosting on tor/i2p with good opsec).
It's this notion that any regulation is good because it's done on a "big bad public company" that is at the heart of what I disagree with. At what point do you become a "big bad company"? Does anna's archive count? they accept donations. It just doesn't seem like a fleshed-out worldview.
Yes, just like even if it's just you and your bakery, you still have to follow a bunch of health and food safety regulations, as you're providing something people can be harmed by.
I don't think it's so out of this world to require similar things for platforms and services available to the public on the internet. Although I wouldn't maybe say it should be straight up illegal, I wouldn't mind more research and understanding of how we could prevent the biggest harms, without infringing on what people do in private. But then is a self-hosted Mastodon instance connected to the public internet and other instances in public or in private? Personally I'd lean towards the first.
Revenue exceeds 0.1% of US GDP or market share exceeds 10% of their own market.
Can I do it on my phone?
You can't just run programs on your phone. You have to run apps, which require approved by the government and the company that made the phone, which tacks on additional fees as well. The phone also has constant cellular/GPS/wifi/bt-mesh location tracking, and it can never be completely turned off by the user without destroying the phone because even the batteries are glued in.
It's basicially the perfect slave device for your average goy. And everyone will need one to to access your bank account, recieve insecure SMS authentication, talk to other NPCs, and generally participate in the neo-economy.
If you don't think this is right, you are literally going to empty the bank account of my dumb ass grandma who can't stop installing malware, and in every way is better served by a flip phone from the early 2000's.
Then why are you demanding that everyone else's mobile computers have to be locked down instead of demanding that somebody make a mobile phone that only makes phone calls?
All my programs and data are open. It is something that anybody can pick up, and use as they wish
- https://github.com/rumca-js/Internet-Places-Database - domains I found
- https://github.com/rumca-js/Internet-feeds - feeds I found
- https://github.com/rumca-js/RSS-Link-Database-2026 - news from 2026
- https://github.com/rumca-js/RSS-Link-Database-2025 - news from 2025, etc.
Does make any change? I don't know. I run web crawlers. It is interesting for me to see what my crawlers pick up from the Internet. It did change my life, these project changed how I see the Internet. More pro-activly.
I think there are many projects which can be useful for niche groups. I suppose I have 390 stars on one repo. I hope at least my projects were useful for them. That is a hopeful thought.
Support https://edri.org and https://noyb.eu
For me the freedom to own my computer means I can run any software I want on it.
Self hosting is predicated on some openness of computing in general. Interestingly it still does not practically allow you to use certain services like Google Maps, where even if the end user has great benefit, they get it for free because they give back their data.
Or maybe Comaps, https://news.ycombinator.com/item?id=43961908
1) Use HTTP (secure is not the way to decentralize).
2) Selfhost DNS server (hard to scale in practice).
3) Selfhost SMTP server (also tricky).
4) Know and backup your router (dd-wrt or iptables).
JSON over HTTP is the way.
XML is not bad for certain things too; even if I understand the legacy of abuse.
Also if people need more food for (decentralized) thought:
This doesn't seem like useful advice. If you're going to use HTTP at all there is essentially zero practical advantage in not using Let's Encrypt.
The better alternative would be to use new protocols that support alternative methods of key distribution (e.g. QR codes, trust on first use) instead of none.
> Selfhost DNS server (hard to scale in practice).
This is actually very easy to do.
DNS is easy for yourself, but if you host it for others (1000+ of people) and it needs to have all domains in the world, then it becomes a struggle.
DNS can answer thousands of queries per second on a Raspberry Pi and crazy numbers on a single piece of old server hardware that costs less than $500.
If your DNS port is closed by your ISP, you can't have people use your DNS server from the outside and then you need Google or Amazon which are not decentralized.
Also to be selfhosted you can't just forward what root DNS servers say, you need to store all domains and their IPs in a huge database.
The root certificates are pretty decentralized. There isn't just one and you can use whichever one you like for your certificate. The browsers or other clients then themselves choose which roots to trust.
The main thing that isn't very decentralized here is Google/Chrome being the one to de facto choose who gets to be root CA for the web, but then it seems like your beef should be with people using Chrome rather than people using Let's Encrypt.
> If your DNS port is closed by your ISP, you can't have people use your DNS server from the outside and then you need Google or Amazon which are not decentralized.
It's pretty uncommon for ISPs to close the DNS port and even if they did, you could then use any VPS on any hosting provider.
> Also to be selfhosted you can't just forward what root DNS servers say, you need to store all domains and their IPs in a huge database.
I suspect you're not familiar with how DNS works.
Authoritative DNS servers are only required to have a database of their own domains. If your personal domain is example.com then you only need to store the DNS records for example.com. Even if you were hosting a thousand personal domains, the database would generally be measured in megabytes.
Recursive DNS servers (like 1.1.1.1 or 8.8.8.8) aren't strictly required to store anything except for the root hints file, which is tiny. In practice they will cache responses to queries for the TTL (typically up to a day) so they can answer queries from the cache instead of needing to make another recursive query for each client request, but they aren't required to cache any specific number of records. A lot of DNS caches are designed to have a fixed-sized cache and LRU evict records when it gets full. A recursive DNS server with a 1GB cache will have reasonable performance even under high load because the most commonly accessed records will be in it and the least commonly accessed records are likely to have expired before they're requested again anyway. A much larger cache gets you only a small performance improvement.
DNS records are small so storing a very large number of them can be done on a machine with few resources. A DNS RRset is usually going to be under 100 bytes. You can fit tens of millions of them in RAM on a 4GB Raspberry Pi.
You may not store your personal data with the surveillance companies computers, but your friends will store your data there. Your mail, your chats, your contact information. Without even asking you, because our society has during the past two decades completely normalized this. Your local hospital and your police force does this too, at least to some extent, even where legislation disallows it.
The largest companies on our planet is now aligned with the largest governments. I am not sure those alive today will own our data again in our lifetimes.
In a perhaps desperate hope for ironic turnaround, I now put my hope with laws such as Chat Control and Age Verification. It is a hard sell with techies and Internet aligned people these days, so I tend to stay low about it. But my feeling is that legislative overreach may be one of very few things that can normalize self hosting again. Because in practice, these laws are only enforcable for companies. If I chat with my friend by way of my our own computers only, nobody complains and everyone is happy. I will not fight for these laws but I will be honestly happy when they pass.
In fact, having given it a bit of thought, I do not feel it is entirely unreasonable for the surveillance companies to take responsibility for things like CSAM and encouragement to genocide. They do own our data, after all, and they already police it to influence elections and shape consumer trends, to take two well-documented examples, so why should they not also police it in legal ways?
Those were my soapbox minutes for today. I will not take questions about how legislative overreach will create moats and economic incentives for centralization, benefit Facebook and hurt Tik Tok, because that already happened and is too late to change now. The outcome of that particular race between competing evils changes nothing.
[1] Gonggrijp, 2005. https://media.ccc.de/v/22C3-920-en-we_lost_the_war
Image you are a 1960s household and RCA tries to sell you a TV to only watch ABC and Zenith has a TV to only watch CBS. 60 years later linear TV is unwatchable by normal humans IMO. It's not like "let advertising pay for this" enshittifying an entire industry hasn't happened before.
Interoperability can save the open web - https://news.ycombinator.com/item?id=37399799 - Sept 2023 (97 comments)
I don't even understand what the first point is even proposing, legislating use cases now? It's gotta be some dog whistle about Twitter pushing "fascism" and entirely hinges on the weasel word "overweight"
The second statement just seems like a category error. In what way can you leave twitter yet still retain followers and followees. Those words only mean something in the context of Twitter. We have no relationship in the world. If I follow someone on twitter and then they exercise their "right to exit and retain" do i now follow them on tiktok and mastodon and telegram and etc. No of course not.
Suppose I hate nazis and follow all the nazis on twitter. Now I exercise my right to exit. What data about the people I hate will Twitter be forced to provide me?
Seems pretty clear, and subject neutral
Pre and post acquisition it was a clear shift. I would only see that style of poster when people I followed purposefully interacted with them. Post Acquisition, I began to get many more anti-immigrant, pro-white, pro-nationalism style posts in my feed.