How do dat:// sites interact with servers?
pfrazee.hashbase.io
pfrazee.hashbase.io
CORS only applies to requests for restricted resources like fonts or XHR requests that aren't simple GETs and POSTs.
"So, while this is typically not possible: foo.com/index.html ---GET---> bar.com/pic.jpg"
Typically this is possible. Images, stylesheets, scripts, iframes, and videos aren't subject to CORS.
"You can solve it by routing the request through your host server: foo.com/index.html ---GET---> foo.com ---GET---> bar.com/pic.jpg
Not necessary, the client's browser can get bar.com/pic.jpg just fine all by itself.
"Pinning tools like Hashbase and Homebase help keep dat:// sites online"
If you publish a dat archive, how do you notify Hashbase to pin it? Can you do it through dat?
To keep my dat alive with Hashbase, do I have to set up an account, provide and confirm an email address, link a credit card, etc.?
Are centralized servers, financial institutions and surveillance all required components of the anonymous, decentralized, peer-to-peer web?
This is provided by hashbase in their ui. Sign up for an account and add your dat. It is email based, no credit card/address unless you go over the data cap.
If you are uncomfortable using the service you can run your own node, it is open source: https://github.com/beakerbrowser/hashbase
You're right, that was a misleading example. I changed it to data.json to be more clear.
> If you publish a dat archive, how do you notify Hashbase to pin it? Can you do it through dat? Does it require setting up an account on Hashbase, providing an email address, linking a credit card, etc.?
The "Pinning" system is very similar to Git remotes. You can pin using any endpoint that complies with https://www.datprotocol.com/deps/0003-http-pinning-service-a.... So, similar to a git remote, you do need some kind of authentication with the pinning service - unless somebody writes one that's open for anybody to push to.
The UX flow will be similar to a git remote as well, you use an HTTPS to tell the server to sync the dat. So, it's an explicit user action.
We've got two nodejs implementations of the pinning service API you can self-deploy, https://github.com/beakerbrowser/hashbase and https://github.com/beakerbrowser/homebase, and then we run a Hashbase instance at hashbase.io
> You're right, that was a misleading example. I changed it to data.json to be more clear.
CORS doesn't preflight GET, POST or HEAD methods unless they have custom headers or content-type other than application/x-www-form-urlencoded, multipart/form-data or text/plain.
So a simple GET bar.com/data.json works just fine in today's browsers.
While fetch doesn't preflight for GET, it does require an Access-Control-Allow-Origin header. You can specify `no-cors` in the mode to circumvent this, but then you cant access the response body (https://developer.mozilla.org/en-US/docs/Web/API/Request/mod...)
Open the console in an empty Chrome window and paste this:
fetch('https://pfrazee.hashbase.io/feed.xml')
.then(response => response.text())
.then(str => console.log(str))
fetch('https://beakerbrowser.com/dat.json')
.then(response => response.text())
.then(str => console.log(str))
application/json is a content-type other than application/x-www-form-urlencoded, multipart/form-data or text/plain, and thus a request for it will fail unless the required CORS headers are present
Additionally, you're thinking about the "wrong" Content-type header: the limitation you're mentioning about urlencoded and so on is a limitation on request headers, not response headers.
The CORS headers are required for the GP's described request to succeed, but not for the reasons you give.
To some extent, you're conflating some of the requirements to avoid preflighting with cross-origin requests simply being allowed, and they're not the same.
Servers and financial institutions will never go away, no matter how hard we try. I'm hoping that we can make surveillance go away by sharing much less data with organisations who rely on surveillance to survive.
Hashbase is not "centralized" in the sense that you are always free to choose a different provider of its hosting services. You can host your own Hashbase: https://github.com/beakerbrowser/hashbase
You can even choose multiple providers at once, providing you with resilience in case one of your chosen providers violates your trust, eg. by losing your data or using it to spy on you.
CORS applies to XHR. Including GETs and POSTs, and including fetching images over XHR.
There's some other sibling comments here discussing preflight requests, which it sounds like you might be referring to, but CORS is not limited to just preflight requests.
This is completely incorrect. You can not make any XHR GET or POST requests cross-origin without CORS.
Fonts also do not require CORS, you can link to them in your CSS/styles/etc without technical restrictions (might be legal restrictions of course).
You do this for example with `fetch('./any-resource', {mode: 'no-cors'})`.
You can then for example do a `.then(x => x.blob())` and then use the resulting blob as an image.
This is a little backwards, the goal of CORS isn't just to protect the _user_ it is also to protect the _third party website_.
All it takes for a website to opt-in to this is just adding a single header - it's possible for bar.com to allow the request from foo.com by opting into it.
It depends on what you want to do with the image: https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_enabl...
I don't think this is an adequate approach to security. When the browser presents me with a prompt to load data from a third party site, I don't know what data is being loaded, what it's being used for, or whether this prompt is expected (as part of the regular functioning of the application) or unexpected (indicating that the application has been compromised, and I should navigate away from it).
In general, I've noticed that users react in one of two ways to these sorts of prompts. Naive users will blanket allow -- allowing all sites to access all of the capabilities of their browsers, regardless of the reason or necessity of that access. More sophisticated users will blanket deny. If it's not immediately apparent why a site needs the permission that it requests, that request will get denied, even if it's a valid requirement. Very very few users will think about why a site is requesting the permissions that it is requesting and consider those requests on a case by case basis.
What I can understand, is, they use new protocols and this is an issue in the Web today. I think the only way they can succeed would be with some laws or misstakes by big corps that would drive customers away from them.
What I also liked was remoteStorage [0] it is a bit like localStorage, but the data is managed independendly from the application itself.
I've mostly done Dat. I want to do a bit more IPFS.
Dat feels a bit more like git for files - you can create a local file archive using the command line tools, and it's a separate step to sync it to the peer-to-peer network. There's a global discovery service for advertising archive keys, but it doesn't work at the level of single files. It's very lightweight.
IPFS supports many of the same operations, but you're mostly interacting with a local gateway server which is continuously connected to the network. I believe IPFS tries to content hash every single file so they are de-duplicated globally.
If you use Dat through its command-line app, you get real-time updates to the files shared using it.
CRDTs naturally fit with P2P/decentralized topologies, and we've generalized them in https://github.com/amark/gun which is the most popular (8K+ stars) open source (MIT/Zlib/Apache2) realtime decentralized database.
It is running in production on P2P alternatives to Reddit and other apps, that have pushed over half a terabyte in a day.
Dat is realtime. You are notified about updates as soon as they are distributed. In Beaker, the files-archive API has a `watch()` method to do this. If you're accessing a dat files-archive, you're participating in the syncing swarm and so you'll receive those updates automatically.
You'll want to use a UDP socket if you're streaming a high volume of data with low latency requirements, for instance for a multiplayer FPS. But Dat has been used to stream live video, so it's probably real-time enough for most Web use-cases.
Small aside: Comparing Dat to CRDTs is apples to oranges. It's like comparing Javascript to B-Trees; they're not quite the same kind of technology. In fact, the new version of Dat uses CRDTs in order to allow multiple users to write to a files-archive.
Append only logs have overhead that would make it a poor choice for most realtime applications, like GPS tracking, yes FPS games, google Docs, website creators, and many more. Basically any use case where data mutates.
In 2010 I used and built all my own custom event source system, I was the hugest proponent of this / append only logs. It was so futuristic. But 4 years in I hit all sorts of scaling problems and had to redesign everything from scratch and that is when I found CRDTs. On all accounts they are superior, in a mathematical or logical manner, because they are a superset to DAGs, immutable/append only logs, and many other popular data structures. Not apples and oranges.
https://dat-tiddlywiki.glitch.me
The UX definitely needs some improvement, but it’s just a personal project so far.
If you want to make a library with a C compatible API I'd be tempted to explore SubstrateVM. It can export C symbols to the generated standalone .so / .dll files, maybe you can even reuse some of the JS code, or failing that, a Java/Kotlin implementation would be compiled down to native code or be usable from other scripting languages like Ruby.
Where are you getting 4 years from?
Check out the first commit: https://github.com/datproject/dat/tree/464679267049899eafa34...
Bit more on the funding history, etc. : https://blog.datproject.org/2017/09/15/dat-funding-history/
I'm sure hashbase.io will be blocked really quick, so it's important that the core P2P address system stay in the forefront. Transports also need to find many ways to communicate over https, shadowsocks, tor, DNS, and others.
https://www.scuttlebutt.nz/stories/design-challenge-avoid-ce...
UDP hole punching (using the UTP protocol) and the discovery network works a lot of the time.
Much of the people publishing public content for access by Beaker are using hashbase.io to "pin" the content and to act as a public peer, and those ports aren't behind a firewall, so the data can be directly replicated easily.
At least add to the title on HN something about what a dat:// link is :)
We’re not there yet, but Mozilla is making a very positive step in that direction, and hopefully other browser vendors will follow. If I were creating a new protocol, I would be doing it now, to “skate where the puck is going” as they say.
That’s our approach with Beaker. We use peer-to-peer systems as much as possible, and then plug in servers as minimally as possible.
If I'm making a new system or setting policy for a government or other high-security minded client (like a political campaign, military contractor, activist group, or private intelligence corp) I need off the shelf stuff with zero known attack surface OR I need to individual vet every single offering within that protocol suite. This is why you can email members that work for The Government of Ontario, but they won't click on links to non-whitelisted places. The attack surface when clicking a link is fucking huuuuuge (pdf 0days anyone?), while the attack surface for loading an email is much smaller.
There are a ton of interesting web-replacements that hackers are playing around with right now, but the one that wins for the next web is the one that lets stupid people do whatever they want without worrying. In my opinion, 3rd party means worrying, and in an ideal world it would go away.
The irony of this whole thing is that I'm actively arguing against my own long-term interests. A structural change of the kind I advocate for would dramatically reduce the profitability of being in either data science or cybersecurity; both fields I have a foot in. But I don't care.
Securing the flow of information between people is too important to humanity's long term survival.
I think one other area that the Web hasn't tapped into enough is using protocol/scheme identifiers to introduce strong guarantees to URLs. You can compose schemes with a '+', so I think if you wanted an "on click guarantee" that a site is going to have certain security properties, you might try something like:
http+safe://.../ dat+safe://.../
And then the site would load in a "safe mode" which, like the "uninstalled" mode, is extremely limited in what it can do.
Everyone is a server is pretty much the opposite of "serverless". The article is just illustrating the migration path from traditional web assets from centralized servers and how you can use them in the distributed web.
A real life use case would be something like:
You want to build a financial tracking app in the distributed web. But your bank is the central source for your data. With the method shown in the article you could request the api endpoint from bank.com inside your p2p/Beaker app then load that data into local storage/a dat archive/memory and use it to track your data. Think mint but without giving your bank credentials to a 3rd party.