Hacker News Reader

Comment by jeswin | Hacker News Reader

Databases we have today aren't designed for the kind of consumer applications we use today. They are designed for banks, Amazons and their smaller counterparts - where a central entity (the business or corp) needs full control over the data.

The most popular consumer apps (FB, LinkedIn, blogs, photo sharing, goal trackers, slack, basecamp etc) shouldn't be centralized in this fashion. Data(bases) should be owned by users, and synced with friends and colleagues. So instead of a single centralized database, we need a million tiny little databases which are synced with peers.

Tiny databases fit well into today's hardware capabilities landscape. We have plenty of fast storage, and powerful CPUs on computers and phones. It's time to stop building as if the server is the only place where queries can run efficiently. Truly serverless, that is. This also improves data security in general; there is no central place to hack.

Happy to talk about this in detail. Please feel free to email me.

View on HN

>Databases we have today aren't designed for the kind of consumer applications we use today.

Yes they are? Relational databases are very good for consumer applications with relational data. NoSQL databases are good for everything else.

>Data(bases) should be owned by users, and synced with friends and colleagues.

Why? You use words like should - why SHOULD it be that way? Users are able to create / buy / rent infrastructure to manage their own data - and yet they don't ... why?

>Shouldn't be centralized in this fashion. Data(bases) should be owned by users, and synced with friends and colleagues. So instead of a single centralized database, we need a million tiny little databases which are synced with peers.

Nice in theory (maybe). The problem is that centralization provides enormous quality and performance metrics. Distributed systems are resilient, but slow, and require a lot of care to maintain consistency and integrity of the network. Take email as an example. It took us 20 years to get email spam somewhat under control and a major part of that was creating a centralized infrastructure (from using centralized email providers to third party whitelists and blacklists). It's gotten to the point where hosting your own mail server is a huge hassle and pretty much impossible for regular people. I spent about a week trying to figure out why our Office365-originating mail wasn't being received by one of our customers. It turned out, their mail provider matched a particular phrase in our automatically-inserted disclaimer and completely rejected the mail.

Decentralized systems also make controlling your data all but impossible. If I want to remove my data from Facebook, that's the only entity that I need to deal with and it's relatively easy for regulators to set sound policies in place. In your distributed system, once your data leaves your node, you've lost all control all of it forever.

> Decentralized systems also make controlling your data all but impossible. If I want to remove my data from Facebook, that's the only entity that I need to deal with and it's relatively easy for regulators to set sound policies in place. In your distributed system, once your data leaves your node, you've lost all control all of it forever.

That's already the case though. When I ask Facebook to delete my data they delete their copies of it. However, anyone who has downloaded photos, any advertisers who have scraped that data, any archival attempts, any search results, and a whole slew of other partial copies of that data still exist. It's almost better for data to be explicitly distributed to make it more obvious that it can't be easily removed.

> where a central entity (the business or corp) needs full control over the data.

Or where we want some central control. Until we solve decentralised spam, I really don't want to share databases directly with other users. If you can generate a new identity yourself and then inject your own content into other databases, that's a recipe for pain.

Federation kind of works, but I can't imagine a fully distributed slack for example. I'm on matrix and get the "elon musk wants to share bitcoin" images every day from telegram bridge.

Even in email, spam is still not really solved. I still get both false positives and false negatives.

> This also improves data security in general; there is no central place to hack.

I see your point here, but would also counter with: degraded security because we need N databases to patch every issue after the issue is public, rather than 1 database (potentially) before the issue is public.

> Even in email, spam is still not really solved. I still get both false positives and false negatives.

See my comment about Secure Scuttlebutt below. Email is a little different, in that complete strangers (and apps) should be able to reach you. However, that's not the case with FaceBook, Slack, Instagram etc. If you sync exclusively with people you know (or your organization) and their connections, spam won't exist by design.

What you gotta do is to model online communities similar to the physical world. In meat space, you choose the people who you are accessible to. That'll work here as well.

> I see your point here, but would also counter with: degraded security because we need N databases to patch every issue after the issue is public, rather than 1 database (potentially) before the issue is public.

WASM-based db engine could be a good option, if the app is browser based. We could treat it as part of the app, and the standard security considerations apply.

SQLite running in wasi with a sync protocol, maybe like couch, or a CRDT depending on multiple writers.

When I use an application on the web, it should save data into my database not the app providers.

The approach adopted by zeronet maybe what you're looking for that allows peers to synchronize their data in master-less manner.

> The most popular consumer apps (FB, LinkedIn, blogs, photo sharing, goal trackers, slack, basecamp etc) shouldn't be centralized in this fashion

The evil CAP theorem enters the town here. Consumers want things to work fast, it doesn't fit the decentralized approach.

> Databases we have today aren't designed for the kind of consumer applications we use today

Consumers don't really care what kind of database do you use. They want to open the page and click the button, quick, now.

You are correct but part of this can be alleviated by aggressive caching by the client devices and delayed (optimistic) updates -- to a large extent though. Doesn't apply to all problems in this space.

> The evil CAP theorem enters the town here. Consumers want things to work fast, it doesn't fit the decentralized approach.

I think we have to distinguish 'decentralized' from 'distributed' in this case. In a decentralized approach we reduce (by many orders of magnitude) the size of data we're dealing with. Locally installed/browser-based applications only have to deal with data that's created by, or directed at that user (and locally stored).

Add: For a number of applications, hard consistency guarantees are overkill. You could totally live with eventual consistency (some don't even need that). In fact, other than e-com and banking I can't think of any of my daily-use apps needing such guarantees. And indeed, decentralized data storage is a bad fit for banking and commerce.

Network hops cost time regardless of payload size. Distributed consensus (which is what is required for a “decentralized” system) requires interaction between distinct nodes.

You should note most consensus algorithm designs start with sharing a single ‘register’ (or ‘value’).

Most banks were doing “decentralized” relaxed consensus from the beginning of “banking”. Local office accepts your deposit of your check. “Funds available next day”. Because by “next day” they have confirmation from counter party for the funds.

In your example, bank system can be seen as employing eventual consistency approach.

The C in CAP isn't required if we're assuming that there's an explicit "sync" that the user only expects to happen occasionally.

FB, LinkedIn and Slack are exceptionally happy that it is centralized; what do you mean “shouldn’t be centralized”? Who cares about what’s better for the user? He’s not the customer anyway.

Sadly.

> Who cares about what’s better for the user?

All of these apps chose technology that was available to them when they started building. So the focus could be to enable building modern usecases on decentralized data platforms.

If one of them becomes the next FaceBook, personal ownership could become a baseline expectation for similar apps. Once data is owned by users, apps become a layer on top of data which can be substituted as needed.

Edit: simplified.

Decentralization has its own issues - spam filtering, lack of moderation (people often do actually want it), anonymity.

Sure. But there are good examples of decentralization making spam difficult or impossible.

One such is Secure ScuttleButt (SSB). You sync with friends and their friends (and their friends maybe, based on configuration). Now people aren't generally friends with spammers so you don't see their feeds. And if they are, a sufficient number of your friends would be blocking them - once a person is blocked by a certain number of people, stop syncing with them.

I am not convinced that SSB is a solution for all types of apps, but for Social Networks it works darn well. Oh, and zero ads too because you'd never sync with ad networks.

Great idea. Perhaps we could apply it in different areas as well, such as consumer reviews?

I don't know. Facebook can spot fake profiles because of the centralisation, not sure how that'd be avoided in that system. Also, if users are really anonymous, the system will end up turning into a cesspool of disgusting and illegal stuff. At least that was the fate of all peer to peer social networks I'm aware of.

That would be handled client-side, like email. You could build in a standard set of rules, and then have some kind of ML process profiles you manually mark as spam. Just have it filter profiles in API requests through the fake profile detector.

> At least that was the fate of all peer to peer social networks I'm aware of.

That doesn't seem true of Mastodon. The few servers I've been on were fairly tame. I think the worst I saw was someone posted a sexualized anime-style image, but there wasn't any nudity.

I didn't immediately spot anything weird on Diaspora either, but I only spent like 45 minutes on there.

> Also, if users are really anonymous, the system will end up turning into a cesspool of disgusting and illegal stuff.

Anonymity can be okay, but only in the presence of heavy moderation. HN is basically anonymous if you want it to be, but it stays on target because of the moderation. You can see it on Reddit too. AskHistorians is a wonderful place to be; the subreddits without even basic moderation do indeed turn into cesspools.

Federation has historically made this difficult because you can only moderate your own instance, which is often a minority of the content you consume. There are potential solutions for that, though. You could allow users to "flag" posts, and add those flags to the data you federate. Then, like the fake profile system, you could filter them based on user parameters. I.e. hide posts that have at least 1 "racism" flag.

If necessary to prevent brigades, you could also filter the flags that you accept based on the user that flagged the post. Maybe weight them based on similarities in accounts you follow.

I have to admit I haven't look into p2p social networks in years. The ones I've tried in early 2000s (such as freenet) had really dodgy users. Perhaps a lot was learned since then.

Anyway, I agree with you that what really prevents people from using the system for nastiness is moderation and not necessarily identity. However, the problem is not only fringe and extremist beliefs, but actually becoming a platform for actual crime, through sharing of photos and videos of abuse. That's what worries me the most about p2p social networks and the reason I'm generally not ok with storing other people's data in my computer, even if encrypted.

A lot of facebook's value prop for users is in what you're saying - not as a social media company, but as a centralized identity provider for the social graph.

Perhaps a decentralized solution could still use a centralized identity provider to help with things like spam & impersonation/fake profiles and prove who they are, while still allowing users to own the information _about_ themselves.

The problem here is incentives; there's not much money in being a dumb identity resolver unless you're harvesting everyones data AFAICT, so I sort of doubt a private consumer solution will emerge in the near future.

It’ll never happen (in the United States at least) but this then sounds like a service that should be run by the local government, or perhaps something more global. People on HN often talk of making the internet an actual public utility, perhaps something like centralized online identify would follow a move like that

yes absolutely. In my scifi utopia, the government provides a digital identity to each citizen as a social service :)

One of the downsides of this is that it actually degrades privacy by improving private institutions (and the governments) ability to track you.

One way I've thought of to get around this is to allow people to spawn new identities from their own root identity. This could allow a certain amount of anonymity, while allowing institutions to filter identities by requiring a certain amount of social proof (e.g. length of time since identity has been created).

Some services might ban non-root identities (e.g. banks), while others (like a bar) just needs to be able to trust that you have a valid ID and can be used to prove you're over a certain bio age.

In the event that you need to prove you own a specific ID, there could be a way that given a root ID and a non-root ID, you re-generate the non-root ID from the root - thus proving you own it.

I think it’s a naïve position. Mail was decentralized. Now it’s mostly federated among 4-5 big providers.

Historically, people value low cost and convenience over any other feature. Unless there is some GDPR like directive requiring decentralization and self hosting, it will likely stay centralized and become even more so.

There are so many more email providers being used worldwide. Who are the 4 or 5 big providers everyone is using? gmail, yahoo, hotmail, fastmail and protonmail? Don't forget outlook and zoho

Many people still use aol or their email given by ips or work email domain.

Worldwide yandex.mail is popular.

Colleges still give email addresses.

People like myself run their own

Vanity domains are popular.

Mail is still decentralized and news of its death have been greatly exaggerated

Google, Fastmail, Microsoft(Outlook/Office365/Hotmail/Live) already cover something like 90% of one side of emails, and 50% of both sides (that is, 50% of emails originate AND terminate among these three, and 90% originate OR terminate among them). I haven't seen any recent stats, but in 2017, 50% of emails already originated or terminated with Google, one provider.

The significance of "originate OR terminate" is that the provider has the ability to meddle or outright ban, either going out or going in.

> work email domain.

> Vanity domains are popular.

> Colleges still give email addresses.

Most universities, workplaces and vanity domains are running within the Google+Fastmail+Microsoft sphere, regardless of the domain name. Every university I've worked with in the last 10 years (e.g. NYU) was using a G-Suite or Microsoft hosted solution, as was every single workplace.

> People like myself run their own

There is a very long tail for the last 1%, but likely 99% of the users are served by 10 companies worldwide. I call that federated, not distributed.

I used to run my own server as well, for ~15 years. Until one day, google stopped delivering it to gmail or g-suite addresses (sent it to spam). There was no one to talk to to figure out why. Which turned out to be ~70% of email I sent -- despite having SPF, DKIM, DMARC and everything else properly set up. Eventually, I switched to fastmail hosting because I have better things to do with my time than figure out why Google is blacklisting me.

It's federated, not distributed, and we're all at the mercy of the big few.

Most of the responses to your post are critical, but I couldn't agree with you more. Having a local database would solve all kinds of performance, connectivity and privacy problems. Transparent background synchronization would be beautiful.

Unfortunately, this is a really hard problem to solve. Sync is a bear of a problem in the face of conflicts. An immutable datastore with a log file of all mutations would help a lot, but you'll still have to manage the conflicts. This is not only a back end problem. What kind of user interface do you present that actually makes sense to an untrained user? "Bill changed Mary's name to Marybeth, but Fred changed it to Mary Beth. Which is correct?" Most users will say "Whaaaa? Why are you bothering me?"

If you are syncing your databases with friends and colleagues, then there are multiple places to hack.

"tiny little databases which are synced with peers"

Wasn't that the idea behind Lotus Notes?

Yes! There’s a lot of good ideas and features in lotus notes that I want, and despite it being 2020 I can’t seem to get for any price.

There’s reasons why Notes failed, but the data model was way ahead of its time and I think it’s past time we brought some of that innovation back.

Isn't that what CouchDB was doing?

> The most popular consumer apps (FB, LinkedIn, blogs, photo sharing, goal trackers, slack, basecamp etc) shouldn't be centralized in this fashion. Data(bases) should be owned by users, and synced with friends and colleagues. So instead of a single centralized database, we need a million tiny little databases which are synced with peers.

> Tiny databases fit well into today's hardware capabilities landscape. We have plenty of fast storage, and powerful CPUs on computers and phones. It's time to stop building as if the server is the only place where queries can run efficiently. Truly serverless, that is. This also improves data security in general; there is no central place to hack.

Sounds like you're describing the MetaCurrency Project's Holographic chain pattern that is currently being developed:

https://medium.com/holochain/holochain-reinventing-applicati...

Edit: I looked at Gitsta, have you heard of Junto? they are launching their app built on Holochain (which is itself built using Rust) very soon: https://github.com/juntofoundation/Junto + https://www.kickstarter.com/projects/1958309948/junto-a-new-...

Gitsta is work in progress; nothing to show yet, sorry.

I had looked at holochain earlier, but to be honest I found their literature a bit difficult to grasp (with a quick read). But I was aware though that they're trying to solve the same problem.

My view is that once you are able to reduce the amount of data an app has to deal with, you might even be able to rely on plain text files, yaml collections etc. for data storage/exchange; and fed into a db engine for faster querying.

I'll look up Junto, thanks for the link.

> I had looked at holochain earlier, but to be honest I found their literature a bit difficult to grasp (with a quick read). But I was aware though that they're trying to solve the same problem.

I posted your feedback into the Holochain forum, and used it as an entryway into asking some questions I am grappling with:

https://forum.holochain.org/t/technical-holochain-101-for-se...

Do I have your permission to link it to our comments here?

> My view is that once you are able to reduce the amount of data an app has to deal with, you might even be able to rely on plain text files, yaml collections etc. for data storage/exchange; and fed into a db engine for faster querying.

If I understand you correctly then I think this is something Holochain's developers are tackling too. I think for the purpose you describe, they put a big focus on relationships and p2p actions that are 'gossip-ed' (spread) through a content addressable distributed hash table (if me mentioning this strategy seems unrelated, it shows the limitations of my technical competences).

If you're interested to engage, I'd love to know what things you found difficult to grasp in your quick read? This is a big subject with a lot of entry points, so my question is a broad question, on purpose. I'm interested in which functional use cases you are seeking/developing, and whether your critique of the current centralized architectures is similar to Arthur and Eric's critiques (the two Holochain originators), versus whether we have reached - and agree on - the same conclusions. The first one (critique) being more important to me: do we see the same challenges/shortcomings? I hope that make sense.

From my understanding Holochain's main thesis is that centralized software obscures 1) an app's inner logic/workings, and 2) the relationships between peers and the data they create (as well as the shape and size of the data they generate).

Through my own my research the overarching patterns I found is that corporations monopolize blueprints. Through the use of centralized black box software, they can also act as middlemen, extracting parasitic rents. They can control and manipulate us by tracking our behavior and steering us into the direction it's algorithms (and often the profit motive, e.g. advertising) have decided for us.

Society's innermost services (like money, law, identity, intellectual property databases, healthcare, land registry) are often invisible or semi-invisible architectures (they require a high clearance level, or the information is stuck in old proprietary formats that make them expensive and difficult to maintain). Because these services are still centralized, they evolve very slowly, meaning that their use remains limited.

A big part of my interest in Holochain comes from the work of the Mikorizal.org team (Bob and Lynn, and Pospi) being done on Open Value Network economic protocols. Using Holochain they are building a Resource Event Agent (REA) acounting model that is completely P2P. Their projects aims to provide an alternative to the closed corporate Enterprise Resouce Planning (ERP) software that is the norm today. I think this new paradigm can help us to see more clearly the actual scarcity of natural resources, as well as use it as a tool to cooperatively fund and develop open source hardware (and software) in the open.

Edit: updated for clarity.

Thanks, I'll engage on the forum directly. But I won't get time to look at it until this Saturday.

Ah nice, I saw you engaged with my post there. Also just a note that I edited my previous comment for clarity.

so I spill hot coffee on my laptop and brick it I lose my entire Facebook history? Not seeing it really.

Centralised _is_ what consumers want, without an generalised and public IT infrastructure for users to handle backups, which we don't have.

But you don't lose all your code if you spill coffee on it. Similar mechanisms could work here as well.

> generalised and public IT infrastructure

You don't need public infrastructure for this. There could be free and paid options (similar to Dropbox) to hold your data securely in encrypted form.

I mean public in so far as it just happens without you thinking about it, that could be commercial too, its just more pervasive as public. When such infrastructure becomes a _guaranteed_ aspect of a given operating system. But until then people implicitly _want_ centralised.

One healthy middle ground here is the situation with git. Git is fundamentally decentralized, but I can use it in a centralised way if I want to via services like github, gitlab, bitbucket, etc. Those services allow me to collaborate with other users and back up my projects. And because it’s decentralised, I can trivially move / clone my repository to other providers (or to my own servers) without losing any data. (... Well, except for github issues; which arguably should be stored in repository itself.)

I can imagine a similar situation for a small database too. Live clone to Dropbox or something similar. The user gets all the benefits of a centralised system but stays in control, and they can switch providers whenever they want.

ye but you know in practice that means at least 95% of users are on the centralised version. We continue to build up this 5% of technocracy that have different rules applied to them because they know enough to care.

That sounds centralized again.

I think it is a common misconception that banks use database transactions the old fashioned way. They seem to be more like eventual consistency. I initiate a transfer and there is some check about available balance (although with various overdrafts, etc. it is unclear how strict), then it is marked pending and eventually the transaction succeeds or fails days later (an international transaction).

This is conflating things. At a database level it is absolutely ACID compliant, full transactions are used for situations like you mention within one bank.

The fact there may be interplay between custodian banks and various triggered events has nothing to do with the type of database consistency features of an RDBMS.

Are you familiar with Scuttlebutt? It's exactly what you're describing.

Look at the hyper-stack https://hypercore-protocol.org/

mnesia seems like what you are looking for.

Fascinating! To save others a search:

Mnesia - "A heavy duty real-time distributed database" (written in Erlang)

Git repo - https://github.com/erlang-labs/mnesia

Documentation - http://erlang.org/doc/apps/mnesia/Mnesia_overview.html

Yeah, it is being around forever. Erisson or some Nordic company use it for phone switches. It is design for concurrency. Personally I think everything pretty much has being invented, it is all about people looking for it and bring into fashion.

I see, Erlang makes perfect sense for the purpose. I think I did read about Mnesia once before, but I must have forgotten about it. ;)

Well, some projects use Software Transactional Memory, which shows that some database concepts are already used in normal applications.

This is a way to do concurrency without explicit locks. It doesn’t help directly with a decentralised database.

Sync between friends? That will basically require standards hell, and the Standards+1 problem XKCD made famous.

I actually feel like Fossil could be adapted to do this fairly easily. It uses Sqlite (it's made by the author of sqlite) so it has the same downsides described in the article, but if you could solve those problems I think it does pretty much exactly what you and the author are describing.

https://fossil-scm.org/home/doc/trunk/www/index.wiki

Reply on news.ycombinator.com