Hacker News Reader

This whole NSA thing should be enough to push things like named data networking/content centric networking, WebRTC and meshnets (http://www.reddit.com/r/darknetplan) into the mainstream.

I've tried multiple times to engage on this topic. From my experience, the people set of people who are interested in these topics for "freedom" reasons and the set of people who are technically informed enough to contribute even minimally to technical discussions are non-overlapping.

Don't be so sure. In the great game there must be gambits and feints, misdirections and intentional losses. Some things are better left unsaid in public places with many ears.

Certainly not speaking about myself, but that's the sense I get from talking to people whose interests overlap like you'd hope.

Reminds me of this:

http://www.paulgraham.com/say.html

Yeah, this is mostly bullshit.

The people who hide away are the same as the people who want you to sign an NDA before they talk about their startup because it's just too revolutionary and cranks who invent new crypto algorithms in their basement.

DS design is extremely complex and requires the coordinated effort of a lot of really smart people to get right. Anyone building these systems in secret just has no idea how terrible they are at it.

The people I talk with about this, who also have the technical acumen to build these things, or those with the political connections to drive change, are both groups of people who have known this level of spying has been going on through their lifetimes, and are unconcerned.

The response on HN has been an order of magnitude larger than the response from any person I know. Of everyone I know, I'm more concerned about the NSA spying than anyone else, and I'm less concerned than most commenters. I begin to wonder if this isn't some kind of nerd-sniping paranoia-groupthink.

People's feelings about something is an irrelevant metric. It doesn't fucking matter if no one is concerned. There are plenty of past disasters and tragedies that happened because no one was concerned.

If other people aren't worried about it, why should I be?

Because other people are clueless and apathetic?

If other people litter, why shouldn't I? If other people forgo vaccinations, why shouldn't I? If other people are just following orders, why shouldn't I?

But other people do not litter. Others do get their kids vaccinated, regardless of what the vocal antivox minority says. Why are the others just following orders -- what are the orders like?

If all of your friends were jumping off of a bridge, why wouldn't you? If you don't trust them in what they do, why are they your friends?

http://xkcd.com/1170/

This xkcd inspired me

As a human adult, you are probably capable of making moral judgements, and making limited predictions about how your actions will affect you & others in the future.

The minority of people who litter make the experience of walking/living/sitting in that area less pleasant for everyone else, and they increase the likelihood that others will litter ("this park is already full of trash; should I really go chasing after that wrapper that fell?"). Alternatively, you could not make it worse, or you could even make it better by picking up a few things and throwing them out as you pass through.

The minority who don't vaccinate encourage others to avoid vaccinations, and it doesn't take very many of them to undercut or even nullify the benefits of population vaccination (like protecting newborns who can't yet be vaccinated).

Etc. etc..

Do you want to be worse, meaner, less moral, etc. than the average? Or do you want to strive for a little more than that in life?

Realistically, most of us aren't going to have a huge positive or negative impact on the world; but thinking locally, if you are going to seek a mate, friendships, etc., you'll probably do better (and be happier) in life if you at least try to do better than the average.

You may even get an opportunity to make a seriously good decision somewhere along the way, if you have good moral habits and keep your wits about you.

Your third sentence is a pretty effective rebuttal for the first two...?

No, I'm saying no one cared then and disater or tragedy struck anyway. There is little correlation about how we feel vs. what happens to us.

Obviously not caring doesn't accomplish anything, because people who don't care about something won't act.

But if people do care about something, they are motivated to act, and may significantly alter the course of what happens -- not always, and not always as they wanted, but even a small number of motivated people can make serious waves, depending on their skill, charisma, cleverness, opportunity, etc..

Hence, what we feel matters.

Most people aren't "visionaries" or aren't as good at "connecting the dots" as the NSA likes to say. They live their lives in the present - today. Does the NSA spying affect them today? If not, then they aren't concerned about it.

But that's a very wrong way to approach things. You have to connect the dots and see where this is going, before it's too late to do anything to stop it.

Plus, I bet most don't even realize that even if authorities don't come knocking at their door because through their spying they saw they downloaded that last Game of Thrones episode (how long until that happens, anyway?), they could still use the spying to influence judges, politicians, mayors, and so on - which will affect their day to day lives, depending on the decisions those people take through blackmail.

But as I said, most people can't connect the dots like that. Ignorance is a bliss, as they say.

You make a very good point. I was looking for someone to make this point to me, but I didn't know that when I made the post.

The first and easiest argument to make about this is the chilling effect it will have on journalism -- even some things that are unclassified and should be public knowledge can be prosecuted as whistle-blowing.

There are plenty of people who want to build things in a decentralized, distributed way for fault tolerance and cost reasons -- those people tend to be fairly competent.

Or, decentralization in the face of a very specific threat (e.g. MAFIAA and piracy-optimized things like BitTorrent, and FinCEN and things like Bitcoin).

Until recently, there weren't widely-believed threats to general Internet use.

I have a friend who is really into WebRTC, and help him with debugging sometimes. Honestly, I've never found a network I use often that it works on. Doesn't work on my AT&T LTE except for presence. Doesn't work at all on my work internet. Video doesn't work on my work guest internet outside our firewall. Doesn't work from my hotel internet. I haven't tried at home, but I'm only home a couple weekends a month, so that's useless anyway.

WebRTC depends on things non-home internet filters out or prevents, so it is useless. The way to get around that? Use central servers and make your traffic look like HTTP. Heck, even STUN protocols admit there are cases where it won't work, but WebRTC fans seem to think they'll always be able to get a P2P connection. The technology is just naive as far as I'm concerned. Something that only works on home internet and a few open networks is worthless.

Sure, but as long as there's federation or something like it, you get to pick your choice of a server (including running your own). That's good enough for me.

> This whole NSA thing should be enough to push things like named data networking/content centric networking

You're damn right. It wasn't even a few days after this whole thing broke that I started working on a simple, distributed and automatic file synchronization program. No third parties and open source.

I was absolutely stunned to find out that nothing out there meets those criteria (despite my best efforts). The only thing that was close was git-annex assistant, but I don't want to use git for simple file synchronization (particularly for large files).

There's also Bittorrent Sync, but it's completely locked down. :-/

This is all good but the biggest thing right now is that there is no coordinated effort to build a system that's easy to use, secure, and open source.

Everyone can spend their time working on their own solution but no one would use it as everyone's work will be incompatible with each other. This means no one (or rather, not enough people) will use any solutions that are developed and revert back to something like Skype, FB message, etc.

I think what we need is to establish a foundation that coordinates the development efforts for these secure and reliable software packages with a few centralized projects that become a standard (here's hope for WebRTC).

Out of curiosity, what's wrong with Unison?

That's what I use now. But it's not automatic and is designed for use between two end points. I want it to be automatic across N end points. (And I intend to use a bittorrent-like protocol to achieve that, rather than an rsync-like protocol that unison uses.)

[EDIT] I forgot one of my most frustrating nits with unison: you need to have the same version on both ends. It's solvable by building the right version yourself, but it's damn annoying.

Wouldn't it be easier to contribute those changes to Unison instead of starting over?

Dunno. They feel like pretty big changes to me. I imagine it'd have to be completely gutted since the entire protocol would have to change.

Plus, I am not fond at all of its interface/configuration. Very complex.

"I want it to be automatic across N end points."

I solved that problem, using unison, I guess near 15 years ago the unix way, not the windows way.

Its been awhile but with a little symlink fun you can make a little shell script that builds a partial mesh and syncs both the script itself, its list of possible mesh members (as opposed to the 3 or so I used) and all the other stuff, more or less silently run out of .bash_profile or .bashrc or whatever it was. There's already excellent tools in unix for scripting, automation, batch, etc, so no need to include them all into one tool.

The windows way, would be to put all those tools into one (buggy) executable with a gui and an installer that also installs browser toolbars or whatever.

I switched to git away from unison and never looked back. Eventually it got to be too big of a PITA to install 2 or 3 debian packages of different versions of unison all of which can only talk to certain other versions. So then my shell script had to detect which unison versions are installed on this node, and match up to the master list of other nodes and their versions, ... oh forget all this garbage just try this new subversion svn thing. And later, forget this whole script, just put everything important thats small in git and everything thats big in multiple AFS servers running on RAID.

Yes, most of my stuff is kept in sync with git. But there are many things I want to keep in sync that I don't want to check into git. I'd rather they just sync automatically without intervention from me.

Your synchronisation project sounds intersting, do you have source code online yet?

Not yet. I have a piece or two on my github, but otherwise it hasn't been written yet. I'll be sure to post here when I'm done, but I've never had much luck with that in the past.

> There's also Bittorrent Sync, but it's completely locked down.

This really surprised me; for some reason, I assumed it was open source. Since the storage is entirely peer-to-peer, I wonder what their business model is?

Who knows. It's completely opaque to me. I just don't want to be left high-and-dry if the maintainers decide to abandon the project. Plus, I want to be able to tinker, of course. :-)

[EDIT] This was all I could find.[1] Which isn't much.

[1] - http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-...

Not open source, probably because the bittorrent protocol isn't open source.

How so? Several open-source clients exist. Were they reverse-engineered?

I believe that's not correct. First a nitpick, I think it's only software that can be open or closed source. Protocol specification might be kept secret, though.

Regarding specifically BitTorrent protocol specification[0], it's free to use. Anyone can build their open or closed source BitTorrent client without reverse engineering.

[0] http://www.bittorrent.org/beps/bep_0003.html

I believe the first bittorrent client was written in python. http://en.wikipedia.org/wiki/Bram_Cohen

Just for checking: git-annex doesn't actually check the files into the git repository, so whether the file is small or large is irrelevant to it. Git is only used to keep track of the filenames and their locations.

Interesting. The web site really makes it sound otherwise:[1]

    Then any changes you make to its folder will
    automatically be committed to git, and synced to
    repositories on other computers.

Nevertheless, I checked it out myself and you are indeed correct.

I guess that means I'm out of legitimate excuses? :P

[1] - http://git-annex.branchable.com/assistant/quickstart/

> Then any changes you make to its folder will automatically be committed to git

The _changes_ are tracked (addition, removals, etc.) not the content.

Well, yes...... I know that now. But my initial reading did not catch that. Hence the confusion.

You can try the open sourced project, webrtc based, Sharefest.me for filesharing p2p with no cloud storage...

There's a WebRTC demo that does something like this. I can't seem to find the link, but it was linked from one of the official/reference implementation sites.

FYI,

Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly) worked. We spent a lot of time trying to solve things that are not largely solved, so if I were to start over today, a lot of the hard work would be done.

To be more specific, if I were you, I would start by using the p2p connectivity and data channel code built into WebRTC (at webrtc.org) or libjingle (at https://code.google.com/p/libjingle/). This is what AeroFS (at https://aerofs.com/) is doing.

By the way, writing a distributed file sync system is not that simple, even though it might seem simple at first. Things get complicated really fast. I think that ones reason one of these hasn't taken off is because it's really hard to get right. But, I still hope it will happen one day.

Oh, and there is http://camlistore.org/, which is an open source project run by a really smart Googler. It's not specific to file syncing, but it has a powerful foundation. A combination of the p2p bits from WebRTC and the distributed data bits from camlistore would be pretty powerful. It makes me wish I could find some more 20% time :).

(Full disclosure: I work at Google, often on the WebRTC code base, and am the maintainer of libjingle)

I can hardly consider myself an amateur with respect to distributed computing, so I am ignorant enough to believe that it is not that complex. Even worse, I won't believe it until I find out for myself. :-)

I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency.

Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail.

Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant.

[1] - http://camlistore.org/pkg/

To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly. Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide.

By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.

> To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly.

Those are exactly the things I intend to leave out in my initial work. If it happens to catch on, hopefully I'll get some help. If not, it will still be very useful to me without those features.

> Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide.

Yup. I think this is one of the things that Bittorrent Sync gets right: strong symmetric AES encrpytion.

> By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.

Not to write in it. In the Go community, there's a heavy bias to avoid linking to external libraries if it can be avoided. There is a certain amount of cachet that comes from being able to do `go get ...` and have it work without having to make sure another library is installed.

I'm at the infancy stages of my understanding, so it's certainly possible that I'll come to my senses and build on top of something else. But for now, I'm going to blaze my own trail. At least partially to learn. (Two of the most important pieces---encryption and p2p---are things I have very little experience with.)

C++ free ICE implementations:

C: http://nice.freedesktop.org/wiki/ (glib-based so wrapping friendly to other gnome-supported languages)

Java: https://java.net/projects/siptools/pages/IcedJava

Have you looked into CurveCP? It would provide security and automatic bandwidth throttling with minimal effort.

> WebRTC still uses central servers to establish connections

WebRTC without a signalling server: http://blog.printf.net/articles/2013/05/17/webrtc-without-a-...

That is not even close to a solution. Then problem of establishing a connection (sharing connection info with both parties) remains in its entirety. The user still has to manually give the other user signaling information. The suggestion given is instant messaging chat (which requires a server).

Not to mention that a STUN server is still needed for NAT traversal, as the author acknowledges.

What do you think about http://openpeer.org/ as a possible solution to the signaling problem?

Check out http://blog.grimwire.com/#2013-04-04-grimwire.md

>I believe that we are going to move away from a lot of traditional ways of doing things pretty soon. For one, obviously centralized servers have to go

This is a pretty bold statement, and I'd like to hear your reasoning for it.

I've always seen that centralized is better than decenteralized, just because its easier. Which is why when we see github go down, some teams can't continue to work even though git itself is decentralized. One click solutions, and decentralized applications tend to not go hand in hand.

I'll agree, if we want to maintain privacy these things will have to be decentralized. However I have a hard time making the case that decentralization will become the defacto standard. Even the web has a psuedo-central hub - DNS.

I would say centralised is easier but usually worse instead of better. Most people use some centralised middleman out of convenience. Decentralisation is universally better IMO because there is no single point of failure and no choke point to nickel-and-dime users to death.

> some teams can't continue to work even though git is decentralized.

I think you mean some teams won't continue to work because they don't know how to use git. If the de facto center goes down, a team is still able to use git to share changes with each other.

And IPv4/6 is decentralised, DNS makes it convenient for regular people to use. I would say DNS is decentralised as well, just the allocation of domain names comes from a few sources.

A nitpick, but decentralized & distributed systems do not guarantee "no single point of failure". For example, Skype switched from pure p2p because of a software bug which affected the whole network (ie the client software was the single point of failure)...

That's not true. The whole network was effected because the centralization point they chose had a client bug and since the whole network had to use these nodes it took them down.

To take down a decentralized network with a bug you'd have to have every user on the exact same version and no way for them to simply role back to the previous version. I.e. nearly impossible.

That's also not true - all you need is to bring down a high enough percentage of nodes, which will effectively take down the network through capacity constraints.

Combine this with auto-updating clients and you've got a crashing software bug that brings down the whole network.

Agreed however that the centralisation point effectively still exists thanks to automated updates.

>all you need is to bring down a high enough percentage of nodes, which will effectively take down the network through capacity constraints

On a fully distributed network, that's pretty hard to do. DDOS works by having tons of nodes to attack with. That won't work very well when your target has as many or more nodes than you do.

Taking down a centralized target is vastly easier.

>Combine this with auto-updating clients and you've got a crashing software bug that brings down the whole network.

Only if I can't role my client back nor turn off auto-update.

>>>>Its very nice to have a simple programming language and API available with things like Webkit and Mozilla browsers, but that still puts too much power in the hands of a few companies, and locks us into particular implementations, and also traps us within the browser. Programmers and businesses do need easy-to-use, cross-platform and cross-device APIs and programming languages. I think we can port a lot of the Web APIs to a language-neutral, binary-neutral, implementation-neutral semantic representation. Dump as much of the unnecessary complexity from Web APIs as possible.

Well, this is exactly why I am a huge fan of technologies like MOAI (http://getmoai.com/) where the VM is under the control of the implementor, whose efforts to put the working VM on Platform-[X] results in a single codebase that will run on all Platform[s]. In my opinion, it pretty much matches your requirements, from the simple language, easy to use API, and more importantly: control over the host environment.

The only issue is that competence in MOAI in developing a cross-platform truly 100% compatible code base means you have to scale your attention from the up-front App, down to the lower-level host code. Or else. But thats 100% of the fun, imho ..

Note that this is just one of many such solutions to the issues you describe, and it is indeed a fertile playground for anyone with as much affinity for Lua - and similar tools - as one must have, to suffer the platform wars pleasantly, these days ..

Why do you think WebRTC puts too much power in the hands of a few companies? It's completely open source and can do everything you want. You can take all the source code from either webrtc.org or chromium.org and use it to build your own end-to-end full encrypted communication system that no one can hack (as far as we know, and assuming you have secure device and OS).

The implementation at webrtc.org gives you cross-platform, cross-device source code in C++. You can build any API on top of that, in any language you want. It's already really easy to use if you an to use the HTML5 WebRTC API built into Chrome (and the same API in Firefox).

It seems to me that you have everything you want, or at least all the pieces of the puzzle. Someone just needs to do the work of assembling them the way you want them, and then of convincing the people with whom you want to talk (your friends, family, or business associates) to use your securely built communication system. Technology-wise it's possible, even fairly easy if you know what you want. But the social aspect (convincing others to use it) is more difficult.

(Full Disclosure: I work at Google, and I have worked both on Hangouts and WebRTC)

I'm actually really excited about WebRTC and have a project using it.

I was talking more generally about Web browsers and web APIs, not specifically WebRTC. I still see the web platform as currently the most open, practical solution we have.

However, the reality is that Google IS still a business. And so quite a lot of this key infrastructure, although it is open source, is still being packaged up by a giant private company.

Yes, we can use the web platform including things like WebRTC to do what we want, but I believe the the medium and long-term future will move away from specific implementations by a few companies and towards open machine-processable specifications.

I believe that the way forward will be to move away from textual programming languages and into truly semantic representations of programs, protocols, and implementations. Obviously it will be many years before this is mainstream.

> This whole NSA thing should be enough to push things like named data networking/content centric networking, WebRTC and meshnets (http://www.reddit.com/r/darknetplan) into the mainstream.

Mozilla is looking at making a foray into this with its "Firecloud" initiative[1]. While Firecloud itself appears to be a relatively innocuous endeavour, it would lay the infrastructure for P2P RTC-based meshnets, which are something I've been hoping to see for a while now.

1: http://literaci.es/firecloud

Bear in mind that there are at least five spying programs going on. One is server-based (PRISM), but the others (FAIRVIEW, BLARNEY, and two others whose names were redacted) apparently monitor the pipes.

This isn't so much of a technical problem as a political one. Although, in principle I agree- I would much rather have p2p so I could at least have a fighting chance via encryption.

P2P is harder to build and harder to monetize than client-server, so it's like poison to startups. Crackpots are still into it, but they don't give a good appearance.

"Crackpots"? Is this a serious, honest comment? Or is a crackpot anyone who doesn't choose and create philosophically and technically inferior products for the sake of "Just one more social network that will surely be worth millions if I can make it centralized and proprietary and non-inter-operable and get just enough users." SOUNDS GREAT.

"P2P is harder to build" is an equally loaded statement, though you've caught my intrigue. I wrote a WebRTC signaling server in a weekend that is capable of letting peers establish N:N connections with each other for video conferencing.

Outside of cases where TURN would have been necessary, the clients were all connected, P2P with encrypted connections. (And even with TURN, I believe end-to-end encryption could be ensured, but the traffic would go through a third party)

Sorry, I didn't take the time to sugar-coat my comment. One lesson I've taken from Web 2.0 over the last decade is that users do not care. So why should I care about them more than they care about themselves? We'll see if PRISM changes that, but I suspect it won't.

WebRTC makes P2P a lot easier, especially since it doesn't require installing software. I think getting peer connectivity is 10% of the problem. Video conferencing is pretty much a best case for P2P; general apps are much harder. Things like data sync and handing churn are very complex and not needed in client-server architectures.

everything still goes through your ISP when using peer to peer... but yes to encryption.

Some entity (or service) has to provide your internet access. So perhaps you are always going to need an ISP to some extent. Or you are thinking mesh? Maybe in the future ISPs will compete on privacy issues and concerns in addition to how they compete now: bandwidth and cost. Maybe we'll have municipal WiFi, maybe we'll have mesh. We should work towards a mix of mesh and muni as well as a more privacy-focused ISP model. One can dream :)

Those techs are not taking off because there are no standards.