Weave is kinda slow
generictestdomain.net
generictestdomain.net
Spot on
Possible answers that are discussed are only 'some sort of IP encapsulation', which is vague and GRE, which is just a single solution. He doesn't seem to disapprove of VXlan, so probably something was missing in 'IP encapsulation and GRE'. Was 'all problems solved decades ago' merely hyperbole or is there actually something to it?
Look, I'm just trying to understand the playing field here, for when the moment comes that I need that knowledge. I don't currently have a need for funky networking between Docker containers, but I do have Docker containers and can imagine a future need for funky networking. The article slams new technologies, but doesn't clearly explain the alternatives, which is what I'm interested in, so I'm asking follow-up questions. There is nothing rhetorical here.
In IIS this is done with Bindings.
Also, even if they were using a standard transport encryption like SSL/TLS or IPSEC, PSKs are generally frowned upon for anything other than point-to-point connections.
They describe the PSK as a "password", so what they really want is a PAKE algorithm, however they do not use a password hashing function, so weak "passwords" are susceptible to brute force attacks.
Anyway, all these things are why you should just stick to standard protocols like SSL/TLS or IPSEC.
https://twitter.com/lclarkmichalek/status/544882194456776705
That said, you shouldn't be saying "fuck you" in the first place: it's rude, it contributes to bad vibes in the OSS community, and it hurts you more than anybody. Try instead something like: "I'm having trouble understanding your argument, do you mind explaining in more depth in an email?" Even if you're dealing with a troll, this is still the best strategy.
Ultimately we can't work on even a fraction of the features that every person wants, and Laurie said his idea was simple to implement... so why not show how it's done? Honestly, it's not that sinister and it is certainly not rude.
But let's look at the Tweet in question:
"@lclarkmichalek @weavenetwork please, if it is so simple and robust you are very welcome to contribute a patch."
1. He says "please", which in this case is sarcastic.
2. Then he says "if it is so simple," which is a dismissive way of saying "you think that it's simple, but you're wrong – it's actually very complicated."
3. And also "if it so so ... robust," which is a dismissive way of saying "you think that it's (more) robust, but you're wrong – it isn't."
4. And finally "you are very welcome to contribute a patch," which, first, does not need to be said as presumably anybody knows that they are welcome to submit a patch to an open source project, and second, basically amounts to "so, I'm going to make my problem your problem."
This case is different from a feature request from a user, where "feel free to do it yourself" is slightly less inappropriate. There, what is meant is "this feature is not important enough to warrant our endorsement or any allocation of resources, but if you were to take on the burden entirely yourself, we would consider it."
In this case, what is meant is something more like "we believe that your charge that there is a fundamental issue with our software is false and we are not interested in discussing until you have actually done the work for us," or in other words "fuck you."
I admit, "fuck you" is quite strong for the general case, but the tone of the Tweet warrants that translation.
Just admit that you (if you're the one who wrote that response; it's not exactly clear who's who in this discussion...) were being a bit of a twit and move on. We all do it; I do it, the girl next door does it, my grandma even does it and she's the nicest person I know. There's no shame in being honest about it.
Typically (at least in American English vernacular), "please" is very frequently used in a sarcastic manner (e.g. "You think you can jump from the top of that building and not get hurt? Bitch, please." or "Oh please, like you know the difference between a grape and a grapefruit."). While this sarcastic usage is usually accompanied by some other word prefixing it ("Oh please" or "Bitch please" or "Nigga please" or somesuch), it's not uncommon to see a lone "please" used in this sense as well, and the tweet in question very closely resembled that usage.
For what it's worth I lean more to the uncharitable reading of the tweet, although I'm not as firm about that reading as others appear to be.
Tweets for communication are hard so it's not particularly suprising when what you say and what you think you say doesn't match what other people think you said.
It was actually @lclarkmichalek who ended the conversation.
They're letting you know that they have their own priorities, but they're still receptive to your ideas, which is absolutely fine. What is wrong with that?
Crypto is hard, but it's no harder than a lot of other hard things. If you think someone's crypto is broken, you could point out why you think it's broken. I see no evidence that consigning crypto to a forbidden zone is going to improve real world security, and the old "mature" cruftpiles seem to manifest problems at least as often as newer systems designed with the benefit of hindsight. For example, any competent designer of a newer system would always authenticate before decrypting-- something that was not clearly understood when SSL was developed.
Crypto is a LOT harder than some other things.
3d graphics programmers don't have to worry about side channel attacks through timing disclosures through random numbers returned over an HTTP GET.
Physics simulations don't have to worry about tens to hundreds of millions of dollars of losses because Intel changed the L2 cache slightly in some revision of a processor and now it is possible to glean a couple bits of information about the entropy one uses.
Of all the projects I have worked on, maybe the C/C++ compiler had a set of worries close to what an encryption suite has.
You could toss me into almost any field of software engineering and after a few months I'd feel good. Some of them would have a longer ramp up time (order of months). Some of them might require me to go take a few online courses to learn the field (3D, physics sims, etc).
Encryption requires an entire life of devotion. New attacks are coming out all the time. There is so much financial incentive in the field that the competition is insanely fierce.
3D graphic techniques get pushed forward by publishers wanting the latest AAA game title.
Encryption gets pushed forward because they are trying to outrun either large international criminal organizations or entire governments.
The problem I have with the "crypto should be a forbidden zone" line of reasoning is that the real world evidence shows that the old battle tested systems manifest flaws at least as often as competently designed newer systems do. Crypto, it turns out, is so hard that the probability of lurking issues with mature systems approaches or exceeds the probability of mistakes in new ones.
When I say competently designed, I mean a newer system that passes the sniff tests of experienced crypto engineers. An incomplete list: they're using a cipher that's been peer reviewed and is considered strong by modern standards, they're using that cipher correctly, they're authenticating before doing anything, they are using an IV (if needed), they are not sending anything secret in the clear, they're not branching on secret data, etc.
It's also important to refrain from criticizing people for claims they are not making. As far as I know, Weave is not claiming to implement the entire feature set of IPSec. They're just claiming to offer basic but strong crypto and authentication. If you want more, you are likely using other algorithms like SSL and SSH over the overlay network.
Yes, that comes with a performance penalty, but it's also defense in depth. It's better to trust multiple layers of crypto with independent implementations at each layer so that a compromise of one does not destroy your entire security posture.
It all comes down to the question of how paranoid you are. No encryption will give you the best performance, but no security. If you want maximums security you can run SSL over IPSec over Weave with different sets of keys and different ciphers at each level. Bonus points for generating those keys on different air-gapped hardware, etc.
Compiler design isn't that hard. A perfect optimizing compiler is damn nearly impossible, but good enough is easy. LLVM makes good enough compilers pretty easy, see the number of new languages that get posted to HN every month!
Machine Learning is hard to do right for non-trivial cases, but the only risk is you waste your investor's money and your customer's time, you don't risk identity theft (unless you are using machine learning to write crypto code maybe? :) )
Databases, yeah, everyone screws those up.
The real key is how many people are trying to break your system though. Crypto is exposed, your DB layer can have protections put on it to sanitize your data and rate limit how fast stuff comes in. You can clean up the data that goes into your machine learning algorithm (or use one that is resilient to x% of malicious data).
None of those defenses depend upon defending attacks of context switch timing in the CPU leaking data from a CPU that was releases 2 years after your code shipped through QA!
(and yes new CPU designs can cause problems in any code! But Intel and AMD work hard to avoid the major cases!)
Because as soon as you've found issues with more than, say, 3 things, you no longer have enough of your own free time to volunteer to solve the problem in a better way, let alone whatever you were already working on. Do you honestly believe that criticism has no value?
Furthermore, the conversation at issue was initiated by a community member asking why Weave's authors chose to implement their own security mechanism. The point of this kind of question is to assess whether the authors had good reasons, bad reasons, or no reason at all behind a questionable decision. This helps determine whether the effort to resolve the issue would be well-spent. If the authors aren't convinced that other solutions would be superior, they may be unwilling to accept a contribution, and you are potentially wasting your time producing a patch.
This isn't to say that there's never room for improvement in the crypto space - I personally disagree with the assertion that rolling one's own crypto is inherently bad in all cases, and instead believe that we need a maximum of innovation attempts now so that they can be evaluated and audited and identified as useful - but unless you're actually fixing a problem, Not-Invented-Here syndrome is dangerous and a waste of time better spent elsewhere.
I understand you're trying to "get things done" but crypto is an area where you have to tread carefully, and talking down or ignoring people trying to inform you about security flaws is only encouraging the development of insecure software.
How about a different explanation: After answering time and time again on twitter he found out he had other things to do and played the "show me the code" card?
And yes noobs[0]: Show me the code is a valid card in programming discussions.
[0]: here I am purposefully rude, feel free to take offense if you think it helps - or feel free think twice or even laugh with me.
Open source and computing culture has to a certain degree been a safe haven based on technical skills. Lets try to keep it that way as long as possible, shall we?
If they did it competently, there is no reason they couldn't implement their own crypto encapsulation.
As far as why they rolled their own... have you ever actually tried to use IPSec? It's a usability nightmare. It's also problematic in containers due to container permission issues. I suppose they could have used DSSL (datagram SSL) but that'd probably add more overhead than what they did.
I see little real world evidence that this "let the pros handle it" attitude toward crypto is helping.
"Use SSL, don't roll your own!"
Then we get BEAST, CRIME, Heartbleed, etc., and we discover that the dominant SSL/TLS implementation is a rat's nest of comically awful code:
Look at the older posts for LuLz like: http://opensslrampage.org/post/83007010531/well-even-if-time...
I wonder just how much scrutiny IPSec implementations have gotten, especially since it's such a usability nightmare that nobody uses it.
Even if they have bugs (it is unlikely they don't), they'll still have less bugs than own implementation.
If you are still not convinced, at least give user option what to use.
curl[1] is a good example of giving freedom to the user.
Are you aware that we did not roll our own crypto? Instead we used the NaCl crypto libs[1]. Weave adds about 300 LOC to integrate NaCl.
You can read about it here - http://docs.weave.works/weave/latest_release/how-it-works.ht... You can also read a bit about weave crypto in the comments from mradestock and msackman elsewhere on this page.
I would be extremely grateful if you could provide actionable advice (or help) on which other crypto libraries could fit our requirements for weave. Please note that in addition to functional requirements, any library must be open source, hard to misuse, easy to package, and demonstrably safe.
alexis
this is a bit of a facepalm whenever i hear this.
From the weave crypto docs at http://weaveworks.github.io/weave/how-it-works.html#crypto:
"The public key from the remote peer is combined with the private key for the local peer in the usual Diffie-Hellman way, resulting in both peers arriving at the same shared key. To this is appended the supplied password, and the result is hashed through SHA256, to form the final ephemeral session key."
Disclaimer: I work on Weave.
Furthermore, it's not clear to me what the use of the diffie-hellman is actually for. Perhaps I'm misreading or the linked document is an oversimplification, but... It appears that the public DH key is transferred without any authentication.
If the public DH key is transferred without any authentication, it's trivially MITMable and serves no purpose whatsoever. It's true that mixing in the password later solves MITM at that point, but... yeah: All of the privacy and integrity you could produce with the system described is what comes from the password.
What is the weakness with this approach? Is your point that there's nothing gained from doing the DH - you may as well just generate a secret key randomly, exchange that between the peers in the clear, and still have them combine that with the password secret, and the level of information exposed is the same?
I think there is a difference. Say you have a captured stream of traffic and you want to decrypt it. With the simpler non-DH scheme, you already have the basic key, so all you need to do is guess the password, and run that through sha256 and then you can decrypt the entire stream. Now you have the password, you can decrypt every stream you capture from now on. But with the DH scheme, you have to both guess the password, and the ephemeral private keys which are never exchanged (either that, or break Curve25519). Even once you've done that, you only get access to that one captured stream - sure, you now know the password, but every connection between weave routers will use different random private keys to their DH, so you'll still have to brute force those for every stream you capture (or, again, break Curve25519). So ISTM there is a substantial difference there. That, to me, is the point of using the DH. But maybe I'm missing something... I'm slightly wondering whether people here are considering the use of DH and Public Key crypto by weave in the context of the usual "generate keys once and save them". This is just not the case in weave - weave generates fresh public and private keys for EVERY connection between routers.
There is no requirement for weave that the password is human readable. It can be supplied through a file, so you can happily dd if=/dev/random of=/my/weave/passwd bs=1k count=1 to create a suitable weave password
Turns out that feature I was thinking about has been removed, so the above is not true.
Though a representation in a particular character set does not prevent the password from being strong.
But what can an adversary learn from doing so? All subsequent messages on the connection are encrypted with the secret key, which has the password mixed in.
That is correct. I guess calling this a 'password' is perhaps misleading in our docs, since it could be seen as implying human-readability and cryptographic weakness. As msackman says, the 'password' can in fact be as strong as you like.
weave has lots of very happy users who find that weave is plenty fast enough for their purposes, see eg http://blog.weave.works/2015/02/24/get-your-kicks-on-cloud66...
the strong points of weave network, as it is right now, are ease of use (not to be sniffed at), and enormous flexibility. it is really quite easy to create an application involving containers, that runs anywhere and does not commit you to specific architectural choices...
typically though, one weave network might be used by one app, or just a few. but you might run a lot of weave networks
weave works very nicely with kubernetes - later I shall dig out a few links for this
in our own tests, throughput varies by payload size; we tend to think of weave-as-is is best compared with using, for example, amazon cloud networking directly
for users with higher perf needs, we have a fast data path in the works, that uses the same data path that ovs implementations use... the hard problem to solve here is making that stuff incredibly easy and robust w.r.t deployment choices -- see above :-)
What I may be failing to communicate here is how trust is established between entities like companies and entities like individuals. When there is a issue with trust with individuals, like in the OPs post, it's best to establish 'point-to-point' communications with people you know so you can build up the trusted relationship. Doing that as a company doesn't really work well for that.
BTW, comments like "weave has lots of very happy users who find that weave is plenty fast enough for their purposes" are implicit trust statements based on bandwagon bias. What you are actually saying is there exist a group of people who don't feel the way the poster feels and are happy with the product's current state. The implication of your statement is that others should feel this way, but there's really no way to establish that unless we heard from all those people directly. This is yet another example of why consensus sucks when trying to establish the truth for an individual. (Bitcoin has figured this out, however.)
I think about trust a lot for work, so take my comments with a grain of salt. Nobody died here. :)
That just means that your users aren't using it for large production workloads or they are just wasting excessive resources to make up for it.
Fast enough is fast enough, as they say...
I use Weave as the default SDN for Clocker [1] because of this simplicity, and also because it is server-less and will work on a single server Docker cloud or a cluster of tens of machines without having to think about architecture. Of course, Clocker supports pluggable SDN providers so if your networking demands are not met by Weave you can change to another provider.
I don't think Alexis (or any other Docker SDN provider) is suggesting that their software should be used for low-latency microsecond sensitive trading applications. You have to use the right tools for the job, and in this case Weave's sweet spot is its simplicity and reliability.
I did pop into their IRc channel a few times with a question around this, but over the space of 3 days all activity on the IRC channel was tumbleweed and crickets..
on each host? what kind of physical network are you using, and how many containers per host? how many hosts? let me know if I should email about this instead.
Either way, you're an ideal candidate for closed beta access to the performance focused container networking tools my startup is developing. I can contact you via email if you're interested.
./bin/clocker.sh jclouds:provider:region calicohere is a kubernetes link: https://github.com/GoogleCloudPlatform/kubernetes/pull/4483#...
the author, who was part of the MS Azure team at the time, said: "as of today, this Weave/CoreOS tool and doc is the only way I was able to provision a Kubernetes cluster in Azure"
SocketPlane was bought by Docker and is now DEAD, to become whatever Docker decides is good enough to silence their critics. "See we bought these network guys, we swear we're taking networking and plugins seriously!"
To be clear, there is nothing fundamental (to the best of my knowledge) that would prevent Weave implementing a VXLAN backend. It is simply a poor implementation at the moment, and even then, only in this regard. Who knows, maybe Weave's use of a gossip protocol vs flannel's CP etcd could make it more suited to some deployments. Though, that does need to be tested.
Is this added latency just VXLAN overhead, or is flannel userspace still involved in per-packet-processing? I'd love to see a test comparing flannel/VXLAN to a manually configured VXLAN tunnel.
(I'm off to read the flannel code...)
1 - http://dpdk.org/
You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.
This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.
Or is there something fundamental that I don't understand? Please edify me.
I agree that if you are running on AWS VPC or some other overlay you should just use VPC for container networking. You shouldn't overlay your overlay. But there isn't any tooling that I know of to do that.
Even if one runs a Docker bridge in his development VM, that doesn't one must do so in production as well.
Are we in this mess because production engineers don't understand networking?
It's only a matter of time before they do, until then...
Here's where I got burned: I set up an elastic search cluster using containers and weave and life was great, but it then grew to need another node. Upon setting up the new host with docker and weave it turns out the new node couldn't talk to the old nodes because they were using different versions of the weave protocol. That was disappointing and a pain, I stopped experimenting with weave at that point.
It should be fairly straightforward to deploy the same version on every host, but maybe that wasn't explained well enough, or didn't work for you. We'd welcome more feedback.
Lastly, I appreciate the positive comments. "Very very easy to use" is exactly what we aimed for.
Although etcd had a lot of issues even when I was in the evaluation stage... I setup a dev cluster, having a node running on half a dozen machines (developer workstations included) in the office. The etcd cluster broke 3-4 times before I abandoned the thought of using it. With the 2.x release, my test network (3 virtual servers) was a lot more reliable, but I decided I didn't need it that much, opting to use my cloud provider's table solution for maintaining my minimal configuration...
For my deployment, the cluster is relatively small, 3 nodes mainly for redunjdancy... our load could pretty easily be handled by a single larger server. That said, I decided to just use dokku-alt for deployments directly to each server (each service with two instances on each server). To make deployments easier to use, I have internal dns setup with wildcards using v#-#-#.instance-#.service.host.domain.local as a deploy pattern for versioned targets and instance-#.service.host.domain.local for default instances. I have two nginx servers setup for ssl, spdy termination and caching configured to relay requests to the dokku cluster for user-facing services. This is turning out to be simpler than implementing etcd/flannel for that communications layer.
Each instance is passed relevant information regarding the host/service information to report it's own health to table storage, internal communication is using ZeroMQ req/res interfaces to minimize communication overhead, and allow for reconnects. Which is working a little better than relying on WebService/REST interfaces internally.
The big question that a system architect needs to ask, if they are designing a system at scale is not "should I use this technique" but "do I NEED to use this technique." We can always add more complexity and technology/layers than we need because we "may need it" in the future, and we almost always end up with a jinga tower when we are done.
So, when laying out your infrastructure, be sure to know what your actual requirements are, and don't add a lot of extraneous capabilities that you have to maintain and trouble-shoot later.
IPV6 was supposed to solve a lot of this by having an address space so huge you could easily give every vm host a few billion IPs. But nobody uses it.
Virtual Machines have a smaller attack interface to the host, so they do have a chance of containing malicious code.
Kernels were supposed to contain processes, providing them with a virtual private memory space, virtual storage access so they can't read or write any file you don't want them to, etc... but only openbsd is really trying to make that work.
A lot of the problems with regards to containment have already been solved in different systems. I believe solaris or opensolaris had Crossbow [1]. Any system that aims to provide connectivity needs to do the least amount of encapsulation possible and probably be a kernel module.
1. http://en.m.wikipedia.org/wiki/OpenSolaris_Network_Virtualiz...
What does that term even mean? Are you talking about encryption? If not, there is no 'strength' to encapsulation. Something is either encapsulated efficiently or it's not.
[1] https://www.joyent.com/blog/triton-docker-and-the-best-of-al...
[2] http://fingolfin.org/illumos/talks/sdn.pdf
[3] https://www.youtube.com/watch?v=mLvi3nPNqiQ&list=PLH8r-Scm3-...
If you want a real ip/ipv6 stack for your container, try unikernels.
We have been building out a series of networking tutorials at flockport from basics; static, public, private IPs, NAT, bridging etc to multi-host container networking with GRE, L2TP, VxLAN, IPSEC focussed on LXC but these will work with VMs and containers in general.
They don't need any special tools, just IP tools and the kernel and deliver performance and security. A lot of the Docker centric networking projects use these under the hood but they are easy enough to use on their own.
In the table, those latency numbers are specified in "ms". Also microseconds? Couldn't possibly be milliseconds, for two VMs that should only be a couple dozen metres away from each other, right?
char *tab[] ={ "ns", "us", "ms", "sec" };
Now I'll just be waiting for someone to complain about inconsistent precision. I am not a good scientist...