Envoy: 7 months later | Hacker News Reader

Envoy: 7 months later

eng.lyft.com

176 pointsmattklein12346 comments

There was a very interesting talk at Google Next in March that described a way in which the Kubernetes open source team is bringing this sidecar architecture to Kub. In fact I think they are working with the Lyft team. The whole talk is interesting but I've added a time offset to skip to the bit where they talk about sidecar/service mesh.

https://www.youtube.com/watch?v=3quCoi5YHz4#t=27m59s

Also, there is https://istio.io/ but it's a placeholder site at the moment.

You can find the project here https://github.com/istio/manager

Looks interesting except imo the configuration makes it basically unusable.

They use jinja2 templated json... and there are hundreds of lines for their "simple" examples.

Even the "hello world" type example of proxying to google is a mess https://github.com/lyft/envoy/blob/master/configs/google_com...

The example you linked isn't so bad. Perhaps a bit verbose for a simple example. The formatting of the json is ugly though, and I think they would do better to not collapse the whitespace inside arrays.

Seems like they could have used something a bit nicer to work with too (yaml, libucl, hocon, etc). I am honestly getting tired of editing fiddly json (no comments, last element in list can't have a comma, etc) all over the place.

You can use hjson for human editable json.

http://hjson.org/try.html

Human editable json, with comments, and doesn't kill you for a lack of trailing comma.

Available libraries in tons of languages.

That actually looks pretty nice. Thanks for the link!

Given Google's involvement, I am wondering why they did not choose jsonnet?

Hopefully, it was avoided on purpose because it's a complete nightmare to use.

Honest question: How so?

I have not used it in anger - just played around with it.

Too much indirection.

You get big files filled with variables, loaded from other big files also full of variables. It's hard to know what you are setting after a while or where it comes from.

After a while playing with configuration systems, I'm starting to think that configuration should just be hardcoded once and for all.

I would rather contact you directly but your profile is empty.

I would like to introduce you to HiveMind described here: http://blog.crudzilla.com/2016/10/managing-configurations-wi...

It addresses the issues that JSSONET does without inventing a new templating language.

I am happy to give a skype demo, email me if interested.

I love how history repeats itself.

I remember everyone complaining about Spring and similar having these massive XML configuration files, and then people said "never again!", and moved to very simple JSON formats for their frameworks. And then everyone adds a little bit to the best practices, and over time we get this, again.

Seriously. How about we just call it EJB 4.0?

JSON Server Faces™

That's how they use it, that's not how you have to choose to use it. It is one of the most tedious aspects of the tool but all the different aspects exist for a reason, there just needs to be an easy mode for people starting off.

This is a battle that repeats itself ever so often. Same deal with kubernetes config files.

It's the same opinionated positions of puppet vs chef/ansible. Building the tooling in the form of an embedded programming language (lua for nginx, python for ansible) is so much better.

But that is a much more polarising position to take. It's politically untenable. Ruby engineers will say no to python and vice versa... So you start arriving at Json or XML based configurations which are a programmatic neutral ground.

IMHO lua-jit occupies the same political neutrality as XML.

I tried to give it a try, and gave up for now.

1. quite hard to "give it a spin" without using docker (vomit). I eventually found some random github repo that had centos7 compatible build scripts, and it took quite a while to compile it on a test vm since it had to build gcc and a ton of other stuff. Not sure why they can't just supply a static binary "release".

2. The config really is a mess. I managed to get something almost working (only / requests went through the proxy, but nothing else for some reason, even though I was using a prefix config). There aren't really any good example configs I could find either. There is a weird config builder script with json templates in the configs dir, but it just seemed to spew out some preconfigured madness.

I am sure it works great and has more tunables than you can shake a stick at, but for now it seems to be only usable by people who have lots of time to invest in trying it out.

Envoy is an extremely exciting project. I've been following it closely since OSS inception and am thrilled to see the upcoming deep integration with the Kubernetes project. Such a natural and complimentary fit!

With flexibility as both a side car and host proxy network call fabric for polyglot distributed services, I see Envoy as a kind of modern ESB for the reincarnation of SOA we are seeing in the microservices movement. Congrats to the awesome contributors from Lyft/IBM/Google and others!

Interesting that it seems like there is a bunch of overlap with linkerd in the k8s space. With both k8s and linkerd being CNCF projects, I wonder why google is putting so many resources behind envoy rather than linkerd. I guess that's the beauty of the k8s ecosystem though, freedom and choice to use what you see fit. K8s just provides the rock-solid primitives.

linkerd has a dependency on the JVM, k8s doesn't, and not everyone wants to pull in the JVM if they're not already using it.

I guess that's one the reasons they created https://github.com/linkerd/linkerd-tcp

One downside to using Linkerd is that unless you have very few apps — or an overabundance of RAM — you can't realistically deploy it as a "sidecar container". The JVM consumes about 100MB of RAM, by their account; it's not lightweight.

The sidecar pattern (used by Envoy and by Google's new project, istio) is desirable in that it compartmentalizes the rules and logic in the pod that uses them, as opposed to sharing all settings across all apps. Zero dependencies (between pods) is a great way to manage complexity. I haven't thought much about it, but there might be latency benefits, too.

Are envoy and linkerd comparable? They look like they fill similar niches, but I'm not familiar enough to know what I don't know.

The Lyft team cover the comparison in their docs [1].

1. https://lyft.github.io/envoy/docs/intro/comparison.html#id7

This. Linkerd has been a pretty cool piece of tech. In fact, their rewrite linkerd-tcp is in Rust and is blazingly fast. Also, it works on L4 and can do more stuff than Envoy... though I think that advantage is going to be shortlived.

Linkerd guys have been focusing on building deep integration with k8s (as an ingress or sidecar), but this announcement of Envoy trumps it all.

>"We are excited to announce that we are working in partnership with both Google and IBM to bring Envoy to Kubernetes. Fun fact: there are now more people working on Envoy at Google than there are at Lyft! We have a lot of other things planned with Google that we will be able to share more about in the coming months."

That's a bummer for the amazing guys at Buoyant. I had hoped for linkerd to be the next generation of ingress in k8s.

Envoy is in C++, while linkerd-tcp is in Rust. I wonder how much role did the choice of tooling play in Google's decision to adopt.

Isn't linkerd in Scala and linkerd-tcp a small subset of it?

That's correct. linkerd/namerd run on the JVM.

that's true - but i was hopeful !

Also see: eBay/fabio, Traefik.io .

We've been experimenting with Envoy for a few months and have been really impressed with the technology and the responsiveness of the community. We're actually deploying an API Gateway built on Envoy so you can easily extend your service mesh all the way out to the edge. Planning on open sourcing it soon (if anyone wants to give it a try, my email is in my profile).

Any reason you chose to use API Gateway for your edge instead of another Envoy with an ELB in front?

Sorry, I wasn't clear. I was referring to using Envoy as an API gateway, not a specific API gateway product. We had to write some wrapper code on top of Envoy that lets you use a REST API to map URLs to services running in your Kube cluster and then dynamically update Envoy config, etc.

For Googlers out there: Envoy is basically GFE

More like GFE

Wut? More like Kubernetes is like Borg.

Sorry mixed up. Yes, GFE is more accurate

I had no idea what GFE stands for and had to track down [https://medium.com/@jerub/the-production-environment-at-goog...] in order to get a good answer.

Envoy bills itself primarily as a service mesh (seeming closest to GSLB / Global Software Load Balancer), but also as an edge proxy (seeming closest to GFE / Google Front End).

Is there any reason to think that it's more like GFE than it is like GSLB?

Thanks for the translation! :)

I wonder why they chose C++ and not Go.

I presume because the author was more familiar and experienced with C++.

Any numbers on latency Envoy adds for each request ?

according to the talk I just watched, latency/performance was their concern coming out of the gate and was the impetus for writing it in C++...

Thanks.. just tried it out with my service, beats nginx for simple http proxying pretty easily by a nice margin. Envoy supports way more features than nginx too !

Just tried it locally, latency is < 1ms..

An engineering blog without any diagrams?

Reply on news.ycombinator.com