Hacker News Reader

Comment by ram_rar | Hacker News Reader

I feel so old now. There was a time, when I used to discuss with senior engineers @ Yahoo! to use NginX over Apache. Nginx was the hot thing, popularizing C10k [1]. Now in my current team, I have junior devs in my team pushing for Envoy over HAProxy/Nginx setup.

Is this trend happening primarily because devs are pushing for GRPC over REST? What benefits does Envoy offer over Nginx, if you're still a REST based service. I am not fully convinced of operational overhead that NGINX brings.

[1] https://en.wikipedia.org/wiki/C10k_problem

View on HN

The sibling comments point towards the difference in configuration if you take the "out of the box" product. But there is also a vast difference in how code is organized, in case you ever have to touch it.

From my point of view Nginx feels "old". It's a C codebase without a great amount of abstractions and interfaces, and instead having a bunch of #ifdefs here and there. Unit-tests and comments are not to be found. Build via autotools.

Envoy looks as modern as it gets for a C++ codebase - apart from maybe the lack of using coroutines which could be interesting for that use-case. It uses C++14, seems to be extremely structured and documented, has unit-tests, uses Bazel for builds, etc.

So I think the experience for anyone being interested in working on the code will be very different, and people that prefer the style of project A will have a very hard time with the style of project B and the other way around.

> uses Bazel for builds

Is this unanimously good? I've heard both praise and horror, never used it myself.

It's good for Dropbox, since they use Bazel.

One of the senior engineers once said to me that "Bazel is like a sewer: you get back what you put in."

Bazel requires a lot of upfront effort but the power of (a programmatically accessible/modifiable) dependency graph and a common build/test system across all the languages is very hard to underestimate.

> very hard to underestimate

Are you sure?

I looked around at the code in Envoy.

"As modern as it gets"? Very, very far from it. Everywhere I looked it was all-over public virtual functions. It looked, more than anything, like Java, which is essentially, more or less, C++92 with some bells on.

The code might be OK, but, as with typical Java code, everywhere I looked was boilerplate, hardly any of it doing any actual work. I would hate for somebody to look at Envoy and think that was what good, modern C++ code looks like.

Virtual functions are a good answer to certain problems that come up, once in a while--in C, for such a problem, you would use function pointers. Inheritance is a pretty good answer to certain problems that come up a little more often.

But neither is a good answer to any organizational need, and a big project that reaches for virtual functions and inheritance as first resort makes me shiver.

The operational overhead shifts to more API stuff, so people can write 100 lines of code instead of modifying 1 line of config, it feels like.

This is never going to end as more things shift towards being core APIs that allow you to write code instead of configure things. It's not even configuration-as-code, it's just code managing configuration files.

edit: I think my comment comes across maybe kinda rude. My beef with Envoy is that the documentation is _extremely_ complex, and I've repeatedly asked 'How do I get started with xDS?' and been pointed to the spec, which I think took some time to read through and when I asked others about how to setup LDS/RDS/CDS/SDS was met with a like 'what are these things...? just use xDS,' which led me to a lot of frustration. This has been my experience each time trying to approach Envoy, and xDS.

I think the problem with xDS is that their example go-control-plane repository is completely useless. It's overly complicated with frightening-sounding details that don't matter to someone experimenting ("you MUST MUST MUST CACHE THIS how to do so is an exercise left to the reader").

I ended up reading the specs and found them very clear, and wrote my own xDS implementation: https://github.com/jrockway/ekglue/blob/master/pkg/xds/xds.g... I did this after reading the source code for the most popular xDS implementations and finding myself horrified (you know the popular xDS implementation I'm talking about). Now I have a framework for writing whatever xDS server I desire, and it can be as simple or as complex as I want it. For example, for my use cases, I'm perfectly happy with a static route table. It is very clear what it does, so I have that. What annoyed me was having to configure the backends from Kubernetes for every little service I wanted to expose to the outside world. So I wrote ekglue, which turns Kubernetes services and endpoints into Envoy clusters and Envoy cluster load assignments. This means that I never have to touch the tedious per-cluster configs, and still get features like zone aware load balancing. And I don't have to take on complexity I don't want -- the woefully under-specified Kubernetes Ingress standard, service meshes, etc. (I also plan to use ekglue for service-to-service traffic because xDS is built into gRPC now... just haven't needed it yet. It's great to use the same piece of software for two use cases, without having to maintain and read about features I don't need.)

TL;DR: take a look at the spec. It's really well thought out and easy to implement. Just don't cut-n-paste from Istio because they got it really wrong.

On that note, the grpc spec specifically calls for load balancing that doesn't actually do the proxying, but instead hands out assignments, with the server passing it's current load back to the load balancer service. it sounds like in this case the grpc client is using some array of xDS, but the server is using xDS along with...?

I feel like xDS is a relatively new addition to gRPC. I think there is another parallel implementation inside gRPC of external load balancing, which may convey server load information back to the gRPC client.

I looked up the current state of the xDS code, and there's a lot more of it than I remember. The EndpointDiscoveryService based gRPC balancer is here: https://github.com/grpc/grpc-go/blob/master/xds/internal/bal.... It appears to balance similarly to Envoy; locality-aware with priorities.

(That doesn't surprise me because I don't remember any field in the ClusterLoadAssignement proto that sends load information back to the client. Health, yes; load, no. But I could easily not remember it being there because it hasn't been something I've tried to implement.)

But yeah, the way to look at endpoint discovery is like DNS. DNS can return multiple hosts, and clients will spread the load by picking one at random (sometimes, if you're lucky). EDS is similar to this, but is a streaming RPC protocol instead of connectionless UDP, so it's theoretically easier to operate and monitor.

The other xDSes do more things -- CDS lets you discover services (that EDS then gives you endpoints for). RDS lets you make routing decisions (send 10% of traffic to the canary backend). SDS distributes TLS keys (and other secrets). ADS aggregates all of these xDSes into one RPC, so that you can atomically change the various config versions (whereas requesting each type of stream would only be "eventually consistent"; doing it atomically is good where route table changes add new clusters, the client is guaranteed to see the new cluster at the same time the route to that cluster becomes available).

It is all somewhat complicated but very well designed. This reminds me that I want to look more deeply into gRPC's support of xDS and add some integration tests between gRPC and ekglue.

Yep, gRPC is the new toy for distributed computing, after everyone realised that DCOM, CORBA, RMI, Remoting actually made sense instead of parsing XML and JSON text formats all the time.

I had to chuckle as well when I read that article and the part about gRPC. Seems like the pendulum is swinging into the other direction again - back to where we've already been ten or twenty years ago. New name of course, but same concepts.

One really starts to feel old at such occasions.

It certainly does look like that. I do think though that we've learned a number of central lessons in the process:

- treat messaging as a first class concept, not something to hide & abstract away.

- do not attempt to implement polymorphism in a messaging protocol. Do not bind your messaging protocol to a programming language's type system (they serve different purposes).

- bake fundamental monitoring & maintainability concepts into the protocol (e.g. intermediaries must be able to understand what responses are errors).

- have a well understood, simple backwards and forwards compatibility story.

- etc.

All of this is stuff we didn't understand in RMI or CORBA or SOAP etc. REST was a great wakeup call, both in simplicity and some of the messaging protocol concepts (such as error modelling). It is missing the application level binding - there's just no good reason why you wouldn't have a statically checkable method/request/response type binding.

I am a bit weary on whether gRPC will go over board again in complexity. We'll see.

Sure we did understand it, that is why they used an IDL. independent of programming language's type system.

Apparently DCE IDL now comes in proto files.

What newcomers did not bother to understand is why we were using those formats in first place.

Rest assured, maybe in 20 years we will be introducing this cool RPC protocol, based on YAML on something. Thankfully by then I should be retired.

I think IDLs are an important point among many others (I attempted to list some above). I think we might be missing substantial improvements if we'd just say "oh so we're back to a static IDL description, all old is new again".

And even within IDLs, we've made major progress. Compare the mess of SOAP's data type system, various attempts at inheritance and polymorphism in SOAP and CORBA, pointers in CORBA etc.

What's the beef with polymorphism?

I read that part as:

Protocol Buffers are good enough to make us forget the traumas caused by CORBA.

Totally get it! The team (@veshji and @euroelessar) struggled a bit in convincing me that the new Envoy way is a simpler one. I do not regret giving in.

Operationally, there are many differences (esp. around Observability) but if I were to distill it down to one thing it is a clean separation between data- and control-plane. This basically means that it was designed to be automated and the automation layer (xDS) itself runs just like any other normal service in production.

This is just the software industry. Maybe it’s because we’re so young. Maybe it’s because software is relatively easy to change and experiment with.

Who knows. All I know is, it’s exhausting, and ultimately it’s terrible for the end user. We have no idea what we’re doing when we pull in a new dependency like this. There’s tiny corner cases we don’t think about, and those get passed on to the user.

Innovating is fun, but exhausting in aggregate.

Envoy is lot more configurable and rivals nginx on performance (especially throughput). Codebase is a lot more manageable (but that’s my personal preference). Runs circles around nginx on observability features.

Reply on news.ycombinator.com