Service Weaver: A Framework for Writing Distributed Applications | Hacker News Reader

Service Weaver: A Framework for Writing Distributed Applications

opensource.googleblog.com

117 pointsdavidgomes39 comments

> microservices severely impacted our ability to make cross-binary changes. It made us do things like flag-gate new features in each binary, evolve our data formats carefully, and maintain intimate knowledge of our rollout processes. Finally, having a predetermined number of specific microservices effectively froze our APIs; they became so difficult to change that it was easier to squeeze all of our changes into the existing APIs rather than evolve them.

I'm amused by how the rise of microservices was in part due to the promise of solving some of these problems as they arose in the classic monolith. Independent teams, decoupled deploys, etc.

Putting a network request between components doesn't decouple them, it just trades one kind of coupling for another. Even worse, some previously explicit coupling becomes hidden, but remains present.

OK.

We've been here before, the "transparent RPCs" path.

SunRPC, Mach Messages and MiG, various even more transparent distributed object systems etc. etc. etc.

So it would be awesome to have some reference to these earlier systems and how this project overcomes the problems they encountered. I checked the FAQ ( https://serviceweaver.dev/docs.html#faq ) and didn't find anything.

Or a brief explanation why this is so incredibly different that those issues don't apply. Or a brief explanation that those issues weren't problems and everything is just dandy. Or even we didn't consider those systems at all.

Just something.

Cheers!

"Transparent Networking" achieved. To reach the next level you should unlock an "Actor Model" :)

PS for me, it's more like reinventing Erlang ideas in Golang. If you want to cut the corner here is the ready-to-use Framework in Golang https://github.com/ergo-services/ergo - implements all Erlang' neats.

I always dreamed of something like this where functions could be called as normal but they could be an RPC behind the scenes. The compiler would take care of serialization/deserialization and routing.

But how is it possible to not worry about network. Every function is now able to fail. Why don't you need to handle this explicitly? Or is there just a default behaviour that can be overriden?

Edit: after reading the docs I think I understand a bit more. You'll have have to deal with network errors on anything crossing module boundaries.

You do have to worry about the network when you call between components. The documentation talks about this explicitly in "Semantics":

"Method calls are executed with at-most-once semantics. This means that Service Weaver does not automatically retry method calls that fail. However, you can detect and retry failed method calls explicitly using weaver.ErrRetriable. If a method call fails because of a transient system error (e.g., a component replica crashed, the network is partitioned), it returns an error with an embedded weaver.ErrRetriable."

With the following example:

    // Retry the cache.Get method call up to five times.
    var val string
    var err error
    for i := 0; i < 5; i++ {
        val, err = cache.Get(ctx, "key")
        if errors.Is(err, weaver.ErrRetriable) {
            // Retriable system error! Retry.
            continue
        }
        break
    }

[The docs at https://serviceweaver.dev/docs.html are fairly comprehensive, kudos to the team!]

Thanks for this as I was also wondering about network partitioning, and all those failure fun and games that distributed systems must handle.

This golang package helps to remedy that by providing a bus / messsge queue that can run in process or over the network.

So you get rpc without thinking about it and failure is managed by the data plane forwarding the message when the network is healthy again.

https://github.com/suborbital/e2core/blob/main/foundation/bu...

This however forces all other code using a package like this to adapt to the realities that you’re in an async non blocking world now. A rpc call will give you a response eventually.

So the calling func and all parent funcs will just have to wait around when the network is unhealthy?

I think it's safe to say that this doesn't go as far as the dream. In addition to functions failing, you can't share memory between functions etc.

With Service Weaver, you have to be aware that you're writing a distributed application, but you get to write it as a single binary. So it's not as magical as it can be, but it improves on the status quo.

You're dreaming of Erlang and it's brilliant use of actors and supervisors to manage failures to make it work.

That raced through my mind as I read their docs also.

https://github.com/orgs/ergo-services/repositories Is erlang written in golang.

The examples are great and clearly shows what it offers in terms of failure scenarios that it handles. Supervisors tress etc.

Weaver with ergo would be a nice mix ?

You don’t specify your physical topology in code but instead at deployment time. Weaver gke but written for ergo where the Trees are decided at deployment time perhaps.

In the old days, Microsoft encouraged people to do exactly that with their COM based applications, where a software component could be transparently migrated to another machine via DCOM. The original application needn't care where the component was now running since it's location in the universe was controlled by the registry. You could now scale your big fat client app out to many servers. What was once slow and bloated on the client machine could suddenly become fast and nimble on the 'high powered' server(s).

Needless to say this caused, firstly much excitement among VB devs, and selling of Windows NT 4 Server licenses, soon followed by consternation and disappointment.

Turns out: - software design of components running locally, and components running remotely should be approached in veeery different ways. e.g. making 100 property assignments on a component running in your local process space is just fine, but doing that on an object that exists on a server 100 miles of copper away has very different performance characteristic. Apps were suddenly unresponsive in many different and frustrating ways.

- massively increasing the network load as storms of calls and responses to the servers just to display a form, and big round-trips of data that never existed before strained existing infrastructure. E.g. previously the db would fill the GUIs data grid directly with a query. Now... the gui would call the server component, which would get the data, then it would serialize that data into a different form (for ADO) and send it via DCOM to the GUI which would show 10,000 rows. All very slowly.

- massive increase in load on the domain controller, as GUI apps happily created and destroyed objects that used to be local, now they resulted in a DCOM call to create the object which required security checks that were never needed before. Saw this bring domain controllers to their knees.

- As you said, applications could suddenly fail in new and exciting ways since they're now on the network and that entailed a whole new set of failures to be aware of.

- massively increased complexity in installation and configuration, since every little service had to be configured and installed in its own unique way. Security also became harder since permissions per-service (e.g. db permission, filesystem permissions) also had to be managed.

Microsoft tried to help with MTS (their transaction service to host DCOM components), but by that time (early 2000's) people were already exploring web applications as the replacement for fat client apps and migrating away from this approach.

>I always dreamed of something like this where functions could be called as normal but they could be an RPC behind the scenes. >The compiler would take care of serialization/deserialization and routing. You can still do it today if you want, just investigate COM+ if you're on Windows. It's not the compiler that decides whether RPC is needed, but the COM runtime (via the Registry). It's all still there.

Erlang/Elixir does this!

There has never been any issues with hiding distributed RPC calls.

https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

This reminds me of "Distributed Applications" in Erlang (https://www.erlang.org/doc/design_principles/distributed_app...).

This is... EJBs. It is probably less painful to use, but the model is pretty much the same. Fascinating.

I was thinking the same thing, which leads me to say "This is where I came in." That is to say, I have seen what happens next. Maybe it time for retirement.

Man I was thinking the exact same thing. We have looped around back to J2EE EJB's. Wow, so Google copies Enterprise Java! Kinda funny to see the old stuff re-branded as the shiny, new stuff - albeit in another language.

EJB3 wasn't that painful, it was actually quite nice. Ran one of the largest porn sites out there on JBoss4 tech and it worked really really well.

Hah, wave. That is a name (latchkey, not jboss) I have not heard in a very long time :-)

I was thinking of the EJB 1 & 2 era xml wiring of all the things.

hi brian! long time as well. i hope you're well. =)

Any links that described how they scaled their services?

3 medium sized dell frontend servers running jboss4 and one beefy backend mysql server. we ran our own bare metal because this was before 'cloud' services would allow porn sites to operate.

the ejb3 caching was so efficient that we really only needed one server, we just had the other two as backup so that we could do CI driven rolling deploys integrated with the load balancer. we used jgroups mcast to expire entities in the cache. jvms were carefully monitored and all the settings were heavily tuned for our environment.

the hardest thing about all of it was simply packaging it all up into a war file correctly. it is all knowledge that i've long since forgotten how to do.

Using the term microservices here is confusing. The main point of microservices as an architectural pattern is decoupling release cadence between teams in very large organizations. The scaling/redundancy part is not unique to microservices. What this framework seem to be doing is allowing a team to deploy a system developed as a single binary monolith as a distributed system.

The organizational use case is orthogonal to what microservices were all about.

Granted, microservices have been cargoculted like crazy, so this distinction is probably lost on a lot of engs. But for those of us that remember the original meaning, mentioning microservices in the description of this framework, is odd.

Super cool. If I understand correctly:

- There's no IDL file, like a .proto file. Instead, weaver looks for marker interfaces by embedding weaver.Implements[T].

- Weaver interfaces must be serializable, which explicitly support protobufs. Was the intent to be able to port gRPC services to weaver?

- Weaver maintains a list of deployments, and each deployment has components. Components may only communicate with components belonging to the same deployment. Sounds like a way to implement atomic deploys.

- Named listeners are mapping over a net.Listener?

Sidenote: I'm going to steal the metrics implementation. I haven't found a lightweight metrics implementation for GCP.

This is a great summary.

Some comments:

- Service Weaver implements its own serialization mechanism which is very efficient/achieves high performance.

- You write an application as components (where each component is defined by an interface). You can deploy the same application using different deployers: (1) on the local machine as single/multi process; (2) on GKE and we also have an experimental (3) SSH deployer that allows you to deploy across a cluster of machines reachable over SSH.

- However, the abstractions around deployment are agnostic to the deployment environment (local, cluster, cloud), hence someone can improve/write new deployers without deep knowledge of Service Weaver internals.

One of the reasons I break code down into smaller pieces ( microservice ) is to have fast edit and compile times.

The versioning hell is why so many went to mono repos. There is only one git version and that’s it.

Also what about runtime version difference and schema evolution?

So many thing come into play.

I would rather just buy the bullet and use protobufs with NATS from day one . No load balances and heavy expensive gke / k8 stuff

Then deploy of fly where I have regions with auto scaling based on metric feedback loops.

Then use nats client on all client to get geo physical load balancing out to the nearest region.

Then use their postresql multi Region. Cockroach multi Region replication costs big money.

I did like their tooling in Weaver though. Even the converged logging was pretty nice.

https://github.com/ServiceWeaver/weaver/blob/main/runtime/lo...

Microsoft Orleans is a really mature version of the actor model for the .NET stack that operates just like this. And the key difference is that it's used by Microsoft themselves in huge systems like Xbox Live.

Is this currently used at Google?

Unison lang has a lot of overlap with model: https://www.unison-lang.org/learn/the-big-idea/

Is it correctly understood that this is useful for scaling the compute of a single logical application, probably maintained by a single team, and less for communicating across applications, where a traditional RPC or REST API might still make the most sense?

Yes, that's pretty spot on. Though we do have plans to add cross-application support as well, at the minimum stub generation and likely service discovery. But some of the nicer properties like versioning and language-native types will likely be lost on the cross-application communication path.

I was wondering about Service discovery

Is there any suppptt for Publishing an event to all subscribes from a Module so all other modules that are interested could get that event payload ?

unfortunately the Service Weaver Google Group seems highly moderated (my message didn’t go through). Does anyone who’s tried SW over the weekend want to join #serviceweaver on Gophers Slack? I have questions. :)

How does this complement/differ from Istio?

Developers are too lazy to write specs for API, so there's this framework ?

Now the function call will have this kind of signature:

func Add(apiKey, apiSecret,...) {}

Helps you make spaghetti systems

Is this only for Go?

We provide only Go support for now. We're still early and do plan to support other languages in the future.

Until then, "A Go Framework..." would be a more appropriate title?

For sure, the GitHub page should label this as a framework for Go until there's anything substantial to allow running this for non-Go apps.

Reply on news.ycombinator.com