Show HN: Go Micro – A distributed systems development framework | Hacker News Reader

Show HN: Go Micro – A distributed systems development framework

go-micro.dev

115 pointsasim57 comments

Previous discussion: https://news.ycombinator.com/item?id=21878659

Not dissing your efforts, but you keep posting your Go Micro stuff, under different account names (chunk) in Show HN and normal threads. 90% of these posts add nothing and are just duplicates. Maybe add a blog post or something of value instead of the same old link every time.

Thanks for your comment. I post a link to the github repo periodically because the project is moving incredibly fast and actively evolving to become something that's fundamentally valuable to those building distributed systems, cloud services and want to build applications with Go using a framework. People also come and go on HN so don't always get a chance to see it or dont' remember the project. I'm also always actively looking for feedback and so appreciate your comments and will look to publish a blog post next time with valuable content as I've done in times past on the blog https://micro.mu/blog/.

Otherwise as someone who started this project 5 years ago as his first open source endeavour and was really scared about the reaction from a developer community about the quality of my software or whether there would be any interest at all, I'd hope that positive encouragement is something we advocate for rather than negativity. And I hope others are not disuaded by your comment and actively post on HN to get positive feedback on their project!

Otherwise thanks for your comment.

Oh and also now being an old man I wanted to use an account with my real name rather than a pseudonym :)

> the project is moving incredibly fast and actively evolving

Not that I'm hating, but one of the comments above made a valid point that this part of your comment suggests is correct: maybe you should be blogging about progress of the project and gain plenty of eyeballs?

One thing I saw that fascinated me is you guys are using mDNS for discovery, I was debating this discovery approach for a small microservices foundation for D but I couldnt even find a decent library so I havent really attempted much work in that approach. I did something that in a sense feels like a micro service architecture entirely with mDNS but in Python, then months later I worked with microservices in Java and it felt so bloated and overcomplicated compared to mDNS.

Don't apologize for selling yourself. If they're complaining, its working.

Thank you :)

Aka spam

That is very common practice on HN. Submitting a link here is a lottery that one rarely wins.

genuinely curious to know why you created a throwaway account solely to post this?

If you can only confront someone using a throwaway account, you’re probably being unnecessarily mean. Glad they’re trying to get word out there on what they’re working on. Let the voting do the voting. Thank for your OS work OP, upvote from me. Looks like a cool project!

I understand the positive viewpoint, but don’t feel it’s in the spirit of HN to repost your own project every couple months. Back in the day even just posting your own stuff (instead of someone else who found it interesting) was frowned upon.

IMO a culture of curiosity vs blatant promotion is an asset for this board and should be preserved. I counted at least 26 reposts of go-micro. At what frequency should we just consider it a recurring (free) ad?

https://news.ycombinator.com/submitted?id=chuhnk

When has self promotion here been frowned upon? It’s even officially supported with “Show HN” posts.

I don’t disagree but that is the exact purpose of forum software: voting, spam detection, etc. The fact that 99.999% of HN users had no idea these posts were made (most have no votes) shows it’s working correctly.

On the other hand, to a large extent getting traction on HN is a somewhat random process reliant on a few high momentum votes in the early drop. A lot of the stuff we see on the front page is pushed by voting rings. Anyway given this persons hard work on a project and desire for traction, I’d rather they try/learn to market at the expense of a couple of bytes on a DB and a few scanning eyes on /new.

And it seems like this post might be their best yet :)

You’re not thinking through the consequences. If this becomes the norm, what will the “new” and index page look like when 90% of the content is comprised of reposted project homepages looking for attention?

> You’re not thinking through the consequences.

Although everyone has their blind spots, I certainly thought of your trivial example (pollution of /newest), and thats what I suggested is the "purpose of forum software". That is what "spam" means by the way. Writing a similar-link detection mechanism is trivial and I'm sure will be done, same way reddit has, if it's an issue. Happy to discuss any other consequences you think I'm not thinking through.

In fact the more interesting thing here is how inconsequential this thread is as it certainly wont have any bearing on the link submission culture @ HN. I'd suggest reaching out to the site admins instead of shaming the individual.

there's nothing new about this norm -- you'll find a ton of accounts like that. often they never make it to the front page, but occasionally a useful gem will surface.

/newest is and always has been a shit show of self promotion

Thank you!

I do a little hobby work in Go. I looked at micro several times. For some reason I felt it is bit too heavy or unnecessary complexity. It does not feel fit in go ecosystem. But may be a lot of people are liking it. I could be wrong because I have not done much real work in Go. Does anyone use it for production apps?

I think the Go community in its early years advocated for libraries over frameworks. 1. Because of the very powerful standard library which dictated the development model and culture of the community. 2. Because Go was still such a young project. I think its only natural over time that frameworks and tools emerge to simplify the development experience in every language. We realistically cannot ask people to piece together complex systems in that way.

If you imagine the thing you will end up writing in 200-300 lines of code in your main function, we're just encapsulating in a single function for you. The thing most companies end up building, a common shared library is by any other name a framework and so we're just piecing together that common experience for everyone else.

Now the key thing to note is you can use any of these packages independently. In fact you can think of Go Micro as a standard library for distributed systems development. Its just that at the top level we import the common features and create the construct of a "Service" and give you one way to initialise it with `micro.NewService`.

But the great thing is you have the freedom of choice to use things like go-kit, go-cloud, or any of the other thousands of tools in the ecosystem :)

Go is already an intentionally simple language with an excellent standard library, and dictates how to do things in the right way. Do we really need something on top of it? Not to mention, it's quite common that engineers like to reinvent the wheel, at least a little bit. It's just part of the fun, no one can blame them for that)

With all due respect I think thats a bit of a short sighted statement. To say a language offers enough is to negate all of the tools built on top which others have leveraged to build other pieces of software. If you imagine all the things you import are they purely the standard library? Very unlikely. In the same way, Go Micro is built on not just Go but many other libraries.

I think if you find value in using the pure standard library, good for you and I commend you on the effort of the not invented here syndrome. But for the rest of us it is much easier when someone else helps us out by laying the foundations for the next thing.

Jeez... with all due respect, not sure it's a great idea to cover yourself with "the rest of us"... but back to the idea of frameworks that make life easier. One of the problems with frameworks is that sooner or later, you start hacking them, breaking the ideas behind them, and make your life complicated. So it's better to think twice before start using any framework. BTW Why do you call it a framework?

The “powerful standard library” to my experience is a myth that holds no water. Even trivial projects use external packages to handle trivial things like an HTTP server or logging.

No project I saw, however small, uses just the standard library.

I've written several services in my line of work where I was able to stick completely within the standard libraries. Half of my personal go projects likewise use the stdlib only. One of the projects that doesn't conform could do with a little coaxing, I only use afero for test convenience of creating a virtual fs. I'm likely to remove it in a future update.

I've written dozens of Go programs which use only the standard library - including very large ones. However, there are two primary areas where it is deficient (necessitating pulling additional packages) to my mind:

- Logging - a lack of ability to do structured logging is the main downside here.

- The `syscall` replacements - almost inevitably for larger projects it's necessary to pull in the x/sys/unix or x/sys/windows since package `syscall` was frozen.

The lack of "wrapped" errors also used to be an issue, but this has mostly been resolved with the new "%w" formatting directive to `fmt.Errorf`.

I'm not saying that it is not possible I'm saying that it's not happening to the extend that would make that argument true. If 10% of the golang projects were based on stdlib, then I'd buy that argument, but I expect that number for be a lot smaller.

Well objective data on this is out there if anyone can be bothered trawling through it:

https://github.com/search?o=asc&p=4&q=language%3Ago&s=stars&...

I don't see any useful info there, do you mind translating that for me as it would be interesting to know what % of golang projects uses only the stdlib.

It's always a good idea to end up with fewer dependencies

Yes. Many open source application fairly complex & large are written in Golang: kubernetes, terraform, istio, etc.

All these run in countless production systems.

Sorry I intend to say, does anyone use micro in production?

Production users of Micro! https://dev.micro.mu/users

You probably want to present this list somewhere on the landing page.

> Service Discovery - Automatic service registration and name resolution. Service discovery is at the core of micro service development. When service A needs to speak to service B it needs the location of that service. The default discovery mechanism is multicast DNS (mdns), a zeroconf system.

> Load Balancing - Client side load balancing built on service discovery. Once we have the addresses of any number of instances of a service we now need a way to decide which node to route to. We use random hashed load balancing to provide even distribution across the services and retry a different node if there’s a problem.

Trying to understand this a bit here. Is micro a replacement for k8s? Because I don't see any deployment capabilities here. So I'm assuming it's complementary to that. If that's the case, then I'm not sure how service discovery and load balancing would co-exist with the CNI in k8s.

Not everything is k8s centric but also k8s also has very rudimentary models for discovery and load balancing, this is why things like service mesh have emerged to handle those challenges. As a framework Go Micro builds in client side discovery and load balancing which are completely separate to any third party system. You can imagine the next distributed database to be written with Go Micro which can operating independent of k8s.

In the case of k8s a lot of people swap out the load balancing for something like dns or integrate it with an envoy or linkerd considering our protocol usage is gRPC. But otherwise our service discovery acts as a central registry for all services. In a development centric environment this allows us to share feature rich metadata along with endpoints, description, and anything like team info, pager details, etc. All purely written in code rather than through yaml.

Agreed. I'm not saying that this needs to be integrated w/ k8s. I'm merely comparing it to k8s which promises the same abstraction albeit declaratively. The biggest plus point, IMO, for k8s is it makes managing deployments across physical hosts easy. So, I was asking if there was anything equivalent in micro that would help manage my deployments. And your comment on client side discovery and Load balancing partly addresses my original question. My main point here being, I don't see anything in the linked doc on why I'd choose micro over existing frameworks.

EDIT: https://github.com/micro/micro/blob/master/README.md

This page is more informative on why Micro over anything else. So I think this answers my second question.

We used go micro in a production system for a while at a previous company. The nicest thing I found is that it abstracts a few common pubsub systems. But besides that I found it overly abstract. It's not really that hard to write glue for messaging systems as needed and that's the only part of go micro we were using.

You're right. It's not very useful if you're just using it for pubsub. Go Micro was always intended as a full fledged framework for distributed systems development and you'll find if you build systems at scale you end up developing a similar framework in house. For those just using messaging or a database my advise would be to go directly to that system. Abstractions are only useful when they work together collectively for the foundation of something more. In our case when everyone is building Micro services we can actually run these absolutely anywhere regardless infrastructure and share them with anyone as a reusable building block.

Are these things necessarily opposed? It doesn't seem impossible to me that a "full-fledged framework" also addresses the most common uses cases in ways that don't leave people befuddled.

I think its really a matter of opinion and choice of development. Some people want to build it themselves, others would prefer to use a standard. Its the difference between using Spring or writing tools around Java. Using small libraries or Ruby on Rails.

Again, I don't think they are necessarily opposed. The examples you give are, sure, but that's a library vs framework distinction.

As a contrary example, take a lot of Unix command-line tools. Things like ls and ps have very sensible defaults that cover up very complex models. Or my experience with Python's Twisted is that it is very rich, but it's simple to do simple things. And cryptography libraries are a great example of where well-chosen defaults are absolutely vital. Same with Wireguard. Is what it's up to very complex? Definitely. Do I need to understand the details to get good results? No.

it looks interesting, but unfortunately the doc doesn't provide much beyond general diagrams. As an example, i wonder what are the properties of the pub/sub system in terms of delivery guarantees and tradeoffs. Or, if the goal is to have a third-party system implement the interface and just plug-it in, then at least give a list of available wrappers ( eg : "we currently have wrappers available for rabbitmq or kafka if you want to use them as your pub/sub system").

Hi, author here. Thanks for your comment. You're right the readme doesn't go into detail and that's on purpose. I only wanted to provide a high level summary there. I need to do better and write some docs. There is a development repo where we're trying to slowly flesh out design docs https://github.com/micro/development. Please let me know if any of that helps and what we can add.

On the specific thing you're talking about. Yes exactly. We want to define high level abstractions for distributed systems which are pluggable. The idea being that we can't replicate the specific complexity and absolutely phenomal work of the teams building rabbitmq, kafka, etc but we can provide abstractions on top to simplify the developer experience and make it pluggable so those who want to use rabbitmq can but those who want to use kafka can swap that out instead.

The built in broker interface at the moment focuses on lightweight pubsub, notifications and events in a way that's fire and forget moreso than delivery guarantees, queuing, offsets, etc. You can use something like kafka or rabbitmq to get these guarantees but we want to write an Event interface that will be more inline with what people expect from that requirement also.

Hope that helps!

Dunno why the adversity in the comments. Great job, OP. I think there is a real need for something like this as the GitHub stars show, I considered using Kubernetes itself as a distributed infrastructure but it may be a bit heavy for many use cases. I think this can be a serious alternative for distributed systems that do not have intention to be built over Kubernetes.

Why is "microservice" not mentioned anywhere on the page? Sure, it's not a great term, but it would make identification of its goal immediate instead of taking a few seconds.

Reminds me of https://github.com/atomix/atomix

You advertise "a powerful pluggable architecture." But Go binary plugins cannot be unloaded. At least not yet. How do you overcome this?

Go Micro is a framework/library as opposed to a binary or server. You import the framework in code and it gives you the ability to add additional plugins via new imports. These can then be swapped out via environment variables or flags. We make use of Go interfaces to define the abstractions so all you have to do is create a plugin that implements the interface and that effectively makes it pluggable.

Here's the community led Go Micro plugin effort https://github.com/micro/go-plugins

What exactly does it mean to say that plugins "can then be swapped out"? I assume a binary Go plugin can not in fact be unloaded from memory. So modifying and reloading a Go plugin will result in multiple versions of it to then stay in memory. This is a shortcoming of the current Go implementation. Is my understanding correct?

Do not assume "Go Plugin" in this context. We are not using the Go plugin system. Plugin is a name used to describe something that can be "plugged in" or that is "plug and play". In our case as mentioned, we use the Go interface as an abstraction and the "Plugin" is an implementation of that interface which can replace what you're using.

You will import plugins for rabbitmq, kafka, etc as a Go import. You can then use env vars and flags to choose which you want to use. Internally we are taking an initialiser for New[Plugin] from a list and executing it to set it up. There is no Go language plugin loading, its all still compiled code. Its just how we use the abstraction.

Please see the readme for our plugins for more detail https://github.com/micro/go-plugins

I think you are confusing whatever go micro is offering with https://golang.org/pkg/plugin/

I'm pretty sure go micro's plugins are just additional packages each for its purpose ready to be imported and used in your project.

Basically a collection of community maintained packages for the most common utils\middleware\MQ clients etc

The runtime you build will have all plugins micro supports by default at time of compile, and any custom plugins you create that implement the interface. They are hot-loaded at runtime, but you are correct in that you cannot unload them. Since the switching is done by environment variables, you'd have to restart to reload with other plugins.

Plugins are really specific by platform combinations and support, so you're unlikely to change them much after initial launch. It's only useful if you're going from metal to cloud provider or changing the method of interaction between you API layers.

It's pluggable in that large parts of it can be replaced or disabled at compile time. Runtime reconfiguration isn't well supported.

Well if you pack in a new communication bus plugin, you'd still have all of the defaults (rabbitmq, and so on) available in addition to the one you add. You can use Environment variables to pick the one you want by its name.

I am wondering if a similar framework exists for Node.js or Deno?

I am really interested in Go but I have a lot of learning to do before I am confident to write solutions with it.

I believe Molecular is a node.js project with similar ideals https://moleculer.services/ and also Seneca https://senecajs.org/

Thank you :)

Reply on news.ycombinator.com