Show HN: Encore – Go framework for building distributed systems
github.com
github.com
I also feel like this is doing too much to be a framework, in a way. Like, if it just did the code generation, cool. But add in the secret management, cloud deployment and migration stuff, and it starts to feel like a platform - which, coincidentally, it is. With pricing and everything. I can't use this to build a service that runs on AWS/GCP/my own servers - at least it doesn't look like it from first glace.
So if this were actually just a framework that could be separated from the platform stuff, I'd be willing to try it out. But as is, I'd rather write boilerplate and be able to 'go run' my service and deploy it wherever I want.
EDIT: Just found that I could deploy to another cloud provider, which is nice. I wish I didn't have to manage that via a web interface, and that it supported just building a container I could deploy anywhere.
How do you balance that?
Compared to Ruby or Python, there is minimal code generation outside of the initial framework setups.
As for platform vs framework, my realization is that the traditional separation of frameworks/libraries from infrastructure means that it's almost impossible to improve many parts of the developer experience.
The annoying parts of building backends and distributed systems come from the integration between the two. Setting up databases, secrets, and so on all require bridging that gap. Another big area that we're investing in is providing insights about how your application works at runtime, in production, directly at your fingertips when you're writing code. That's also not something you can do without integrating across that gap.
As you pointed out in the edit, you can absolutely use Encore with your own cloud. We're working on making the deployment options much more flexible, but it's very much not another "let's make an infrastructure tool and then charge a premium on top of your cloud bill to manage it for you".
Our business model is much more about aligning the incentives with the developers. We want to provide the best developer experience out there, and by charging for that as opposed to for infrastructure, we'll happily invest time to reduce our users' and customers' cloud bills, rather than just selling them more servers they don't really need.
I was at Spotify for the past 8 years as a Staff Software Engineer. We grew frustrated with the disproportionally large effort needed to build even simple backend applications, and created Encore to solve this problem.
Encore uses static analysis and code generation to reduce the boilerplate you have to write, resulting in an extremely productive developer experience. It's a combination of a framework that combines with a cloud platform in order to simplify all aspects of building distributed systems, from how you write code to shipping it to production.
I shared some more background on Encore here [1]. Feedback very much appreciated :)
Off the top of my head: a lot of stuff from Google (Dapper, Stubby, Zanzibar). Dynamo from Amazon. Leslie Lamport's guide on TLA+. The two generals problem, distributed consensus, and so on.
Most distributed systems don't need all that fancy stuff, but it's useful to know regardless. Even the simplest systems will get weird and interesting bottlenecks with scale.
And then a lot of just doing it in practice! Find a job at a startup with some scale, and learn on the job.
We have everything like CI/CD, distributed tracing and one line project generation.
It's just based on some foundation code and some simple bash scripts to find and replace some service specific things.
And then it's just the basic routing and HTTP handlers to unmarshal and marshal responses.
When all your services are built off the same foundation then creating a new one is pretty much and copy and paste job in my experience.
It's a cool project but it seems like unnecessary complexity, something I strive to avoid.
- Automatically sets up dedicated Preview Environments for your Pull Requests
- Provisions all the infrastructure you need, both for local development as well as in production (and other environments like staging or testing)
- Provides auto-completion and compile-time type safety for making cross-service API calls, with a single line of code
- Integrates secrets management, API documentation, frontend client generation, etc :)
We just use the Elastic client libraries. Tell it where our APM server is, do the little bit of application set up and then view all our application traces in Kibana.
All applications follow the same basic setup. Probably less than 20 lines of code and then wrapping some HTTP handlers and HTTP clients. Then we have some application specific spans.
what i mean is that you have complexity in your existing setup too... its just that you understand the ins and outs. and so the same applies to encore.
Nothing is hidden away from you with code generation. There's no in-between steps from what you write to what you execute other than Go compilation.
I think it's the wrong mentality to require a whole new set of third-party tools to maintain a Go application. It then starts feeling like you're writing Javascript, and does anyone really want that?
If you did down enough, you'll find some form of code generation in your environment of choice. If nothing else, compilers, but it's not limited to them.
The largest difference between that and copypasta is - generated code can be easily updated, by regenerating it.
It's also an implementation detail. I've read elsewhere in this thread that code generation is bad and language supported features are good. I'm not sure it's so clear cut. It shouldn't matter if the generated code is mostly invisible (when it's baked inside your binary by the compiler, due to language support), versus code that is generated in the source language and then compiled.
One thing that's always bad is mixing generated and non-generated code. You have all the disadvantages and little benefit.
I'm getting into go, but there doesn't seem to be a great pattern for this. Used to @Transactional on the service, so feels annoying in having to build that.
If you are saying about micro services here which talk grpc/http then @Transactional does not help.
If you are talking about inside one service then you have write all that boilerplates, and it will not look as simple as Java
The gist is you timeout but provide a way for transactions to resume.
Also obviously there’s 2PC but again it depends on what unite trying to solve.
This. Adding complexity to reduce boilerplate seems like the wrong direction to go in (pun intended).
I like boilerplate. I can overwrite it with special cases whenever I need to, and it takes up very little mental space (as opposed to the rules needed to generate it).
I hated working in Rails because of all the magic. I love working in Go because of the lack of magic.
But y'know. I'm sure it'll suit someone. Good luck with it OP.
I wonder what you think about K8s integration? Spotify is all K8s right? I'm guessing that is tougher to monetize. I tried at a prior startup to work on something like this, offering a PaaS like experience on K8s with development libraries (it's actually still in progress - check out https://cloudstate.io/). It is a large effort to build such a thing in a nice way.
Anyways, I'm looking forward to seeing how Encore develops. Good luck!
I'm going to sound naive but where did running Docker daemon on a VPS and orchestrating deployment with `docker-compose.yml` or `Terraform` fall short?
To me something is only a distributed system if it infact has stateful (even ephemerally) components that interact with each other through some sort of distributed consensus or eventual consistency algorithm.
This on the other hand seems more like an application framework for traditional top-down scale out web applications where requests flow in from the top, through stateless application tier and into a stateful, centralised database.
Arguably it's "distributed" because it's made of multiple processes on multiple machines but it does really feel like a watering down of the term.
Things I would consider distributed systems: Zookeeper, etcd, Kafka, Druid, Hadoop, Bookkeeper, Pulsar, RabbitMQ, crypto stuff, bittorrent, Hazelcast, etc. Things that distribute work and storage across components and are aware of their peers.
Not to diminish the work here, just that I think the wording is a bit rich.
I like Leslie lamport’s definition of a distributed system. I’m paraphrasing but it’s something to the effect, “a distributed system is when your program crashes because of a computer you never even heard of”.
So yeah, I mean a multi-tier web app can definitely be a distributed system.
Typically people don’t think of it like that because those systems are very centralized around a database for example, but in the most technical sense a multi tier application is still a distributed system.
So is an app consisting of composition of docker containers. Message passing in general can kind of blur the lines but yeah this too.
Your definition is too strict. Distributed systems are systems where the components of a system are distributed across a network. There are more issues than just consistency of data. You could have a system that doesn’t store data at all but still is considered distributed due to the nature of it being spread across a network.
I'm unsure where this protection of a term for more "complex" systems come from. It's a rather simple term that doesn't require to be needlessly reserved.
Well you're going to learn something new! Here's the short definition from Wikipedia:
Distributed computing is a field of computer science that studies distributed systems. A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system.
https://en.wikipedia.org/wiki/Distributed_computing
Any system that is distributed between multiple machines over a network and communicate with each other is a distributed system. The term is purposefully generic and not required to be constrained.
It's not another infrastructure offering but rather a new developer experience for building backend applications and distributed systems. The idea is to get the developer feedback loop to be about building your product and writing business logic.
Encore then uses static analysis to understand what your application does, the different backend services involved, and the infrastructure you need. It can then provision all the infrastructure, servers, databases, and hook everything together.
I'd love to see an example using websockets. I've used Go for websockets in the past but find the boilerplate-to-custom code ratio isn't great.
Knowing Go has a solid webrtc library (https://github.com/pion/webrtc), another cool feature would be webrtc support. Tons you can build with that!
[1] https://github.com/encoredev/examples/tree/main/websocket-ec...
You mentioned on another comment that you would like to introduce a pub/sub element to this. Could you talk a bit more about that? I can see how that work for communicating between services managed by Encore but could it be integrated with existing event sources?
In general, I guess it's hard to see how some of this would integrate with existing systems nicely. If you're starting from a blank slate I'm sure the devex is excellent, but what would the workflow be like if only a small percentage of my infrastructure was done this way? (Which is how I imagine all of your customers would start out).
- Is there a story behind the name?
- Out of curiosity, does “Encore Cloud” also run on AWS?
- I think a lot of developers have a mentality of “rather write boilerplate than be tied to a platform not endorsed by $bigCompany”. Do you have a strategy to mitigate this?
As for being tied to us, that's a big part of why we're open sourcing it. We see that Encore can rather reduce lock-in, as the code you end up writing with Encore is very cloud-agnostic, but also quite agnostic of Encore itself (by virtue of using static analysis, you end up writing mostly regular business logic).
And when you use it you can deploy the applications to your own account at any major cloud provider, so you can use your existing trust relationships with your cloud provider of choice.
As for the name, it's a bit of a wordplay :). "en-" for "to bring about [something]", and "core" for "the part of something that is central to its existence or character" (in our case backend development).
We're also working on expanding the set of infrastructure primitives supported, so that you get the Encore experience for doing more types of things. Things like Pub/Sub and message queues are the next things we'd like to add support for.
I know dapr injects a sidecar so you can code in any language and then use their SDK to call other microservices. Encore looks to be Go centric and uses code generation to create actual Go functions, which provides much better IDE support.
I did not see this in the docs but I’m not sure if Encore supports retries with or without exponential backoffs; dapr supports both.
I like how go-kit's transport not just JSON and HTTP (e.g. gRPC and protobufs are supported).
They are fighting an uphill battle to beat the OTP + lib-cluster + horde combo that elixir developers have but that autogenerated api stuff looks neat.
Love this quote by the way.
"With Encore you write your apps using Go, a modern programming language developed by Google to make backend development simpler. Rapidly growing and designed to be easy to learn, Go is the language of the cloud."
I think the irony of Go is that it is easier to "script" things in Go than in NodeJS for instance. This makes it the right tool for infrastructure programming. I always use Go for these kind of things.
Writing enterprise apps in Go is a bad idea with or without Encore. One lib can't simply make up for the large ecosystem in the more modern technologies.