Portable Cloud Programming with Go Cloud
blog.golang.org
blog.golang.org
The majority of the team is in SF for Google Cloud Next to present this. I'm about to hop into the car to head up myself to watch, so I can do a little AMA throughout the day. Reply to this to make sure I see your question!
Here's a question I've already seen:
> Is this a Google project or a Go project?
It is a project led by the Go team, and we are very aware of our responsibility to be cloud provider neutral. At the same time, while the project is so new and evolving so rapidly, we need to have working examples that we can change atomically in a single repository (atomic changes necessary for rollbacks and such). We don't want to have cloud provider-specific code in the Go project, so the code is currently sitting under the Google GitHub org. When the interfaces to different products stabilize, the intention is to graduate them up to the Go GitHub org. Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured. The cloud provider-specific code will always live outside the Go project.
But things are very new, so change is possible.
It also seems to be one of the biggest points of controversy for Go 2, with calls from "everything should have a context" to "context should go away".
The libraries that support it have now all their function signature way longer, and it is not well understood if it is a "Request for cancellation" or something else
Given that all of a Go program sooner or later will need contexts somewhere, it would be much cleaner to let goroutines have this stuff implicitly. I don't know enough about the intervals to say whether this is feasible with the current runtime, though.
Yet, Azure support is listed as unplanned.
The language could mean that this is not going to be planned. The fact that y'all have not worked with Azure Go folks to get this moving is something to ponder.
A reminder that Go is a Google language and isn't vendor neutral?
"Unplanned" on the issue tracker just means we haven't assigned it to a sprint milestone for the team yet.
Azure and more clouds are very much in the works. As an open project, there are many approaches and we're open to all of them.
One other point: since Go interfaces are implicitly fulfilled, anyone can write and share implementations of Go Cloud for new providers or custom services without asking anyone for permission. If you take a look at the implementations we've provided, you'll find it's actually rather easy.
More to come.
My my question is, what if developers want access to innovations in specific cloud providers, such as reduced redundancy storage on S3? Wire up or own provider? Will there be some sort of community-provided provider registry or will we be grepping awesome lists on GH? Are providers extendable in any way?
Thanks and have a sunny day.
The main goal of Go Cloud is that the abstraction is not leaky and that you don't get access to specific features in the way you are describing in the application itself.
Now if the specificity of the thing you want happens at provisioning time but the API is the same (e.g. choosing to use Aurora on AWS vs standard RDS), then there's no problem. Go Cloud doesn't need to know about that.
If the API changes too, then that's a much different thing, and gets back to the leaky abstraction which we want to avoid. If you can't simply wire new providers to your application code and have it Just Work, then we've done our job wrong.
But remember that it's all Just Normal Go, there's no magic here. It's just standard Go. If you want to write an app that ties itself to something specific on some cloud you can; just write the code you'd write now. But that necessarily means you're tied to that product.
In the minute you implement something provider specific, you just lost the whole point of this library and you did not needed it in the first place.
> Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured
Of course they can, but you're working on a Google-centric version first right? Otherwise why would it be at Google Cloud Next?
That makes all other cloud providers second tier, and I don't think you should call yourselves "platform neutral". If you were, you would code up multiple platforms from the beginning, like is normally done for support on different operating systems for example
Please see the comment from our PM regarding Azure: https://news.ycombinator.com/item?id=17604358
Seems pretty weird to include support for an Oracle offering from the get-go, especially given the bad blood between Google and Oracle.
Your people can pick up go reflect package or go routines/channels and co in 1-2 weeks and start contributing? impressive... most people can pick up for loops, if,switch statements, variable and function declaration in a week in any language as well...
In fact, a lot of non programmers use JS regularly to write scripts in the browser. I imagine your people know how to program before hand... So what do you think takes longer to learn? mastering go threading model or being able to add interactivity to the DOM?
It is definitely easier to write well than js IMO, and harder to make a big ball of mud in.
For me, the go threading model is easier. I just can't stand DOM and web programming in general. Currently working on a problem doing some computation across dozens of cores on multiple servers and go makes it a lot easier to spawn the 1000 goroutines and coordinate between them. I guess everyone has different interests and perhaps it is similar for those we hire.
Because you didn't take the time to learn the DOM, that's all. I doubt learning a simple composite API made of dumb nodes is more difficult than writing thread safe programs in Go. I'm not talking about tastes or interests, I'm talking about ease of learning.
Go isn't 'easy to pick up'. For loops, if statements and co are basic, and they are in all languages, that's my point. There is nothing particularly easy with Go.
A lot of what makes anything hard to learn are the non-obvious gotchas. The design philosophy of golang is aimed at avoiding the non-obvious gotchas and the creep-up-on-you-at-scale gotchas.
All languages are going to suck, somewhere. The trick is to actually optimize a language for the difficult, expensive parts of development. Many languages seemingly optimize for writing things from scratch slickly. They're "blog example optimized." Instead, languages should be optimized for debugging, reading code, and deployment.
The fact that Go is not afraid of boilerplate is often actually a good thing. I've had to debug C++ in the confluence of 2 templates, such that there was no source code to display in the debugger. I don't expect to have this problem in Go, ever.
"You can't pay people enough to carefully debug boring boilerplate code. I've tried." (Yaron Minsky of Jane Street)
While C++ goes to one extreme, not having certain expressive means (whatever they may be) has a price, and for certain kinds of problems, this price may be high.
This returns to the idea I very much agree with: "The trick is to actually optimize a language for the difficult, expensive parts of development." The catch is that difficult parts are also problem area-dependent. Go definitely found a sweet spot for certain kinds of problems, though.
Question is, how it will move beyond that, if ever.
I still think that JVM + JIT is hugely important, though. Now it's importance is somehow diminished by presence of LLVM, but 20 and even 10 years ago the situation with native compilation was different. (I know that the canonical Go compiler does not use LLVM.)
When Java came around, latest Oberon variant was Active Oberon (Active Objects are similar to goroutines), then we had Eiffel and Modula-3 as well.
Hence why I say it should have been Java 1.0.
There's boilerplate, and then there's boilerplate. If boilerplate is just a formality, with clean, competently designed underlying concepts, then it should be fine. Someone smart will build a code generator for that part of the project, you have your type safe-whatever and the project goes on its way. On the other hand, if your code base is just numb-nuts and people are just cut & pasting lava-flow code all over the place and rampantly putting business logic into ORM routines, then yes, that's going to impact hiring.
not having certain expressive means (whatever they may be) has a price, and for certain kinds of problems, this price may be high
This is also true. Even fairly innocuous boilerplate can build up to the point where refactoring can become tedious. There is no utopia, only constantly changing cost/benefit tradeoffs.
w, err := b.NewWriter(ctx, "gopher.png", nil)
...
_, err = w.Write(data)
...
if err := w.Close(); err != nil
Not to mention, what is the last argument `nil` supposed to represent? In most high quality Node.js libraries, functions/constructors will take an optional options object so you can omit it, or pass an object like { timeout: 300 } to it.
The "if" block creates a new scope.
It's equivalent to
{ // begin new scope here
err := w.Close()
if err != nil {
foo
} else {
bar
}
}I also hate the inconsistent behavior when you check multiple errors in a row, the first one will get := but not the second one (if you don't put those in a if statement, which I never do). But now if for some reason, you remove the first := statement, the code will error on the second one, that needs to be changed from = to :=
A lot of it has to do with expectations. Going beyond the principle of least surprise, there's also a principle of "no dismay." Go is like the dependable old Corolla. It sets modest expectations, and it delivers a high degree of utility very consistently, and if you're ever let down, it's very rare. C++ can cause you utter dismay due to rather subtle slip-ups. I've only seen this level of dismay in Golang around subtle resource release issues around prepared statements.
This is true, but Go does have a clean and mostly familiar syntax, by design: you can mostly follow along with a piece of Go code even if you haven't learned the language.
I think that what the OP is pointing at is a related issue: readability. This is explicitly a high-priority design goal for Go: lots of things in the language are designed so that developers can easily read and understand the code in the projects that they work on, and that memory use is never hidden.
First, repetitive error handling makes it overly tedious to actually get a quick intuition for what a function is doing. With Rust as a comparison, it’s extremely easy to see what the happy path is trying to accomplish without having the downsides of arbitrary and unexpected exceptions. Go is inarguably worse here.
Second, the inability to express higher-level concepts (due to, yes, lack of genetics) seems like it forces every function to deal with the nitty-gritty details of whatever it’s trying to accomplish, rather than being able to express ideas at a higher level and deal with the specifics in smaller code units. Yes, you don’t get “surprising” behavior, but the cost of this is that you’re forced to keep both the high level goals and the low level details in your head simultaneously, which I find to be a massive hindrance.
As sort of a side effect of both of these, there ends up being a ton of repetitive boilerplate. This actually hides bugs since it’s easy to miss minor differences between overly similar blocks of code dealing with errors or looping. As an example, languages like Rust and Ruby that have proper functional-style iterator methods prevent so many bugs through their inclusion it’s absolutely baffling to me how someone would opt to design a language today without such things.
---
This is a valid Go program.
import "fmt";
func main() {
fmt.Println("Hello world")
}
import "fmt";
func main() {
// fmt.Println("Hello world")
}
---
Creating typed general-use data structures is impossible.
Surprising.
Go is a thoroughly "unsurprising" language.
func setupBucket(ctx context.Context) (*blob.Bucket, error) {
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-2"),
})
I can't see any surprises here.
Contrast this with a language like Ruby which tries very hard to make punctuation optional, and often ends up with ambiguous cases where you aren't 100% certain how the interpreter will parse some code and so you end up throwing in all the punctuation anyway.
I'll take Scheme for a spin if I am doing something for fun.
EDIT: %s/bored/boring/g
I do love how opinionated Go is, though. Every language should have a gofmt equivalent.
- not knowing if "Foo" being returned is a struct or an interface, so you're not sure if you should "if foo != nil" (it could not be nil if it's a struct)
- channels freezing (specially unbuffered ones). I still can't figure out how to debug those well.
- (the most common of all) spawning goroutines on a for loop, and all of them closuring over the same value (instead of one per item on the loop)
it's a surprising amount of sharp edges, for such a tiny language...
2. Always limit the scope of your channels. I like using unidirectional channels with structs that have more unidirectional channels. (For instance, a chan of workRequest{workData, returnChan} where return chan only receives responses for this request and is closed when no values are left)
3. I think you've been treated to well by JS closures, always hand over arguments you're going to use in the goroutine, this also allows the CG to clean up the stack of the function that started the closure goroutine.
2. Yes, I know how to “properly” do it. Does not eliminate human error.
3. Closures in any other language, afaik. I rarely code in Js, fyi
Most of our Kubernetes code looks something like this
_, err := clientInterface.Update(object)
if err != nil {
if machinery_errors.IsNotFound(err) {
//handle nil case
} else
// handle unexpected case
}
}This portable cloud strategy is a mirage. I haven't seen anyone migrate providers easily (we tried in my last company). In reality, you have to break the cloud agnostic abstraction to leverage provider specific functionality. And then you are stuck with unmaintainable mess of 'cloud agnostic' APIs mixed cloud specific APIs.
Interesting that the #2 cloud provider is listed as unplanned in a library coming out of the Go team.
This may further Google's corporate goals, but that doesn't mean it wont actually increase portability, or that it's actually a bad thing for people who want to avoid lock-in.
> In reality, you have to break the cloud agnostic abstraction to leverage provider specific functionality.
I agree this is a difficulty, but I think that it's also avoidable if you make a disciplined commitment to interoperability. Just as you have to put in the effort to use this new "portable" API, you also have to choose not to defeat the former investment. Saying that this API is useless because you end up using provider specific functionality anyway is like complaining loudly that something is broken when you're using it wrong in the first place.
This is like those DB abstraction layers because "we need to change DB", but in the end almost no one doesn't the products lifetime.
Also given that most of our clients don't use these fashionable cloud APIs, rather their own infrastructure, makes also not that much relevant to me.
Are those hybrid/mixed cloud solutions or move between cloud provider viable or that common?
Is there anybody went through a huge cloud migration which went well?
If you need to move, there are huge amount of data you have to move for example between S3 and Google Cloud Storage, is that even possible to do beyond an early stage? I can't imagine.
Go is still new and Google is "old", so there is surprisingly little Go code at Google - it doesn't make sense to rewrite critical infrastructure just because you have a shiny new language.
The derivative is positive though, and slowly Go is being used for more and more new development.
Client-side development is a whole different thing. It's fragmented between browser, Android, and iOS. (And each of those is further fragmented, with multiple languages available.)
Once I wrapped my brain around fx, I really liked it and used it heavily in a project I wrote. It was especially good for initializing 'things' of the same 'type'. Like a bunch of controllers for a web app.
One nice thing about fx is that it does not rely on code generation (uses reflection instead). I'm very curious what other features wire has over fx.
That seems to be a common thing in Kubernetes too, if you don't want to spend the time writing actual code and Go is too "boilerplate heavy" for you, just write a code generator and bring the complexity of other languages right back!
Secondly on the article's mention on multi cloud usage, you would need to be a pretty large place to need (or even bothering to assess and convince yourself that you need) to use multiple cloud providers at once. Just learning and tweaking settings in the cloud providers GUI console is half the battle won sometimes.
Saying all this, this, together with the data portability announcement, it is definitely great for competition and going to bring great resilience in your code base to be SDK agnostic.
For your second point, it's normal for small startups to shop around and pick the cloud provider that's more suitable for their needs. For example on my last job at a small startup, we switched from AWS to GCP for cost reasons, and I implemented an abstraction (similar to go-cloud's) between S3 and GCS because of that.
Oh, this is not the case at all. Do business with any retailer? Try telling them your stuff is hosted on AWS and see how that works for you.
I do think it's an excellent configuration management and provisioning tool, but Go Cloud solves a different problem.
We actually use Terraform for provisioning in our samples, and we see the two working hand in hand.
I am pretty excited about it.
err, speak to me...