Go kit: Go in the modern enterprise | Hacker News Reader

Go kit: Go in the modern enterprise

peter.bourgon.org

140 pointssagichmal125 comments

I'd be pretty nervous about any attempt to recreate Finagle in Go. Scala provides many ways of creating composable complexity. Finagle leverages pretty much every feature of Scala (and there are many) to maximize this composability in a sane way, but because there is simply so much complexity in Finagle, a lot of opportunities for composability have been skipped due to lack of need within Twitter (maybe finagle-http has significantly improved in the last year, which is mostly what I'm thinking about when I say this) but it's a LOT of complexity, sort of like the RoR of distributed systems, and when you want to diverge from Twitter's chosen trade-offs, it may be a hellish experience. It's pretty sweet for when your problem space aligns with Twitter's, however.

Go has basically none of the mechanisms for sanely creating composable complexity that Scala has. It makes me really grossed out to think about something with the complexity that Finagle exposes as a Go library. Go is all about small, sturdy tools that work for most of the cases most of the time and that compose well with each other - IF - they are built into the standard library. And if it's not, have fun spending your time writing uni-typed interfaces for any kind of interop with desired complexity. Go is great for writing dead simple byte slingers, pipelines that will be rigid in the most dynamic possibility, guns that shoot JS and interact with services, etc. That's simply not Finagle.

Could you provide examples (beyond generics) where 'composable complexity' helped finagle achieve a goal?

My experience with Scala has been that an author's 'composable complexity' becomes a reader's 'wtf is this complexity'.

Go's philosophy has been to avoid complexity altogether in most of the tools they give you, forcing you to write lowest-common-denominator code that anyone can easily understand. Generics are sorely missed, but I really don't see additional expressive power from most scala features, just additional things I need to keep in my head for an occasional 5-line savings in the middle of some method (which IMO doesn't change the complexity equation at all). Would love to see some beautiful examples that change my mind.

Clarity of thought is rare. The ability to express those thoughts as code is even more rare.

Go's approach is to makes it easy to say certain things. Goroutines are great. But it also makes it difficult to say complicated things like List<T>. Good programmers suffer, but really it's just a few more keypresses, ala Java.

Scala makes it possible to say even more abstract things like T<int>. Not many programmers can really explain a higher kinded type, or why such a thing is useful. Novice programmers say silly things because they don't really know how to apply all of this abstract power.

Go is pretty wonderful because it lets programmers of varying ability work together effectively. Scala is teaching people new abstractions. Are they worth it? Maybe not in Scala. However, I think languages with powerful type systems will ultimately win, because you can lean hard on the compiler to guarantee correctness.

You forgot the other drawbacks that complicated type systems have: hellish compiler messages and long compile times. Those don't go away just because you learn more about the language.

I also dispute the idea that only novices abuse complicated type systems. Actually I think it's the reverse... only people who know the language well can create something truly horrible. Don't believe me? Look at any of the C++ STL header files and experience Biij.

Of course, just because they can doesn't mean they will. But in a world dominated by people of average intelligence, working around the programming equivalent of a gas lamp with an open flame just isn't a good idea.

> drawbacks that complicated type systems have: hellish compiler messages and long compile times

Scala has incredibly nice compiler messages and you can see Rust for an example of a language with complex types and short compile times.

These issues have _nothing_ to do with complicated type systems - you're projecting the failures of the C++ template system - failures which stem in fair part from the fact that it's not really a type system but simply a templater.

Finally, it's worth recognising that Scala goes to lengths to keep messy features inside their own compilation unit - for example, macros appear as functions to the outside and complex features such as implicit conversions must be explicitly imported for use.

I tried building our project in Go and Node.js explicitly to make it as easy as possible for people to contribute, irrespective of their background. It turned out that the additional complexity I had to pump into abstractions to make it happen made the code just as complex to understand.

It's quite possible that projects just have a natural complexity level and it's worth picking the language to match. There are certainly different times for different languages.

As much as I like Rust, I have to say that it's not advancing a compelling argument for "fast compilers with complex types" (which isn't to say I don't think it can be done, just that Rust is by no means fast at compiling, and really only gets away with the slowness it has because its main competition still lacks a proper module system).

Rust's type system has nothing to do with any slowness in its compiler. There's a `time-passes` flag that you can pass to the compiler to see where it spends its time compiling any given program. Do so and you'll see that the overwhelming majority of Rust's compilation time is spent inside LLVM generating code.

Go libraries are typically highly composable because they operate mostly on primitives or strictly defined common interfaces. Orthogonality is a key feature.

I agree with you completely, yet what could prevent the gokit team from starting with the definition of very simple , orthogonal, interfaces and maybe add the very few new types required, to immitate the go stdlib.

I'm quite sure the result will be quite different from what those kind of frameworks look like in other languages, but that could be interesting still.

That's what I'm hoping for. I imagine many of the services could be designed in a manner similar to database/sql, with its driver interfaces.

If I recall correctly those libraries aren't type safe and composable, but perhaps I'm remembering wrong.

They are type safe, except for the unsafe package but it's hard not to know what you're getting into with that one.

One of the best examples of composability is the Reader and Writer interface. They're designed to operate on slices of bytes, so a package intended to read from a Reader can read from files, stdin, networks, sockets, and a myriad other systems with absolutely no changes to the package's code or its implementation.

How would you do decode this json in a type safe way using Go?

    {"purchaseType": "Rent","price": 0.99,"title": "inception"}

Here is an implementation in Haskell:

    {-# LANGUAGE DeriveGeneric #-}
    {-# LANGUAGE OverloadedStrings #-}
    
    import Data.Aeson
    import GHC.Generics
    import qualified Data.ByteString.Lazy as BL
    
    data PurchaseType = Free | Rent | Buy | Subscriber deriving (Show, Generic)
    instance FromJSON PurchaseType
    instance ToJSON PurchaseType
    
    data Media = Media { title :: String, price :: Float, purchaseType :: PurchaseType} deriving (Show, Generic)
    
    instance FromJSON Media
    instance ToJSON Media
    
    jsonData = "{\"title\": \"inception\", \"price\": 0.99, \"purchaseType\": \"Rent\"}"
   
    main = print $ (decode jsonData :: Maybe Media)
    -- running main gets:
    -- λ> main
    -- Just (Media {title = "inception", price = 0.99, purchaseType = Rent})

EDIT: Spec Change, boss wants an order confirmation message!

    orderConfirmationMessage (Media mediaTitle _ Rent) = putStrLn ("Thanks for renting " ++ mediaTitle)

Then we recompile and... what's this?

    Main.hs:19:1-100: Warning: …
        Pattern match(es) are non-exhaustive
        In an equation for ‘orderConfirmationMessage’:
            Patterns not matched:
                Media _ _ Free
                Media _ _ Buy
                Media _ _ Subscriber

Good thing GHC had told us Free/Buy/Subscriber's needed messages instead of our boss! In fact, had I been using -Wall this would be a compile time error.

Getting to the point, my real questions are:

1. What is Go's way of ensuring your function handles cases like this, or is there an idiomatic way of avoiding it?

2. How do you discern between MovieType's without resorting to just using Strings and compromising type safety?

How about an array of Media?

    package main

    import (
        "encoding/json"
        "fmt"
    )

    type Media struct {
        Title        string
        Price        float64
        PurchaseType string
    }

    func main() {
        jsonData := []byte(`
            [
                {"purchaseType":"Rent","price":0.99,"title":"Inception"},
                {"purchaseType":"Free","price":0.00,"title":"Johnny Mnemonic"},
                {"purchaseType":"Buy","price":17.99,"title":"John Wick"}
            ]
        `)
        var media []Media
        err := json.Unmarshal(jsonData, &media)
        if err != nil {
            fmt.Println("error: ", err)
        }
        fmt.Printf("%+v\n", media)
    }

Output:

    $ go run main.go
    [{Title:Inception Price:0.99 PurchaseType:Rent}
    {Title:Johnny Mnemonic Price:0 PurchaseType:Free}
    {Title:John Wick Price:17.99 PurchaseType:Buy}]

Edit: Regarding the MediaType: http://stackoverflow.com/questions/14426366/what-is-an-idiom...

PurchaseType doesn't need to be stringly typed. See https://talks.golang.org/2015/json.slide#1.

    type PurchaseType byte
    
    const (
        Free PurchaseType = iota
        Rent
        Buy
    )
    
    func (t *PurchaseType) UnmarshalJSON(data []byte) error {
        var s string
        if err := json.Unmarshal(data, &s); err != nil {
            return fmt.Errorf("purchase type should be a string, got %s", data)
        }
        have, ok := map[string]PurchaseType{"Free": Free, "Rent": Rent, "Buy": Buy}[s]
        if !ok {
            return fmt.Errorf("invalid purchase type %q", s)
        }
        *t = have
        return nil
    }

That's enough to give this output:

    [{Inception 0.99 Rent} {Johnny Mnemonic 0 Free} {John Wick 17.99 Buy}]

http://play.golang.org/p/dyNVlgmFtu

Check out my update if you haven't seen it already.

tl;dr The problem for me personally is when you start making functions based on the type of media and sacrifice type safety if MediaType is a string.

There is no exhaustive pattern matching in golang. And you would have to define your own unmarshall function that translates "Rent" into, perhaps, a const. So, yes, type safety in golang is not as powerful as it is in Haskell. I think we already knew that though didn't we :)

> So, yes, type safety in golang is not as powerful as it is in Haskell. I think we already knew that though didn't we :)

That is just a really big one for me I wanted to share and get others opinion on. I guess I just feel like a lot of people don't realize how much having that stringly typed hole means. Perhaps others just disagree.

I would love to hear a counter argument for why others don't think this matters.

I've been coding professionally for 16 years. I've written a lot of this kind of code:

    switch (someEnum) {
      case foo:
        <something>
      case bar:
        <something>
      default:
        throw DeveloperScrewedUpException("Unknown enum val: " + someEnum)
    }

...You know how often I've seen those exceptions? Never. Not once. I'm not saying it can't happen, and I'm not saying that I wouldn't want a compiler error to tell me I missed a value... but I am saying that it just doesn't actually happen all that often in my experience, especially with decent tests. And yes, I've worked on systems for many years where they've evolved and things have changed significantly, so it's not just that I've only worked on brand new projects.

> If I recall correctly those libraries aren't type safe and composable

You may be confusing interfaces (which enforce type compatibility) and the empty interface{}, which is effectively a dynamic type in the Go world.

Take a look at the Stack class (an example of a free monad) and how it's used to compose clients:

https://github.com/twitter/finagle/blob/master/finagle-core/...

http://twitter.github.io/finagle/guide/Clients.html

A client is a layer of various components like load balancers and stats recorders, but they're also all configurable. Only using the expressiveness of scala could such a data structure be created to compose all of these pieces, yet still keep them modifiable and type-safe.

I didn't mention in it my sibling post here, but we started writing the core of wym (our microservices framework) in Go, then Node.js and finally settled on Scala for exactly the reasons you outline.

Writing in those other languages was indeed a hellish experience - it was very clear that they lacked the expressivity to model half of even just this very core piece.

> Writing in those other languages was indeed a hellish experience - it was very clear that they lacked the expressivity to model half of even just this very core piece.

This is something that keeps sending me back to Haskell, I'm glad to hear Scala applies as well.

I've been meaning to learn more Scala, is there any "Scala for Haskellers" literature you are aware of? I'm going to start searching for some, but you could know of one that doesn't come up in my search results.

There is an awesome book called "Functional Programming in Scala" that has a lot of great questions to work along to, and is great for somebody interested in both Haskell and Scala but doesn't assume prior experience with things like combinators and monads. If you are pretty far down the Haskell route and want to learn about one of the more advanced type-level libraries, You may want to jump into http://eed3si9n.com/learning-scalaz/ and use https://twitter.github.io/scala_school/ as a quick reference or intro. If you have the time and desire, a lot of people swear by a cover-to-cover reading of Odersky's "Programming in Scala".

Hey, sorry - when I came to Scala I'd seen enough other languages that I read the language spec[1] and learned most of it in a day.

Since I never spent time onboarding myself, I'm not sure I can be any help. :/

The docs website is amazingly good, though - the tutorial[2] covers most of the unusual-in-Scala bases very quickly.

[1]: http://www.scala-lang.org/docu/files/ScalaReference.pdf [2]: http://docs.scala-lang.org/tutorials/tour/tour-of-scala.html

Generally speaking, Go is not a good fit for frameworks (which are meta-tools by definition). Implement the actual tool itself in Go, not a framework to build the tool.

I would disagree with this - it's a very broad statement. While building frameworks often encourages developers to abstract to the nth degree and add far more flexibility than is required, if you have specific requirements what's the problem?

We built a microservice architecture in Go at Hailo which runs approx 170 services. Abstracting core functionality into a 'framework' allowed developers to build and deploy services extremely quickly, and let the platform team get on with adding and improving core platform functionality, such as monitoring, instrumentation, tracing, provisioning, configuration etc etc.

Let me clarify a bit:

One of the things I love about Go is that I don't have to learn all kinds of conceptually heavy frameworks to get real work done. I read the stdlib docs and get to coding.

I've also found that when I'm tempted to stray from Go idioms and do something too generically, I tend to get punished by the language. So I've learned to just suck it up and do everything the "Go way". I don't consider this a negative--rather that Go is just not very tolerant of other language styles. You have to accept this.

I think the best way to describe it would be to make a distinction between frameworks and libraries. A framework integrates itself too much into the application's overall structure, while a library provides another tool that lets you write your code the way you've always been writing it, but to do something new.

The weakness of the type system becomes a problem pretty quickly - simply having a different implementation of an adaptor, be it per language, transport or even simply type of node makes for an endlessly leaking abstraction.

That's what I felt that overlapped with the GP's comments on composing complexity. It's not so much that you _can't_ implement multiple cases (not to the nth degree, just a handful), it's that the abstraction leaks painfully.

I find your and the parent's arguments really interesting.

For me Go reminds me so much of Java 1. And it seems to be getting pulled by the community in the enterprise Java direction e.g. generics, annotations, dependency injection, broad frameworks and the simple, tightly focused tooling direction.

The community isn't pulling Go into any of those things.

* Go had annotations since pretty much day 1. Look at the annotations for JSON serialization, for example.

* Generics is something that Go is probably not getting. It's just become a troll topic at this point since everything that could be said about this topic, has been said.

* Dependency injection is something that Go isn't getting. It's a Java solution for a Java problem. Create well-thought out code with clean interfaces and unit tests... problem solved.

* Broad frameworks: it's just not going to happen. As a lot of other commenters have said, Go just doesn't support the level of abstraction that you need to create a FactoryPatternPatternStrategy. And that is a good thing, in my opinion.

* Tooling: I have no idea what you mean really. Java's tooling is hardly "simple" or "tightly focused" (have you ever USED Maven?) and Go's community didn't "pull" the golang authors to create simple tools... they've been simple and focused since day one.

I would say one of the benefits of Scala is that it allows greater abstraction than some other languages which can provide more clarity. Example a service in Finagle is defined as;

Request => Future[Response]

I don't know what the equivalent would be in Go but I can't imagine its any clearer than this.

    type Service interface {
        Handle(req Request) Response
    }

Now make Request and Response generic so that you can tell different requests and responses apart.

Then make Future generic so that you can use all the existing combinators out there, like `sequence` which turns `List[Future[Response]]` into a Future[List[Response]]`.

Your reply actually highlights one of the key features of Go: if you design it right using channels, constructs like Futures and Promises aren't necessary.

So how would I write code which works regardless of whether I'm using channels, lists or other applicable structures?

> design it right using channels, constructs like Futures and Promises aren't necessary

Careful implying that using futures and promises are incorrect unless you can prove that channels can do anything futures and promises can do while doing it better.

Do you have any examples?

In Go they are incorrect at worst, not idiomatic at best. That's due to the nature of the language as channels are built-in primitives capable of fulfilling the roles of either, not a slight on the efficacy of the constructs themselves.

http://golang.org/doc/effective_go.html#channels

I don't think the idea is to recreate Finagle in Go, but to create a toolkit that solves the same set of problems that Finagle solves in Go.

And I don't think the problem domain is intractably complex; I think we have enough collective operational experience to have informed opinions, and I think Go's type system may be sufficiently sophisticated to encode those opinions.

Go, as of today, is a tailors language. (For context: C#, Java, Scala, etc. are plumbers' tools. And tailoring requires far more élan than plumbing.)

My recommendation to the Gophers is to replicate OTP and definitely figure out dynamic linking. The criticism of the OP are legitimate.

    Go has basically none of the mechanisms for sanely 
    creating composable complexity that Scala has. It makes 
    me really grossed out to think about something with the 
    complexity that Finagle exposes as a Go library. Go is 
    all about small, sturdy tools that work for most of the 
    cases most of the time and that compose well with each 
    other - IF - they are built into the standard library. 
    
    And if it's not, have fun spending your time writing 
    uni-typed interfaces for any kind of interop with 
    desired complexity. Go is great for writing dead simple 
    byte slingers, pipelines that will be rigid in the most 
    dynamic possibility, guns that shoot JS and interact 
    with services, etc. That's simply not Finagle.

A thousand times this.

This is well written.

I gave a talk at the (unaffiliated) Hacker News London Meetup on composing a microservice architecture and why I think a framework to bind today's disparate components needs to exist: http://vimeo.com/118895501

I don't so much believe that any language is perfectly poised to implement this, however - we're working on a framework[1] that focuses on making every language a first-class participant.

Furthermore, it's self-hosting, with most of the core itself implemented as communicating microservices.

It's possible to reimplement bits of the core in different languages!

We've been waiting to finish a polished implementation before releasing anything (and talking to individual companies about their needs), but I'm starting to think that's a bad idea. :)

[1]: http://wym.io/

What's the difference between a spec and a "multi-language" framework? I mean there is none. You can't have a multi language framework without a strict interface therefore a spec. Now what you provide seems to be the actual framework and just not a spec,which is great, but how is it different from Rest or SOAP, in essence?

Also,since languages are vastly different(in features), don't you think it will limit what your framework can do?

Great talk anyway.

I believe however,that having a central authority that orchestrates services is better than micro services that can talk directly to each other.In my opinion,some kind of mediator is mandatory,one that registers services and manage service communication.

Addressing each of your points in turn:

- What's the difference from a spec?

There used to be a FAQ on our site emphasising that at its core _wym_ is a spec. Our reference implementation, itself made up of multiple pieces (wymd, wymsy, oneweb, ...) provides a way to actually make use of that spec and to validate all the guarantees of the strict interfaces.

Also, I'm not a fan of entities who would define a spec in isolation without a reference implementation alongside it - where's the proof that you're making the right decisions? We provide that implementation to keep ourselves grounded.

- Won't multi-language limit features?

Not really.

For one thing, since we're self-hosting we can't afford to lack power and expressivity. For example everyone has the ability to refer to types (as per generics) else we wouldn't be able to implement the core. ;)

Our interface language - it's quite simple, to make the addition of new language bindings trivial, but we expose built-in services to use more powerful features simply by switching which nodes you're holding on to - there's a data-implementation divide, where a node needn't understand a feature in order to deal in its data.

The net effect is that language bindings can be created trivially and evolved to support features as a good way of dealing with them is found (e.g. in the simplest case, Javascript Int64 support).

- Having a central authority is better.

This is an interesting one :D - I agree, in a way. A wymd cluster self-starts a consensus using a lock service, performs master election and then uses this consensus service for registration and discovery.

We have an elected distributed authority. :P

sorry about the off topic, but wym.io w/o js enabled on the page shows this:

> function (){ return React.renderComponentToString(Application(null)); }

Just looks a bit messy tbh. ;)

Hm, that's interesting. I went to great lengths to ensure it rendered without JS (since it's all React)! :P

I'll look into it. Thanks. :)

Looking at the source-code sent to the browser, I'm afraid there's scant evidence of that. I take it the idea was that the snippet in the body (as mentioned above) was to be rendered on the server?

If you figure it out, please let us know/write it up -- being able to use react and have it work without javascript would be a very nice feature.

Ask and I shall deliver. Fixed. :P

I'll upload our build setup right about when I write a blog response to the OP's post. :)

The chief point to note is that the components need to render as you want non-JS viewers to see them when they're passed the default props/state.

Any efforts towards a version-numbering standard for dependencies? Godeps is ok as a workaround, but published versions of things like logging packages, etc, would make me feel a lot safer when depending on them.

And no, I'd rather not 'vendor' source-snapshots of a logging package into every project I ever work on.

If your dependency isn't publishing their own versions of their libraries, maybe they shouldn't be depended upon.

What I mean is... there's a clear and easy way to version libraries in Go - you encode the version in the import path in some way. When the version changes, the import path changes (which means it's a completely different dependency, as far as Go is concerned). How you get those versions in the import path is up to you - you can use gopkg.in, you can host your own redirector, or just have different repos for each version. It's really not rocket science.

For example, I released github.com/natefinch/lumberjack as v1... when I decided I needed to break the API, I released v2 as gopkg.in/natefinch/lumberjack.v2 (which redirects to the v2 branch of the same repo). The very fact that I'm using version numbers indicates I have a commitment to keeping my library stable for third parties. Now, you don't have to trust me, you can use godep or another tool to pin your code to a specific revision of my repo, and if you're really paranoid, you can use godep to vendor my code into your own repo, to insulate you from the possibility that I delete my repo in a fit of anger. (though in theory, that would only stop you for a short period of time, since you should have a copy of my code on every machine that has ever built your code that uses my library).

I just don't understand why everyone thinks versioning in Go is a problem. It's not. It's solved... seven ways to Sunday. It's just solved in a slightly different way than other languages solve it... but so what?

I agree with you: package versioning in Go is solved already.

I think the thing that conflicts people most often is that there is truly no "official" opinion on the subject aside from what you mentioned: use a different import path.

I disagree that it's solved. 90% of community libraries aren't even following the gp's recommendation, let alone minor and incremental release versions. And as you said, there's no standard. With nexus/maven repos, when you push v1.4.2 of your library, that's it, it's only published once and everyone sees the same thing. Bug fixes can go out in 1.4.3, new features in 1.5.x and incompatible features in 2.x. Right now, even if everyone followed GP'S advice, I still risk a bug fix breaking my code, or different behavior depending on when I last ran go get.

It is solved -- just not widely employed.

There is no official stance on package versions, because the core Go development doesn't really need/use it.

The Go 1 compatibility guarantees no backwards incompat changes to stdlib. Experimental package development occurs at golang.org/x/foo ("x" for experimental, presumably). Those packages are unstable, and there are no versions. If they change at all you have to update.

People follow the 'official' approach blindly, but aren't experienced enough to know what the tradeoff's in the official approach are.

I don't understand how any software can possibly enforce semantic versioning. Sure, you can ensure that 1.4.2 is immutable, but you can't enforce that breaking changes aren't put in 1.4.3.

As I said, there are tools that let you freeze the exact commit number your code depends on, which effectively gives you frozen minor version numbers if you want it. And of course you can also copy the code into your repo with godep, and not need any external services outside your own VCS.

> you can't enforce that breaking changes aren't put in 1.4.3

Eh ... why? In some languages, it's standard to run source/binary compatibility checks against new versions and fail a build if the new version doesn't satisfy the requirements.

With decent static analysis you could check that the signatures of all types and functions are identical or compatible, I suppose, but checking that the functionality is compatible in any kind of generic way is impossible. The former is necessary, but not sufficient.

> With decent static analysis you could check that the signatures of all types and functions are identical or compatible, I suppose

"I suppose"? This has been done for years already. There are even GUIs for that (so that people don't have to use the CLI) which tell you "between 1.2.3 and 1.2.4 you did the following: ... This will break compatibility in the following cases: ..."

So it's better to not even try?

Additionally, for many changes it is reasonable and possible to check whether behavior has changed. Not every function has an unlimited number of inputs and outputs.

That's one of the biggest reasons I avoid Go and PHP: It's not only that the developers ignore existing solutions if they haven't "invented" them themselves, but that they don't even care about improving things where no "perfect" solution is available.

Ignorance and arrogance is a combination which doesn't make me trust them, and products based on their software. (OwnCloud, Docker, ...)

I'm not saying "don't write a tool that does this". Sure, having an API diff tool for my code would be awesome. And it's possible someone has already written such a thing for Go. If not, someone definitely should.

The problem I have is when other developers say "How can anyone possibly live without this?!" People have been living without X or Y feature some other language / toolchain has for a long time, and doing pretty ok. That doesn't mean we don't want that feature, just that we don't think it's the end of the world if we don't have it. Everyone seems to have their own pet "I can't live without this" thing. The thing is, when you've been coding for a long time, you start to realize that you can live without pretty much any feature. It may make some parts of your job harder, but unless we're talking about dropping down to assembly, I can deal with pretty much any language.

Well, let's just agree that there is some kind of selection bias in the Go community.

Go developers seem to be emphatically content with not using the best tool for the job. That makes sense, otherwise those developers wouldn't have ended up with Go in the first place.

For me it's not a question about "I can live without X?", but "why would I want to live without X?". I want the best tools to solve a job. Not some of the best tools. _All_ of them.

I'm gonna just throw it out there for you: You Do Not Get It when it comes to Go. Internalize that. Nobody, fucking nobody ever, has said "this tool sucks, I think I'll use it exclusively from now on".

They're seeing something that you are not seeing, most likely an appreciation of simplicity.

> Nobody, fucking nobody ever, has said "this tool sucks, I think I'll use it exclusively from now on".

You obviously don't read the Golang mailing list ... :-)

You read the golang mailing list? In addition to showing up on every HN and probably proggit thread about Go? How much time have you spent actually programming?

godep isn't really meant to be for versioning dependencies. It's meant to freeze source code so that you can reproduce builds 100%.

With that being said, I'd recommend http://gopkg.in, or if you prefer hosting yourself https://azul3d.org/semver

Disclaimer: I created `azul3d/semver`.

That's an interesting point. Does Go have an equivalent of Sonatype's Nexus ?

This is basically mandatory in most enterprises since often you can't just go into the wild to pickup new dependencies. Plus keeping build artefacts managed is pretty important.

Not really. Moreover, as far as I know there really isn't a good story around shipping dependencies in binary format rather than source code (as Maven repositories do by default).

It is fine to assume the availability of source code when you're working on tiny web microservices... and you're only depending on things like Gorilla, or whatever the leading Sinatra-clone is this week. However, a typical "enterprise" project has numerous proprietary dependencies for which the vendor doesn't want to ship source code.

It will be interesting to see whether the Go community eventually adapts to fit enterprise culture, whether enterprise culture adapts to fit Go's model, or whether the two worlds never really come together at all.

Man, that page needs a "What is Nexus?" link. I can't really tell what it is.. I guess some sort of central repo for packages?

What go has is Godep (https://github.com/tools/godep) which with a single command can copy all your project's dependencies into your repo. from there on out, the dependencies are just part of source control like anything else, so you don't have to open up your build machine to the external internet, for example.

Yes. It is just a central repo that also does caching/proxying of external repositories. In many enterprises developers won't have access to download from the external internet so godep won't work.

It also has a fine grained security model and allows for security evaluations of libraries (check for outdated/insecure etc).

The problem is in enterprise Go is really hard to push currently due to management no having proper understanding of it. I believe Go is the future and I recently tried to push Go in using it to write a new semi-real time system to handle many little messages with complex business logic but nope CTO went with node.js and have a callback hell.

Sorry didn't mean to rant... Love Go its the future, use it.

> I believe Go is the future and I recently tried to push Go in using it to write a new semi-real time system to handle many little messages with complex business logic but nope CTO went with node.js and have a callback hell.

I used to think and hope Go was the future, and I agree it's better than node.js's callback hell, but I no longer hope Go is the future.

Something like Nim[0] is an improvement in every way.

Not my main point, but I personally moved from Go to Haskell. So, If you don't mind educating your developers Haskell, Scala, or Ocaml offers you true type safety and compile time correctness.

0: http://nim-lang.org/

In this instance, I'm not sure that Haskell will help. If the GP couldn't get Go past the CTO, I strongly suspect that Haskell would also be vetoed.

So why is node, with it's callback hell and single thread, worse? Is it syntax? That single thread assumption can free you from some significant problems, you know. golang's own http lib is event driven and uses callbacks, but perhaps less hellishly. I agree it is hard to push go if you don't properly understand it.

Go's http package doesn't use callbacks at the API level. See here http://golang.org/pkg/net/http/

Comparing Go and Node is like asking whether someone wants his paster served with vomit or feces.

Yes, some people might prefer one over the another, but asking how they ended up with such a ridiculous issue would be a much better question.

As for metrics - we've been working precisely on that @ https://deferpanic.com . To some extent logging as well. This means (exactly as OP mentioned) - memory, gc, go-routines, request durations, db query latency, logging errors/panics, etc.

We are extremely open to suggestions/feedback.

One key point I'd like to point out though, is that OP specifically mentioned the go kit being targeted towards a company with "100–1000+ engineers". I think this is highly on point - if you have less engineers than this you probably don't have enough resources to maintain these sorts of systems yourselves and this is where we think we can help out.

When we read the word enterprise we probably think of older, slow-moving bureaucracies, like IBM, HP, or even Red Hat. But it’s been a long time since companies like those have been technical leaders. Decades, in some cases. Now, companies like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or even SoundCloud set the tone for our industry. These modern enterprises tend to share a few characteristics...

Ouch! I work for HP and while I agree that we're not exactly a nimble, fast moving startup I think it's not a good comparison. IBM and HP still build a lot of physical things while the listed "modern" enterprises are primarily SW. Google would be the exception here but from an engineering point of view Google uses it's vast war chest to play around with very cool HW but isn't much concerned with running a business based on that. So, I hope Go has a good run of it but it's not going to build your next printer.

He's talking about software.

This is a great start. I'm actually in the middle of a writing a Golang service, and this list checks off everything I've been dealing with.

A lot of these things aren't golang-specific, though. Ideally it would be easy to swap in services written in different languages, and the conventions stay the same.

This is great. I hope something like this is built. Go needs it.

However, not only did we need something like this 5 years ago, we needed it to be language/platform agnostic then. We need it more so now. We won't be able to function without it 5 years from now.

The reality is Pandora's box has been opened there is no going back. We have software we need to run in production at scale that is written in a multitude of languages. We have our Python/Perl monitoring systems talking to our Java search APIs talking to our NodeJS frontend APIs talking to our legacy C#/Java/C++ applications over messaging systems written in Erlang/Ruby/Java/Go.

Every single one of these platforms need this functionality and everything needs to run in a cutthroat environment with fewer people, resources, and money to manage it.

We need performance, discoverability, security, scalability, deployability, testability, reliability and most importantly EASY. Not enterprisey over-engineered bullshit. I mean real easy to use plug and play, simple APIs, minimal JSON/YAML, small focused tools simple.

We need to do it in a way that is portable to every cloud provider so we can get the best bang for our buck and keep the cloud providers honest and avoid lock in.

We need to do it in a way that makes the "hardware" vanish. We don't want to manage hardware anymore. We want to deploy our software and we want to know it will work and we don't want to compromise functionality (ala Google App Engine or being forced into the "Tomcat 7" box).

I applaud this line of thinking, but I don't think it goes far enough. In 5 years time, I don't think we'll deploy servers anymore. We'll deploy services and those services will be in whatever language and those services will find each other and communicate with each other over the network and will be reliable, secure, and scalable.

Most importantly, it needs to be open. Truly open. We don't want to trade one master for another.

That's the software I would like to see. This could be a foundational component of something like this, but it's only the beginning.

I know about CoreOS/Docker/Mesos/Marathon, etc. Tools like those are an important piece of this puzzle, but not the only piece. They are a vast improvement over past initiatives (cough openstack cough), but they too are only a step in the right direction. We still have a lot of work to do to realize this vision of the future.

It sounds like you want Cloud Foundry (or OpenShift, which I have no experience with). (http://www.cloudfoundry.org/about/index.html)

I am old enough that I used CORBA. That basically checked all your boxes... multi-language, platform agnostic, discoverable, standardized serialization formats, you name it.

It was also an overly complex piece of shit.

People learn nothing from the past.

I beg to disagree. I just gave a talk[1] about exactly how we[2] went back and took time to learn lessons from designs like CORBA (we're building another microservices framework).

Not least I talk about how it's important to build something that modularises complexity and above all else, focuses first on being _usable by humans_ rather than on technical ascension.

_Some_ of us _are_ listening.

[1]: https://vimeo.com/118895501

[2]: http://wym.io/

I agree in the broad strokes. But the only way to this utopia is stepwise. I hope this Go kit is one of those steps.

Agreed. My hope is that we don't stop at this, however, we continue to push forward. Too many times we build the beginning of something great but never see it through to the end.

Uh oh. The "framework people" found us.

Hopefully they'll be too busy writing AbstractFactoryBuilderServiceDispatcherProxies to actually do any damage to production systems...

I've talked to the author extensively about these ideas, and let me assure you he is firmly in the "no monolithic frameworks please" camp. (He's also the author of some great Go code, btw.)

What he proposes here looks more to me like a standard library for building distributed systems, which makes a lot of sense to me.

I don't see the difference between "a standard libary for building systems" and a framework. Just write libraries for what you need and forget the idea of some overarching framework.

When we built the Go standard library it was valuable for us to think of it as a whole. What would most of our users need? Are there any gaps? I think the gokit project is a good opportunity to focus the energies of likeminded people in the Go community, to close the gaps and work together.

Hey just curious, why did you pick spacemonkeygo/monitor over mozilla-services/heka for data collection?

I was thinking of using heka for a logstash-like project and just wanted to see if you had found several know issues, or that monitor really shined, or that maybe the feature set you needed just didn't overlap to the point they were comparable for your needs?

It wasn't chosen for anything, it was just a related project. And I guess not for its data collection functionality -- that's orthogonal to the goals of the toolkit -- but presumably for the instrumentation and request tracing bits.

Sounds like a package manager with s/glibc/go packages/ and s/otherlibs/go packages/

Not necessarily bad but.

In fact this is similar to how Go replaces the OS scheduling and message passing.

Maybe its a necessary path to evolution. Til the OS itself can get fixed without "it doesnt run X or Z" problems.

I´m all for in increased uptake of go. But I don want it getting dragged into Enterprise with a big ¨E¨ and going the way of java with its monolithic enterprise frameworks.

Somehow we will end up losing the features of the language that make is currently so endearing.

Golang just needs more database drivers for enterprise products (Informix...), and some libraries that make office products and the windows enironment (active direcory...) more assimilable.

I completely agree. Active directory. Better database drivers have blocked me from using Go on several projects at work.

You mean Go needs libraries for the "enterprise" world, because devs actually need languages that integrate well with their client's infrastructure.And you're totally right.

I would really like to have a Golang Teradata driver, however I've thus far made do with routing it through ODBC and using that driver instead. Not ideal, but functional.

A related musing about Go I wrote somewhere else before...

Option types as well as lack of support for things like either and generics are why Go quickly became unappealing to me for real world work.

Check out this haskell example (note that the exceptions are values as well, so this is more flexible than most languages try or panic in Go):

    failureExample :: IO (Either SomeException (Response LBS.ByteString))
    failureExample = try $ do
      get "http://www.httpbin.org/status/200"
      get "http://www.httpbin.org/status/200"
      get "http://www.httpbin.org/status/200"
      get "http://www.httpbin.org/status/404"
    
    main = failureExample >>= \case
      Right r -> putStrLn $ "The successful pages status was (spoiler: it's 200!): " ++ show (r ^. responseStatus)
      Left e -> putStrLn ("error: " ++ show e)

The equivalent Go is much longer and needlessly verbose:

    func failureExample()(*http.Response) {
        // 1st get, do nothing if success else print exception and exit
        response, err := http.Get("http://httpbin.org/status/200")
        if err != nil {
            fmt.Printf("%s", err)
            os.Exit(1)
        } else {
            defer response.Body.Close()
        }
    
        // 2nd get, do nothing if success else print exception and exit
        response2, err := http.Get("http://httpbin.org/status/200")
        if err != nil {
            fmt.Printf("%s", err)
            os.Exit(1)
        } else {
            defer response2.Body.Close()
        }
    
    
        // 3rd get, do nothing if success else print exception and exit
        response3, err := http.Get("http://httpbin.org/status/200")
        if err != nil {
            fmt.Printf("%s", err)
            os.Exit(1)
        } else {
            defer response3.Body.Close()
        }
    
    
        // 4th get, return response if success else print exception and exit
        response4, err := http.Get("http://httpbin.org/status/404")
        if err != nil {
            fmt.Printf("%s", err)
            os.Exit(1)
        } else {
            defer response4.Body.Close()
        }
    
        return response4
    }
    
    func main() {
        fmt.Println("A failure.")
        failure := failureExample();
        fmt.Println(failure);
    }

This code is all tested/runs and I might be putting this along with other issues I've had in a blog post, but no promises. I think this makes it very apparent why some people have an issue with Go, namely that it feels like going backwards and ignores lots of prior research at the altar of some deluded notion of pragmatism. I say that despite a Go Gopher staring at me on my desk.

I'm not familiar with Haskell so I can't exactly follow what you're trying to do, but I can take a stab at improving what your Go code is doing:

    package main

    import (
        "fmt"
        "net/http"
    )

    func main() {
        for _, url := range []string{
            "http://httpbin.org/status/200",
            "http://httpbin.org/status/200",
            "http://httpbin.org/status/200",
            "http://httpbin.org/status/404",
        } {
            fmt.Printf("%s: %d\n", url, checkUrl(url))
        }
    }

    func checkUrl(url string) int {
        response, err := http.Get(url)
        if err != nil {
            panic(err)
        }
        defer response.Body.Close()
        return response.StatusCode
    }

You say of your Haskell code that errors are values, and Go is no different in this regard. checkUrl could easily be modified to return (int, error) and return both values, rather than panicking when an error occurs.

Cool, that's a good refactoring.

What if we don't care about the errors and just want to return nil or the value. The caveat is that when any of your http requests get a non 200 response code no further processing is done and nil is returned.

What I would do in that case is modify checkUrl to return a bool, change the return to return response.StatusCode == 200, and change the for loop contents to

    if !checkUrl(url){
        fmt.Printf("Non-200 response for %s", url)
        break
    }

> In this environment, a language like Go has the opportunity to shine.

Go has another advantage over Scala (and any other JVM langauge), which also happens to be it's killer feature - green threads, the ability to write non-blocking concurrent code in a synchronous style without callbacks. Yes, I am sure Scala has some "leaky abstraction" over what are essentially just callbacks, as does Java, but it's not quite the same.

> Go has another advantage over Scala (and any other JVM langauge), which also happens to be it's killer feature - green threads, the ability to write non-blocking concurrent code in a synchronous style without callbacks.

"Green threads", per se, aren't an advantage. M:N threads are potentially an advantage, since they allow both hardware parallelism and concurrency with a greater degree than the degree of hardware parallelism without the overhead of native threads for each concurrent task.

OTOH, nothing stops either JVM implementations (when compared to the underlying platform) or JVM languages (when compared to the JVM itself) from using an M:N threading model.

> Yes, I am sure Scala has some "leaky abstraction" over what are essentially just callbacks, as does Java, but it's not quite the same.

Green threads (whether 1:N or M:N) are a (potentially leaky) abstraction over what are essentially callbacks, too (while a well-designed green thread implementation can avoid this for code written entirely in the language, for calls linked code written in other languages that isn't specifically aware of and designed to cooperate with the green thread implementation, there is the risk of actions which block the underlying native thread, which will then block all green threads that are running on the same native thread.)

No, it's certainly not an advantage in every case, because that's all you can use. It's not always the best tool for the job. Under the JVM, you have a massive amount of library support for multithreading and concurrency. You have things available like Futures, STM (Software transaction model), Actor Model (Akka), parallel collections, etc... Not to mention taking advantage of immutability, which can simplify concurrency a lot in some cases.

It's not a one-size-fits-all type of game here. So having one choice will essentially lock you into what you can do with it.

I also wouldn't say a Future, which is a Monad, a leaky abstraction. You don't really need to understand monads to use them, in fact, a lot of people use what are monads all the time, without knowing it.

> STM (Software transaction model)

I thought this was software transactional memory? Java has an implementation?

Oops, not sure why I put that, it's indeed software transactional memory. Akka is a pretty known library that supports it.

I guess you haven't read the JLS.

There isn't any requirement for red threads in certified JVMs and a few of them do offer green threads, which is actually how the first versions used to do threading.

Well I meant that in regard to which JVMs people actually use...not what are theoretically possible.

Even then, Java's had ExecutorService, Callable<T> and BlockingQueue<T> for a long time. Effectively the same thing as Go's green threads from a programmer's perspective, with the only real difference being that Go swaps out an executing goroutine on I/O or other syscalls (or calling any C code).

Newer Go ersions will also preempt green threads on normal functions calls and not only on syscalls. Both is not possible with the JVMs primitives where you need "cooperative code".

However there exists Quasar for the JVM which seems to be able to emulate something similar to green threads with the help of bytecode weaving (and inserting preemption points).

Ok, fair enough, but if IO isn't happening, who cares if it's a runtime or the OS swapping out the 'thread'?

I was also talking about JVMs people actually use.

There is much more out there than OpenJDK and Oracle commercial JVM.

If you want a real product example,

http://www.is2t.com/products/specifications/

I wonder if this guy has actually worked in the "enterprise" before because it sure isn't what he is talking about. Companies like Facebook, Soundcloud etc are nothing more than bigger startups. Go work for a bank sometime and then tell me they are comparable to Spotify.

As for Go. What it really needs is a great story around Java integration. JDBC drivers and libraries need to just work out of the box. Then you can talk about building cute little micro services.

And Scala's toolchain is pretty simple at least with something like Play.

He led his article with a distinction between the "enterprise" that you refer to, and the "modern enterprise" (Twitter, Facebook, Google, ...) that he discusses in his article. This article specifically focuses on the problems of those companies.

No he talked about enterprises in terms of IT companies. What I am referring to are enterprises that aren't in the bubble of the IT industry e.g. banks, insurance companies, retail etc. You know what the rest of the world defines enterprise to be.

You have different problems and pressures when you don't have hundreds/thousands of highly experienced engineers.

"IT companies" like Google, Amazon, Twitter, Netflix, and SoundCloud are the ones driving technical conversations and advancing the state of the art. Banks, insurance companies, and retail outfits aren't, and haven't been for a decade or more.

Except that Banks, insurance companies, and retail outfits is where the real money is.

Google, Amazon, Twitter, Netflix, and SoundCloud are for engineeers living in Silicon Valey.

I can't tell if you're trolling or serious. Retail is where the money is, but Amazon, a Seattle-based company, is for engineers living in Silicon Valley?

How is the sector at all relevant? (Software is eating the world, and all that.)

I am serious, just didn't bother to check where their HQ are located.

Google, Amazon, Twitter, Netflix, and SoundCloud and a few others are pointed out in HN as enterprise examples, and they are.

However outside the HN bubble, which I tend to associate with SV, the enterprise looks a bit different.

It is those Fortune 1000 companies, most of which don't have software as main goal, are happy to outsource anywhere in the world, still run mainframes and commercial UNIX systems, have rigid processes in place and so on.

This is the enterprise I know from 20 years in the business. Amazon et all might be the modern enterprise, but they are a very tiny piece of the pie for anyone working on IT around the globe.

Walmart and Sam's Club are both actively developing in Go.

Agree. I also wonder what this inferiority complex "hurr durr, $X is not written in Go, therefore we need to duplicate it in Go!!!" is about ...

Look, it's pretty obvious from your comments that you're not a fan of Go, but if you can come to terms with the fact that some people enjoy languages you do not (or their use is mandated by their work environment) then you'll also be able to understand that they would also like certain tools and libraries available to them in that language.

It's not about recreating everything in Go "just because", it's about being a Go developer and wanting those tools to be available to you.

This highlights a bigger problem: why should you have to reimplement the same idea hundreds of times if what you already want to do already exists?

I don't look at it as reinventing the wheel, any more than one would consider the advent of new languages and technologies a reinvention. They're iterations on the concept, bringing that concept into a new environment to be further refined or built upon.

Other threads in this discussion bring up Finagle's shortcomings, partly due to certain features not being addressed or included by Finagle's authors due to them being low priority. In that case what you want may not already exist, but may work as inspiration for another incarnation.

I once saw Douglas Crockford, when asked about JSON in comparison to XML, say "The great thing about reinventing the wheel is that sometimes you get a round one".

I don't think it's _always an_ inferiority complex.

I've personally run into many situations in which porting a lib to Go is easier -- and more efficient -- than my brain's context switching between multiple languages in order to fix issues with the original lib.

Reply on news.ycombinator.com