REST Servers in Go: Part 1 – standard library
eli.thegreenplace.net
eli.thegreenplace.net
This is mostly the way today services in big OSS projects are exposed outside for consumption in a RESTful style. One exception I know of is sourcegraph/sourcegraph.
gRPC feels to me like a violation/break in expectations of how the layers of abstraction are supposed to work -- gRPC works over HTTP/2, and it's weird that it translates a lower level (HTTP/1) at a higher level (gRPC) to re-encode it into something at the lower level. An appropriately robust protocol would simply support both lower levels of abstraction if it wanted to (and they were widespread in daily use), right?
The only places I feel like I see this kind of layer violations are in lower level networking and it generally just makes everything worse and more complicated there. Of course, I know that gRPC + gateway is actually not a huge deal in practice -- you've got reverse proxies like envoy that will do it for you automatically[0], but it just... doesn't sit great. The benefits of gRPC are not to be sneezed at (better performance, strict typing at the protocol level, schema enforcement, bidirectional streaming, etc), but it feels like it could have accomplished a lot of those goals without throwing out HTTP/1.1 completely (and then that machinery could have been reused to support HTTP/3).
[0]: https://www.envoyproxy.io/docs/envoy/latest/configuration/ht...
[0]: https://rsocket.io/
You only get a certain number of complexity tokens and IMO it's not worth spending any here.
Between a few breaking upstream changes in grpc-gateway (iirc), learning a new framework vs the stdlib + chi router setup that had already been proven in the greater team, feature shipment dropped through the floor. Couple that with the fact that, as stated, these benefits come with well-iterated data models; conversely, our _not_ well-established data models proved to be a constant PITA when having to regenerate this file and that file. No one used the auto-gen'd docs because they were constantly changing anyway. And integrating with our CI/CD pipeline was a nightmare.
Between providing value with a team's previously proven tools and spending innovation tokens, I'd definitely suggest weighing carefully if you need this to be one of them (bi-directional streaming and a well-established project seem like good candidates though).
I believe it supports, http, websocket and grpc.
These are meant to fit grpc-gateway to scenarios where gRPC server might be implemented in another programming language, or accepting clients at a TCP port or another kind of a socket, possibly under a different network realm than grpc-gateway. Pretty much like a reverse proxy, but can also be integrated with services in the same code level, escaping the overhead of a transport layer.
I don't think there's been a single instance where I've thought "why can't stdlib do this?" nor "why the heck is this in stdlib?"
I've had this a few times, most recently with "how do I add this data file to my binary". At least that one made it to master now, and will be in 1.16!
Another gripe is the lack of a proper parallel safe map (no, map[interface{}]interface{} like sync.Map is just not acceptable) which would be a godsend and should honestly just be the normal map implementation. Maybe eventually with generics... (This is more of a language gripe than a stdlib gripe, though, and I'm sure that once generics come out we'll see a sync.Map that doesn't use interface {})
And before 1.16, there is statik: https://github.com/rakyll/statik. Creates an embeddable file system from files or directories. It’s awesome for packaging web front ends into binaries.
0: https://go.googlesource.com/proposal/+/master/design/draft-e...
Ok fair enough, that one bit me too, but honestly I only felt cheated because Rust makes it so easy. `include_bytes!` was like cheating the first time I used it.
Wait, how?? I've done some unholy things.
package main
import (
"embed"
"net/http"
)
//go:embed assets/*
var assets embed.FS
func main() {
fs := http.FileServer(http.FS(assets))
http.ListenAndServe(":8080", fs)
}
0: https://go.googlesource.com/proposal/+/master/design/draft-e...
> I know people will complain about speed of execution of language and framework but do you really care if you are not expecting Google like traffic.
If you don't care that's fine, but there are lots of us that do. This is subjective, but I find the memory use and overall performance of Rails/Django/Spring Boot applications to be completely unacceptable and there are far too many breaking changes.
If we're talking real time sub-millisecond performance, then Rust.
I know this is all opinion and I have colleagues who are excellent engineers that prefer rails, but it goes against everything I enjoy about software development.
Having worked a couple places where Python or Go microservices were hyped up, but every installation, DB schema, and folder structure seemed to be built with a different idea in mind, and or debated, I have a strong appreciation for right standards around convention. Especially when I don’t have to get into debates like whether DB table names should be plural or singular.
On the side I really do enjoy working with Go though, and wouldn’t hesitate to use it if it seemed like the right tool for the moment.
I’m not super convinced that’s a bad thing though. The software industry has always suffered a perennial shortage of engineers. What I predict will happen is that an ecosystem of tools will spring up around rails and there will be serious investment in improving performance rather than the framework being abandoned.
Remember how kubernetes changed infrastructure development to managing config files? It’s what I see happening to most areas of software development. The more experienced software engineers will then be responsible for optimizing/bug fixing/ scaling.
> beginners can use it without understanding too deeply...So there’s just going to be too many folks who will choose rails over more suitable languages.
It's not about "beginners" or "more suitable languages". You need an ability to build highly specialised software and you need general frameworks that will not be optimised for {specialised thing} but can do most of the things that you need for a certain task without having to hand write everything. Without the latter, all that happens is constant yak shaving and reinventions of the wheel. Rails (for example) is highly suitable for many things, getting clever points for not using it is great, but it's not very practical.
Bootcamps sell themselves based on employment placement, not cool technology or deep knowledge. In this regard RoR, Java, and even PHP are going to be bootcamp milch cows for years to come.
There is a language that has faster-than-go speed and better abstractions, but HN seems to be somewhat decided on whether they think it's awesome or terrible, and there was recently an blog post on front-page about how it was bad for APIs (which I heavily disagree with but I am biased so I opted not to comment on).
Go definitely does have some other web frameworks that make work like this simpler, but on the other hand, people sometimes shy away from those because the stdlib is more universal.
If we're talking just pure abstraction I think there are a lot of other choice that could have delivered similar improvements in performance for an IO-dependent workload, with better methods of abstraction, ecosystem, and safety -- that choice for me would be Typescript.
[EDIT] Also to note, even as a Haskell zealot I'm not crazy enough to suggest someone choose Haskell as an alternative where Go would have been good enough. I have enough experience with business needs to know that the purity/safety/whatever other benefits of haskell just aren't worth the lack of ecosystem, difficulty in finding developers, and hit to developer speed. Haskell is too far on the spectrum (on various axes) to be the right choice most of the time, and not enough companies have shared how they've outperformed with it to even start the conversation. Haskell is like the mercedes of programming languages -- airbags show up there first, but regular cars get them eventually.
Rust is hard, but it's not harder than Haskell, and at a high level of abstraction it can be simpler than Golang has to offer in the stdlib, in the happy path with better results, and more safety. Again, this is the happy path case, but it's not impossible, just hard/unlikely -- Golang on the other hand can never achieve this level of simple interfaces built on abstraction because of the goals and design choices the language has made.
The original comment is this:
> Looking at such articles makes me feel we are going back in time rather than improving efficiencies for developers to build RESTful server. If you look at Ruby on Rails you can build the server shown here in one min that is scalable and backed with database. I know people will complain about speed of execution of language and framework but do you really care if you are not expecting Google like traffic.
My point was that Rust gives you the tools to write a rails/sinatra (at a glance) library that in the happy/simple path (which most regular CRUD backends are) can be simpler than Golang because the abstractions to make it simple are there, and speed will just about always be the fastest possible relative to quality of underlying code. Golang can (and does) provide similar, but it is always a step behind (on purpose) on the abstraction front. If you're going to provide a carefully crafted interface that is very easy to use, it matters less that rust can have a really high barrier to entry (rails is valuable because you can be productive without being a ruby expert).
The problem is that as soon as you need something more than a command line call the engineering burden explodes relative to languages that natively support greater abstraction at the lexical and logical level.
True, most languages could do it, but there are some hard stops to how easy it is to abstract in golang, and performance also knocks some languages out.
> The problem is that as soon as you need something more than a command line call the engineering burden explodes relative to languages that natively support greater abstraction at the lexical and logical level.
Agreed, once you're off the happy path things can be many times more painful in Rust than Go. I personally think Go will replace Java in most enterprise-y software shops within 5-10 years. Unless there's a specific reason to use the JVM, Go is more than good enough and it's goals align with industry very well.
It's not about whether you expect Google-like traffic, but about whether you expect a Google-like ratio of traffic/compute to capacity. My day job doesn't deal with "Google-like" traffic (maybe one Google product's worth of traffic), but we also only have about 10 servers to ingest what we do get. My current side project I expect to deal with even fewer orders of magnitude but I also want to be able to run it on a single server to keep costs and operational overhead low.
> There are strong opinions both for and against using frameworks. My goal in these posts is to examine the issue objectively from several angles.
My only question is why the server / HTTP handlers have to deal with the Mutex. That seems like a "leak" from the `TaskStore` abstraction, which otherwise I really like. (Thank you for not using channels in that interface!)
store.Lock()
store.WriteKey("foo", "bar")
x := store.ReadKey("foo")
store.Unlock()
// x is always "bar"
store.Lock()
store.WriteKey("foo", "bar")
store.Unlock()
store.Lock()
x := store.ReadKey("foo")
store.Unlock()
// x could be whatever another goroutine set "foo" to, not the "bar" that you just wrote.
var x string
err := db.DoTx(func(tx *Tx) {
tx.Write("foo", "bar")
x = tx.Read("foo")
})
if err != nil { ... }
// what x is depends on the details of your database; maybe you're running at "read uncommitted", maybe you're running at "serializable".
Can someone chime in? There must be an obvious advantage that I might be missing.
The act of writing these kind of systems is IMHO, somehow, liberating.
Compiled code is also significantly faster in many cases.
2. execution speed - Django's ORM comes to mind
3. concurrency
That being said there are alot of great features with python web frameworks that come out of the box. You need to a lightweight project like gin/echo or something to get these features. Naked net/http is sorta like the 'erector set' of web frameworks.
I remember having the same sentiment a few months back, but investing a considerable amount of time planning the structure of serializers and sticking to DRF's patterns did reduce the amount of "fighting" that I need to do to make my API work as expected.
With "turning it on", you mean you write a systemd configuration file and start it using systemctl, right? Just curious how people do this stuff nowadays.
type Store interface {
GetTask(*Task) error
}
task := store.Task{ID: 5}
if err := db.GetTask(&task); err != nil {
// wahtever
}
task := store.Task{Tag: "foo"}
if err := db.GetTask(&task); err != nil {
// wahtever
}
I mean ... for the second problem that's broadly true of making any changes to your data access layer since by definition your Go compiler is not going to, for example, check the validity of a SQL query. So ... yes? but that's not unique to this approach, that's true generally.
I'm curios about the way you handled zero values, field masks could be a solution, but I think it would get bloaty.
func (db *actualStoreImplementation) GetTask(t *Task) error {
if (t.ID != 0) {
// query by ID, mutate the parameter, return nil
}
if (t.Tag != "") {
// query by tag, mutate the parameter, return nil
}
return ErrWhatever
}
func (t Task) Public() wire.Value {
return wire.Task{
// explicitly generate what you want
}
}-CORS support. Deploy this to a non-local domain and try to reach it from a web browser and it will fail. I like https://github.com/rs/cors. I had rolled my own but then moved to that library.
- input validation. I like go-playground/validator.
The other big issue is the locking by hand around the task store. In reality usually there would be a database to handle concurrent read/writes. I use SQLite in production. I know this is just a demo and you want to use just stdlib, but serializing all data access is sort of unacceptable as a solution in a concurrent language like Go. When I'm not handling concurrency with SQLite I like to implement The actor pattern, having a persistent goroutine listen and respond to "taskstore" requests via channels.
Maybe this is just my opinion as I started a significant golang app with just vanilla stdlib and then added each of these complexities in turn, before switching to libraries to solve most of the boilerplate better than my hand-crafted patterns
It contained no explanation or reasoning to that end. It is purely what it is titled as: An explanation of how to do it with the standard library.
In that sense I don't think there is anything wrong with it. It doesn't make any claims; it simply provides the information it said it would.
Also, one of the main selling points of Golang is that it aims to deliver functionality with minimal additional cruft/add ons. Many of the standard library things are this way and don't provide the "extra stuff" you are desiring.
You mention auth, but many of the frameworks for Golang don't even implement auth and merely provided the ability to add auth if desired.
Logging is a debatable topic because the "standard" logging method is very inadequate, and so each of the various frameworks has to just choose a random logging system to use. ( or several )
While some frameworks do offer the ability to restart without killing active connections / requests, it is a more generalized issue with any service in Golang. It is also best addressed by writing your service in a scalable fashion and doing a rolling upgrade. Stop sending traffic to an instance, wait till current requests are done, then kill it. It doesn't need to be addressed within the service itself.
The example is simply the beginning. It can be extended as needed depending on your use case. This is how most things Golang work. You start with the basics and add what you need on a case by case basis.
If you want something with every bell and whistle out of the box, Golang is going to be a disappointment generally as that is not the Golang mentality.
There are of course sufficient frameworks in Golang these days to provide such things though. Hence the article clearly says "standard library" to indicate that it isn't the last word on how to setup a REST service in Go.
1) Be careful with locks in the form "x.Lock(); x.DoSomething(); x.Unlock()". If DoSomething panics, you will still be holding the lock, and that's pretty much the end of your program. ("x.Lock(); defer x.Unlock(); x.DoSomething()" avoids this problem, but obviously in the non-panic case, the lock is released at a different time than in this implementation. Additional tweaking is required.)
Generally I don't like locks in the request critical path because waiting for a lock is uncancelable, but in this very simple case it doesn't matter. For more complicated concurrency requirements, consider the difference between x.Lock()/x.Do()/x.Unlock vs. select { case x := <-ch: doSomethingWithX(x); case <-request.Context().Done(): error(request.Context().Err()) }. The channel wait can be cancelled when the user disconnects, or hits the stop button in the error, or the request timeout is reached.
2) Long if/else statements are harder to read than a switch statement. Instead of:
if(foo == "bar") {
// Bar
} else if (foo == "baz") {
// Baz
} else {
// Error
}
switch(foo) {
case "bar":
// Bar
case "baz":
// Baz
default:
// Error
}
3) I have always found that error handling in http.HandlerFunc-tions cumbersome. The author runs into this, with code like:
foo, err := Foo()
if err != nil {
http.Error(w, ...)
return
}
bar, err := Bar()
if err != nil {
http.Error(w, ...)
return
}
func DoTheActualThing() ([]byte, error) {
if everythingIsFine {
return []byte(`{"result":"it worked and you are cool"}`), nil
}
return nil, errors.New("not everything is okay, feels sad")
}
func ServeHTTP(w http.ResponseWriter, req *http.Request) {
result, err := DoTheActualThing()
if err != nil {
http.Error(w, ...)
return
}
w.Header().Set("content-type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write(result)
}
I don't disagree that it's more practical, initially, as you've described it. However, I think it's important to point out the tradeoff rather than presenting it as purely more efficient. I've seen this approach result in poor separation of concerns and bloated use cases (`DoTheActualThing`) which become tedious to refactor, albeit in other languages.
One predictable side effect of the above, if you're working with junior engineers, is that they are likely going to write tests for the application logic with a dependency on the request / response as inputs, and asserting on status codes etc. I shudder to think how many lines of code I've read dedicated to mocking req/res that were never needed in the first place.
w := httptest.NewRecorder()
req := httptest.NewRequest("GET", "/foo", nil)
ServeHTTP(w, req)
if got, want := w.Code, http.StatusOK; got != want {
t.Errorf("get /foo: status:\n got: %v\n want: %v", got, want)
}
if got, want := w.Body.String(), "it worked"; got != want {
t.Errorf("get /foo: body:\n got: %v\n want: %v", got, want)
}
Complexity comes from generating requests and parsing the responses, and that is what leads to the desire to factor things out -- test the functions with their native data types instead of http.Request and http.Response. I think most people choose to factor things out to make that possible, but in the simplest of simple cases, many people just use httptest. It gets the job done.
The problem I've seen is over-dependence on writing unit tests with mocks instead of biting the bullet and properly testing all the boundaries. I have seen folk end up with 1000+ tests, of which most are useless because the mocks make far too many assumptions, but are necessary because of the layer coupling.
This was mostly in Node though, where mocking the request/response gets done inconsistently, per framework. Go might have better tooling in that regard, and maybe that sways the equation a bit. IMO there's still merit to decoupling if there's any feasibility of e.g. migrating to GraphQL or another protocol without having to undergo an entire re-write.
Sorry to spring a mostly-unrelated question on you about this, but why do you call this an integration test? I recently interviewed three candidates in a row that described their tests in this way, and I thought it was odd, and now I see many people in this thread doing it also.
I would call this a functional or behavioral test. For me a key aspect of an integration test is that there's something "real" on at least two "sides" - otherwise what is it testing integration with? Is this some side-effect of a generation growing up with Spring's integration testing framework being used for all black-box testing?
(I will not comment about how often I see people referring to all test doubles as "mocks", as I have largely given up trying to bring clarity here...)
To me, it's a contextual term much like 'single responsibility'. In this case, the two "sides" of an integration test are present. A consumer issues a request and a provider responds accordingly. The tests would ascertain that with variations to the client request, the provider behaves in the expected manner.
At which point you might point out that this sounds like an e2e test, but actually using the client web app, for example, might involve far more than a simple http client/library - in no small part because the provider can easily run a simple consumer in memory and avoid the network entirely. E2e tests tend to be far more fragile, so from the perspective of achieving practical continuous deployment, it's a useful distinction.
integration tests in this instance: varying HTTP requests (infrastructure layer) provoke correct behaviour in application layer.
e2e: intended client issues http requests under the correct conditions, which provokes certain provider behaviour, which client then actually utilises correctly.
This, to me, is why the most important part of testing is understanding the boundaries of the tests. Not worrying about their names.
Just want to address (1) quickly. As you've mentioned at the end of the parenthesized note, the reason I did not use `defer` here is to avoid the lock staying across the rest of the handler. I wanted to confine it to the datastore interaction.
Thinking more about this now and having read the comments, I'm considering to just hide the lock in TaskStore and avoid all these explicit locks/unlocks in handlers; it seems like it will avoid some confusion for folks reading the example (as well as quite a few lines of code!), and my goal here is really the HTTP server logic. I prefer to deflect any attention from TaskStore in this series of posts.
type HTTPError interface {
GetHTTPCode() int
}
func ServeHTTP(w http.ResponseWriter, req \*http.Request) {
result, err := DoTheActualThing()
if err != nil {
statusCode := http.StatusInternalServerError
if httpError, ok := err.(HTTPError); ok {
statusCode = httpError.GetHTTPCode()
}
http.Error(w, ..., statusCode)
return
}
w.Header().Set("content-type", "application/json")
w.WriteHeader(http.StatusOK)
w.Write(result)
}
If that goroutine hasn't finished processing then the channel will be blocked, just like a mutex.
So in your handler you can use a select statement to either write to the channel OR read from the request.Context().Done(). The request context only lives as long as the request. So if the connection drops or times out then the context gets cancelled and a value is pushed onto the done channel and your read is unblocked.
Because you use a select statement then which ever operation unblocks first is what happens. If the write channel unblocks then you get to write your value. If your request context gets cancelled then you can report an error. The request context will always get cancelled eventually, unlike a mutex which will wait forever.
Interesting, thanks. But isn't panicking the end of your program anyway? Could you provide another example where no using defer causes problems?
Also, one thing I'm still waiting for is a decent non-complex di that works using go generate. I'm aware of google's wire (?) But I've looked at it multiple times and I still struggle to understand it.
DI and IoC are for people ignorant of or hostile towards composition.
Further, aspects and reflection are for those unwilling or unable to make reasonable architectural assumptions.
Said another way, meta programming is for personal projects. And maybe for small, disciplined, high trust teams.
Most devs are average (axiomatically) and most projects are CRUD or scraping. So choice of these tools is self soothing to mitigate inferiority complexes. Like Mensa.
This isn't helped by the messy terminology. "Dependency Injection" is literally another term for "composition", but dependency injection frameworks imply automating the composition of one's object graph. However, people who like these frameworks don't seem to be aware that they can more easily compose their object graph using the structures available in most general purpose languages (literals for lists, maps, structs, etc as well as function calls and so on).
Arguably there's some repetition in constructing an object graph (initializing a list of objects that vary only slightly) that one might want to DRY up, but we already know how to do that with helper functions in host languages, and anyway this is just boilerplate--it's almost certainly not where your bugs are, and it's not where your developers are spending their time. A framework introduces a bunch of complexity at the top level (near main()) i.e., the stuff that everyone from developers to testers to operators/sysadmins will probably need to dig into at some point, and all that for no material advantage to anyone.
VRML-97 is the near pinnacle of human achievement, for declaring scene graphs, a special use case of object graphs. It had reuse. It had patch cords (specify non parent-child relations). It only lacked path expressions.
I really I wish I could tell younger me to publish my own VMRL successor, way back when. Young me forfeited when confronted by XML's JavaScript-like metastasis, which had overwhelmed all rationale human endeavors. Then maybe I could have spared humanity the indignity of JSON and kin.
Had I only known that all bouts of irrational exuberance eventually implode...
A DI framework allows you to set conventions to avoid repeating yourself in the initialization phase. (ie if a class requires a parameter 'foo', look for a class named 'FooImpl').
Mocking objects for testing purposes and swapping them for the real objects is also something commonly done that is helpful.
There is no "real" DI for Golang as far as I've seen. The only DI I've seen that effectively do the sort of thing I am describing are for Java.
As for testing, you can already swap out real objects for test objects--that's a property that interfaces provide; I don't see how a DI framework (e.g, the DSL) helps you here.
I will say that when I've had to operate a Java application, trying to do even the smallest bit of debugging had me digging through all sorts of layers of XML nonsense (Spring, various Spring plugins, and who knows what else) to figure out where the logs were being written. Note that my objection isn't "XML" (versus some other format/DSL), but rather the pointless indirection. I'm sure someone proficient in Java would have no problem, but now your sysadmins need to be Java developers in addition to system administrators (maybe not such a big deal if your shop is already a Java shop and you have people you can ask, but for people operating third party software this is a real pain point). And again, all of that tedium for no apparent value.
It is certainly a different mentality from straight out coding everything.
I think it is important and will grow bigger in time, because low code is a type of metaprogramming.
You don't "need" to use low code stuffs or do metaprogramming, but if you know how and learn it well you can get much more complex things done quicker than coding everything.
The stuff you are objecting to is due to Spring and such seeking to be a type of generalized metaprogramming instead of focusing on custom DSLs. A custom DSL will have less verbosity, not more.
In a good low-code setup, all your logs should go to the "log" module, and there should be very simple configuration indicating where the logs go.
If there was a DI for go that used generate, then I would have compile time checking of dependencies. This would satisfy the community's sense of purity while satisfying my sense of annoyance at having to write this same process for every project.
The component graph of your application isn't boring, it's the most important part of the thing, and the starting point for anyone trying to build a mental model of the thing. It should be front and center, never hidden away behind generated code.
The "boring, but necessary pieces" are usually just a call to a constructor. In my opinion it's almost never worth using a DI system that obfuscates dependency resolution and usually can't be checked at compile time just to avoid that.
It is more complex than simply "I just make an interface and make everyone agree on that interfere". Everyone agreeing on the interface to use is very unlikely.
The underlying data in different implementations will be different. This is something Golang fails terribly at because it doesn't have polymorphism.
This is, I believe, why there are so few DI systems for Golang, and most of them are of the sort you are referring to as silly.
The way Golang works, and the reccomended patterns for Golang are somewhat anti-DI.
This is already provided by interfaces, as previously discussed. To be clear, dependency injection makes sense; however, dependency injection frameworks don’t make sense to me.
> The underlying data in different implementations will be different. This is something Golang fails terribly at because it doesn't have polymorphism.
Go definitely has always had polymorphism; that’s the whole point of interfaces.
> The way Golang works, and the reccomended patterns for Golang are somewhat anti-DI.
DI (assemble your object graph in main() instead of distributing it across dozens of constructors a la OOP) is idiomatic Go; DI frameworks are not.
This sort of behavior is core to polymorphism. It can be done in three ways in Golang ( and probably more too... ):
1. Use serialized messages in channels to do all messages to objects ( disgusting imo... ) It would though at least let one emulate the behavior of message passing / routing languages ( pursuant to original visions of smalltalk etc )
2. Use "unsafe pointers" and just do everything the pay you would in C, deliberately going against the way Golang authors want you to do things.
3. Use reflection and messy if/else in combination with code-generation at compile time. ( this is what a bunch of Golang DI systems do :( )
I don't think you understand polymorphism very well.
I don't give a shit what people are calling DI frameworks these days. I also don't much care for things that simply instantiate a bunch of objects and tie them together. That is only a very elementary variety of metaprogramming.
Essentially, what I am claiming is the Golang is a bad language for metaprogramming, and that in other languages the DI systems they have have become a type of metaprogramming that I think is respectable.
> Essentially, what I am claiming is the Golang is a bad language for metaprogramming
That's definitely true, and an explicit choice. Thank goodness!
It can only be accomplished by inefficient functional hackery.
In C, you just make a struct, have a type present in the struct, and then cast the struct pointer to extended object types to gain additional functionality.
In this way you can easily accomplish all sorts of fun things like inheritance. Message passing type designs can easily be accomplished also in C.
In Golang? Well... no. You are essentially forbidden from doing any simple casting or extension. You are essentially stuck with hardcoding the crap out of everything or making your own vcall like system build out of Golang types... which you can't really use in the way you want unless you use reflection.
What I can't understand is why anything thinks that Golang does support polymorphism. They admit it themselves. They are working on it. Only the new alpha test versions have a solution for it. The current released version is not polymorphism no matter how much you want to fucking label it that way.
You can't just go "hey it supports a little bit of what everyone knows as polymorphism". That's like saying alcohol is like orange juice because they are both bitter in some cases.
It's 30 years too late to complain about three unrelated approaches to dynamic dispatch having the same name.
> Go is a bad language for metaprogramming
You’re absolutely right here.
> There is no "either-or" data type in Golang. That's why.
Correct here too, Go doesn’t have sum types. If you want sum types, you have to emulate them via interfaces. But I don’t see how that relates since all of this DI stuff seems to be dynamically typed anyway (errors at runtime) assuming you’re not taking a codegen approach anyway.
> In C, you just make a struct, have a type present in the struct, and then cast the struct pointer to extended object types to gain additional functionality.
I don’t understand what you’re trying to do here. First of all, this only works for the first field (and obviously isn’t memory/type safe).
> In Golang? Well... no. You are essentially forbidden from doing any simple casting or extension. You are essentially stuck with hardcoding the crap out of everything or making your own vcall like system build out of Golang types... which you can't really use in the way you want unless you use reflection.
As a general rule of thumb you can do almost anything in Go that you can do in C if only by delving into the unsafe package; however, “unsafe” is almost never necessary—interfaces typically suffice. You certainly can emulate inheritance if you don’t care about type-safety, just like in C. Unfortunately I can’t say more until you clarify your objective.
> What I can't understand is why anything thinks that Golang does support polymorphism. They admit it themselves. They are working on it. Only the new alpha test versions have a solution for it. The current released version is not polymorphism no matter how much you want to fucking label it that way.
I think you must mean some other word because interfaces are the canonical example of polymorphism and Go has the best interfaces in the business. :) I’ve never heard the Go maintainers claim they lack polymorphism (Go does lack type-safe generics and sum types, but so does C). In an earlier post you argued that interfaces weren’t polymorphism because they don’t let you modify the underlying data, which is patently false—this is the whole point of interfaces. In Go:
var r io.Reader // nil
r = someFile // *os.File
r = stringReader // *strings.ReaderAssuming a web application... You can instantiate the object in your main func and then attach it to the server struct so that it is available in every request. When mocking, you can create a mock server that instantiates mock items that implement the interface. If you need something that is contextual, you attach it to the context. In that case, you can still use mock objects, however, you would have to use a different middleware that handles the mock context objects instead of the normal middleware.
I do not mean JSON, as JSON is not a good generalized meta-programming notation. People use it that way but it ends up ugly and hard to read.
JSON can be used alright as a configuration notation, but does not represent itself well as a metaprogramming language, since it does not preserve order or allow mixed content. You can get order by using an array of course, but it is much more verbose than the equivalent content in XML or HTML.
You might say, by DSL, I mean "custom markup languages".
When I write code in most OOP like languages, I follow the Clean Architecture model. The use case is an actual struct with the interfaces defining the repositories/services as members. I am now free to test the use case in isolation. I can test failure cases to since I return an error, which can be easily mocked. I write a factory to create my use cases. The handlers get the factory passed in as an argument to the constructor for server. I can now test handlers in isolation by passing a factor with mocks.
You define and use an interface to model the DB, and use that as the mock point. Basic stuff.
Everything is still 'bound to DBs' but that's because your program needs a source of data. Faking a second data source other than the DB via a higher-level shared interface is just inviting integration failures.
> Would you have the bulk of your code be integration tests then?
I'm not sure if you mean the bulk of my code or just of my tests - for a REST API I would expect mostly functional tests. Do e.g. PUT+GET and make sure the result makes sense. This would be the case regardless of DB architecture.
Unfortunately it's not this simple (and probably multi-idiomatic).
Go definitely adopts more global state than other languages; I don't know any other language that offers a default-global HTTP client and server. Now, part of that is because Go's stdlib goes out of its way to make these appear stateless even though they are not - and this is good, even if you (often rightly) don't use the default ones.
But I think a lot of people saw those carefully engineered APIs and instead ran with "globals are OK in Go!" Lots of packages have global-level configuration properties - some of this is a hacky replacement for DI e.g. most logger injection. Well, OK, I can support/tolerate some of that because DI in these cases is usually a hacky replacement for real AOP language support. But some of it just shouldn't be global. e.g. Gin debug vs. release vs. test mode should be a setting on the Engine.
And then you get into really bad stuff - I don't know why but it is common to to have a global sql.DB, or sarama.AsyncProducer, or whatnot. A lot of novice Go developers - anecodotally predominately skewed towards previous PHP users, I think because they are not used to have really global variables - use a global for anything concurrency-safe. And this has ended up in a lot of low-quality tutorials/examples/SO questions so I don't see it going away any time soon.
Specifically, I encountered it with logging, frameworks( Gin ), and DBs. It is interesting to me that you called out these three things specifically having encountered them all myself.
I agree also that it is a result of having poor DI support from the language.
I also like your point about PHP users using Golang. The approach Golang uses towards serving web content is very similar to me to how initial PHP frameworks did it.
Go tends to make it easy to setup concurrent things occurring, and so I've found myself spending a fair amount of time speculating on what is thread safe and what isn't.
No matter how much I see it I still do a double take when I see globals in Go modules. Seems like a bad practice to me.
It is, and good packages don't have them.
fx doesn't do that. It is primarily just a convenience way for tying things together in the single intended way. You can provide some configuration but it is not via a DSL; it is by making calls to the fx library.
Realistically to do the sort of DI I am referring to with Golang there would need to be a preprocessing step during compilation that generates Golang code.
The closest thing that comes to mind is writing software with some sort of global configuration. i.e.: the user provides some options (CLI flags, environment variables, configuration files, etc.) to change the runtime behavior of the software (e.g. use filesystem storage vs S3 storage, etc.).
In Go, I've always solved this pretty straightforwardly by combining Fx and functional options [1].
[1] https://dave.cheney.net/2014/10/17/functional-options-for-fr...
err = json.NewEncoder(w).Encode(&task)
How are folk managing Go endpoint as apps scale across number of endpoints? I'm skeptical that one struct in one folder leads to good app structure, and have seen this start to break down in a few cases in practice
If you have independent, different elements in that API, you break them out into separate "servers" but still register the endpoints on the same HTTP handler.
I know that people don't like external libraries too much, but I'd like to plug my own here. You declare your API in OpenAPI 3.0 (aka, Swagger) and it generates your server and models for you, so all you need to do is write the business logic. (https://github.com/deepmap/oapi-codegen)