A Go “clone” of the great and famous Requests library
github.com
github.com
Go prefers that you be more explicit and use more function declarations, even if it means repetition in the library code. When developers try to get around this, you see awkward constructs like the pointer-to-a-config-struct used here.
You get a channel - you can read from it until it's closed which is always safe. Go-routines are cheap so you don't care about the background details. For simplifying HTTP, this is great.
I got a lot of feedback that I shouldn't do that, so I moved the asynchronous requests to specific functions e.g GetAsync and made the standard API synchronous.
Idiomatic Java, Perl and other languages were shit for a long time, they only improved when some brave soulds fed up with it and explorer non-idiomatic solutions (like the Play framework). And some things I see in "idiomatic" Go make my hair crawl...
Idiomatic code is so called, because people familiar with the language can quickly read it. It makes reading code a much lighter task.
Frameworks are a different thing. Frameworks build around its host language, but normally are meant to use in a specific way. In its own idiomatic way.
In the case of Java circa 2000-2006 it needlessly messed up code, making it more difficult and slow to read EVEN for people familiar with the language.
Following conventions doesn't make code better to read automatically just because people are already familiar with them -- those conventions should be good in the first place for that to happen.
Node has the callback hell "idiom" many are familiar with. And yet if somebody rewrote the code with async, await, even the people "familiar" with callbacks would be able to read it faster and understand it better.
If the porting language isn't as expressive as the original source -- oh well thats a price i'm willing to pay.
I figured that I would write the code and decide on a name later...
What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do.
Because there's no equivalent of npm in Golang, culturally speaking.
IMHO, dependencies should be pulled in for things that would take you weeks to do correctly. Avoid pulling in a dependency for something you can do yourself in 5 minutes. So I would pull in an http library if it actually did the mechanics of http better than the one in the stdlib, but not one that just wrapped it up for me a bit.
For me, that's not a Go thing, that's just an I'm Older Now thing.
Also, snarkily, dependency management is a mess in Go right now; none of us have any dependencies because we can't figure out how to do it. :P That is definitely a hole right now (though it's starting to get plugged in 1.5 with the /vendor directory) and is definitely exacerbated by the core team not being able to use any solution they could provide for the community.
I wrote this library because I didn't want to rewrite those functions every time I started on a new project. I put it online because I figured that I would be useful to others as well.
Exactly how famous is this library?
It is the de-facto standard for anything that has to interact with a API, or just HTTP in general. The old way to do these sort of things was the urllib2 module (part of the Python standard library). urllib2 has some design flaws and can be a pain to work with.
As a proxy, check out the download counts on this tracker using data from the PyPI, where the vast majority of Python packages are installed from:
https://python3wos.appspot.com/
Looking at the top few packages: * simplejson is the upstream for the json module in stdlib which people install since it gets performance updates first * requests * six: the most common library used to bridge the Python 2/3 transition * virtualenv: near-ubiquitous development tool (it allows you to maintain a separate Python environment for each project to avoid cross-talk, and is also used by popular testing tools like tox which run your tests under a variety of Python versions) * distribute: a few years back, the stdlib setuptools module was forked for a major overhaul. That's since been merged back in but many packages still reference it, particularly in older releases * boto: AWS client library, used by the official awscli tool * pip: Python package installer, now bundled with Python but updates & older versions of Python use the PyPI version
It handles all of the low level stuff for you, such as sessions, cookies, gzip, form encoding POSTs, composing URLs with arguments, thread safety etc. etc. without you even having to be aware of them, all in a terse and more readable format. It does it without taking the ability away to control those if you really want to, not that you're likely to need to anyway.
Take a look at the two examples here, one using the standard library for http, urllib2, and the other using requests: https://gist.github.com/kennethreitz/973705
I want requests to exist in pretty much every language. The API is clean and simple and perfectly expressive.
You'll see quite a few Python libraries that aspire to emulate this quality of API design, often adopting Requests' tagline: "XXXX for humans"
Using http://golang.org/pkg/reflect/#Select to build what you need;
Or building something specific to your use case without the above, which seems likely to be... goofy and unmanageable. Creating objects that create new channels to do `select`s over some fixed number of cases and composing those.
Do you have any experience using `reflect.Select`? While not exactly ergonomic, it seemed a... pretty okay tradeoff, inconvenience for power, and at the very least more power for approximately the same amount of pain as the rest of the `reflect` package.
You can do it, but since the only reason to do it is to avoid spawning goroutines, I'm not sure why you'd bother.
I consider this a Go anti-pattern. Without a good reason, do not try to provide "asynchronousness" in a library. Go natively supports goroutines and channels, and it's considered baseline skill in the language to be able to fire something off in a goroutine (it's literally a two-character keyword) and receive something on a channel, if you want to. It would be not only adequate but preferable to implement this library as a fully synchronous request system and expect the user of the library to implement what asynchronousness they may require. Your hardwiring of exactly how the "asynchronousness" works may conflict with my own needs.
All the "asynchronousness" you need to provide is already hard-wired into the Go runtime itself; what certain language communities have trained you to think is "synchronous" code already isn't. You don't have to super-duper-extra make it even more asynchronous.
I won't quite call exposing channels in a library API a code smell, it's a little too useful for that, but it's still something where you ought to pause for a moment and really think about what it means, especially if you created it rather than receiving it.
Oh, and let me be clear: Levigross, I'm seriously suggesting that you change this library wholesale to be fully synchronous, and the fact that this will be an API change is a feature, not a bug. You'll find it also simplifies the API significantly, also a feature.
Thank you for the suggestion. I plan on adding some functionality to make asynchronous APIs friendlier to use (like a `Do` or `Apply` function etc...). But if you don't choose to use the asynchronous APIs everything still functions the same (in fact the asynchronous APIs use the synchronous functions and slap channels on them)
Oh, what the heck, I'll name names. This is part of why I've been so pissy at the Node community for the past few years. Their definition of synchronous is wrong, and it's really become quite popular. Synchronous does not mean "blocks the whole process". That is a weakness of Node, not a universal programming truth. From the Node perspective, all Go code is always and automatically asynchronous. You don't have to add it. It already is. You're not adding functionality by trying to "nicely" provide an asynchronous API, you're simple redundantly adding what is already there, and what you're adding is significantly less flexible than what Go already provides.
(Now with more correct definitions of the terms, it is meaningful to say that Go code is synchronous within a goroutine. But Go already provides abundant tools for dealing with that within its runtime. We can already compose these tools together trivially to do whatever async patterns we want.)
Now, I suppose, standard disclaimer, this is your library, do as you like. I'm not actually personally passionate about this like I'm yelling at my monitor or anything. I'm trying to help you save time and effort. It's up to you what you decide to do.
1. Thank you for the feedback
As you have already seen, I submitted my library to /r/golang and have gotten a lot of feedback – from which I modified the constructs (originally the functions returned channels).
I want to write something that is useful to as many people as possible (all while not alienating anyone) and therefore try not to force users (like I originally did) to use the "asynchronous APIs".
I didn't expect this to end up on HN and was going to start a discussion on golang-nuts on the pros and cons of this construct. Based on that, I was going to remove or keep the APIs.
wat?
† At least in 1.1, I couldn't write a fast port-scanner without doing my own scheduling, but that's a corner case.
Kind of a meta question, but an interesting one.
Part of the problem is that even as an attempt at a "non-synchronous" interface, this seems clumsy. It demand-creates new channels for requests. That doesn't seem right.
It might make sense for a library to return a new channel for something you do once, or once per goroutine.
It doesn't make sense to use returned channels as a basis for async dispatch.
No, I'm not trolling. #8453 got closed, and immediately, #11081 was opened since it re-broke this.
Original issue reported by myself: https://github.com/golang/go/issues/8124