Python Requests: HTTP for Humans | Hacker News Reader

Python Requests: HTTP for Humans

python-requests.org

397 pointsfrzn85 comments

Requests is an ISC Licensed HTTP library, written in Python, for human beings.

Requests allow you to send HEAD, GET, POST, PUT, PATCH, and DELETE HTTP requests. You can add headers, form data, multipart files, and parameters with simple Python dictionaries, and access the response data in the same way. It’s powered by urllib2, but it does all the hard work and crazy hacks for you.

Open articleView on HN

Absolutely my favourite recent development in the Python world. I won't be mistaken if I say that many of us have built half-arse versions of this on top of urllib(2) multiple times in the past. :-)

The support for keep-alive is particularly welcome, huge kudos to the author.

Question to users of other languages: do equivalents exist for PHP, Perl, C# or Java? Ruby folks rave about restclient, but is there anything else?

For Perl, I've mentioned this elsewhere here but Mojo::UserAgent looks quite neat. This time with Mojolicious url - http://mojolicio.us/perldoc/Mojo/UserAgent

Interesting things can happen with M::UA in a single line:

  use Mojo::UserAgent;
  print Mojo::UserAgent->new->get('http://freegeoip.net/json/8.8.8.8')->res->json->{country_name};

Very powerful.

Indeed. And there are many more nice examples in the cookbook: http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#USER...

Also check out the command line interface which allows you todo things like this:

  $ mojo get http://mojolicio.us 'head > title'

ref: http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Comm...

For Ruby, there's also Faraday https://github.com/technoweenie/faraday.

Which is used on the Twitter gem for ruby: https://github.com/jnunemaker/twitter/blob/master/twitter.ge...

For Ruby there's httparty, https://github.com/jnunemaker/httparty, curb, and a couple of others that escape me right now.

Here's the new one I'm working on:

https://github.com/tarcieri/http

httparty does a few of the same tricks, but I still find its API a bit obtrusive. At least it automatically parses JSON! (my library does this too, if an appropriate library is loaded)

In addition to a humane interface, it will soon be backed by a Ragel-generated HTTP parser:

https://github.com/tarcieri/http/tree/master/parser

Looks very promising. I see it currently uses Net::HTTP under the hood - do you plan to keep it that way, like Httparty and Faraday, or are you going to do away with Net::HTTP altogether? (The comment saying "this is temporary" next to requiring 'net/https' is encouraging... :))

I plan on eliminating Net::HTTP entirely and writing my own client using Carl Lerche's parser for his Picard library which is written in Ragel:

https://github.com/tarcieri/http/tree/master/parser

Did you get to re-use any of Zed Shaw's excellent parser work from Mongrel?

Zed had two parsers: a server parser as part of Mongrel and a client parser as part of RFuzz. I actually used the client parser from RFuzz in my Rev/Cool.io libraries. I talked to Zed about his plans for these, he wanted to do a unified parser that can handle both HTTP requests and responses (since the only difference between the two is the first line) but he never got around to it. Both projects are now unmaintained.

Carl is working on a comprehensive event-driven HTTP (and Websockets) framework for Clojure. I'm just borrowing his parser and re-wrapping it in Ruby. I prefer to use Carl's as he's (at least for now) actively maintaining it, as opposed to using Zed's abandonware.

Also Typhoeus and RestClient.

Probably others.

In the FAQ it says that support for keep-alive is on the way.

I do have a half-baked implementation on urllib2 that I hate. This is really cool.

Keep-alive has first-class support in the soon-to-be-released version!

Is it just me or does the FAQ break the back button?

I always use libcurl for PHP... don't really have any complaints.

C# / .NET had System.Net.WebRequest since 1.1. It's similar in principle to Requests.

No, no it's not. (Http)WebRequest IS AWFUL. I built RestSharp because HWR is so bad. WebClient is closer to requests, but still horribly awful if you know anything about HTTP.

The new System.Net.Http.HttpClient in .NET 4.5 is closer to Requests and not entirely bad, if a little rigid.

Just out of curiosity, what's so awful about WebRequest? I've been using it for years for various things and never really had a problem with it.

  - It's too low level for quick stuff
  - It handles HTTP basic auth wrong
  - It throws an exception, AN EXCEPTION, for any response that isn't 2xx
  - Async is an ugly mess
  - In love with streams
  - Horribly inconsistent across framework profiles
  - ServicePointManager.Expect100Continue
  - Inconsistent header handling and restrictions
  - Terrible cookie API

I think it's fine for simple scenarios (i.e. plain GET), but it breaks down in a hurry and the fact that you have to read out of the response stream is real overhead.

> I think it's fine for simple scenarios (i.e. plain GET)

So's python's urllib2, the point of `request` is that as soon as you go even slightly beyond very basic GET and POST, it breaks down quickly in a horrible mess.

It's not fine for a plain GET, it's awful.

I'm still surprised every time I use it that they've not released a modern library for it in .Net.

The async handling was the deal breaker for me. I gritted my teeth through the other stuff but finally gave up at that point.

I'm not your parent poster but I personally dislike that you have to instantiate a separated Response object, and that you have to fetch a stream, even for tiny responses. In those Python/Ruby frameworks it's just one line. I always end up writing a wrapper just to change it.

And this one is a pet peeve, but I hate that verbose stateful OO idiom where you instantiate an object (with a factory) and then use setters to change properties, instead of setting it all in a single command. I don't know if other .NET HTTP libraries change that, though...

timelines suggest that requests was inspired by https://github.com/mikeal/request from the land of node

I can assure you that it wasn't :)

Just about every library & language has a Request-like wrapper. It's nothing unique to Node, and certainly nothing unique to client-side JavaScript where native XHR is a nightmare.

Discussion from the two month old submission: http://news.ycombinator.com/item?id=2882301

This looks so nice. Is there any upcoming support for HTTP caching? I know it is not the easiest thing to do because you will probably need to make several cache backends (file, db, memcached, redis, etc.) to keep everyone happy.

Also I couldn't tell if there is keep-alive support, is there?

Keep-alive is 100% supported in the upcoming v0.7.0 release, along with async i/o :)

Caching: I think that will be best be served by a hook/helper module, but we'll see.

Cool! Is there anything I can look at to see how async I/O will work? Does all the async support depend on gevent monkey patching (as requests/async.py would suggest)?

It's unreleased, and therefore completely undocumented at the moment. But here's the API: https://github.com/kennethreitz/requests/issues/98

Mmm async.

Does that also mean that it will have better support for timeouts? I'm not interested in socket timeouts, more in a 'finish this in N seconds or just give up' thing.

For that reason I am using pycurl now for some stuff where I had previously used the requests library. Which has a really really horrible API :-/

The most interesting thing about caching is that you also get support for 304/Etags/Last-Modified - which is amazingly useful for any application that might hit the same URL more than once. I'd vote for that logic in requests core, with caching as a key/value interface (ship an in-memory or file-based default, but make it easy to hook in memcached/redis/etc ala Django).

What is the timeline for v0.7? Desperately need the asynchronous stuff!

As soon as I can push it out. Aiming for Friday (if not sooner)!

That is great news! I really appreciate your work, thanks!

I don't mean to badmouth Requests or Python in general, it certainly looks like an improvement, but can anyone enlighten me as to why an API that wraps an HTTP request with concise methods is #1 on HN, twice? This looks more like a good answer to SO question than "news".

This looks like all the other wrapper methods you spend 5 minutes doing, once, when you start a new codebase?

It's #1 because people vote it up; people vote it up because it interests them or they'd like to see more of it. That's the pattern for every link here.

We may agree or disagree with the consensus of the masses, but the masses make the front page.

The philosophy of Python is that those wrapper methods shouldn't be necessary.

I'm glad when language quirks and libraries get linked, it's good for discussion. (And more informative than a good third of the stuff that gets linked.)

I came here with the same question. Considering I come from PHP & JavaScript where the native APIs are awful, wrapping is a necessary step for sanity. Interesting that Python didn't have this until now.

Python has always had it to some degree, but merging these sorts of things into the core language is what major releases are for. No language is perfect. I think the reason you see this upvoted so much is there are so many Python programmers who want to see this on the roadmap for future versions of Python (and it probably will be.)

I couldn't agree more.

I gave a talk about this very thing at PyCodeConf this week. Here are the slides: http://python-for-humans.heroku.com

Maybe because doing the same wrappers over and over again, as short as they might be, is boring?

The design of this library reminds me of Apple's design... While a lot of Python is more like Google, or Microsoft - they just don't get it. (Admittedly, parts of Python are very well-designed, but I still see a lot of room for improvement...)

I just started working on a similar library for Ruby:

https://github.com/tarcieri/http

Looks like Python got something similar to Perl's LWP https://metacpan.org/module/LWP

I think LWP is more equivalent to Python's urllib2 (ie. both big beasts!). So this i believe is closer to what LWP::UserAgent (https://metacpan.org/module/LWP::UserAgent) does.

Also check out Mojo::UserAgent (https://metacpan.org/module/Mojo::UserAgent) for a nice(er) alternative. And don't forget that Perl now comes a nice lightweight alternative as standard: https://metacpan.org/module/HTTP::Tiny

Yes gp, LWP::UserAgent specifically. Never knew about HTTP::Tiny thanks.. I'll check it out.

Link to previous discussion on HN:

http://news.ycombinator.com/item?id=2882301

Unfortunately, no Python 3 support

I can assure you that it's directly around the corner :)

Why is that unfortunate? Why would anyone run python > 2.7?

Do you have a good reason why they shouldn't?

Apparently 113,798 people found a reason to download the 3.2.2 Windows installers (32 and 64 bit) from python.org through only the first 10 days of the month of October. http://www.python.org/webstats/usage_201110.html

(doesn't include mention of the 3.2 source tarball downloads, which are under 5,000, most likely since people get that from their distro anyway)

why not?

Finally, an easy way to make POST requests from Python

Forgive me if I've missed this, but does this library have support for multipart/form-data POSTing? That's a capability in the python library that's sorely missing - especially in 2.X.

yes, it does

I'm using it in a project I'm working on. It's really easy to use!

Although Requests looks very nice, I've wondered why most people wouldn't simply use pycurl - http://pycurl.sourceforge.net/ ? It's stable and you get all the power that cURL has (which is a lot), and the API is fairly simple - although maybe not as pythonic as the Requests API is.

On a side note: last time I checked urllib2 (which actually drives Requests) could not do SSL requests over a proxy, which gave me the feeling that this implementation still isn't very mature.

I'm moving off of urllib2 as we speak, to sit directly on top of httplib.

One thing at a time :)

I love Requests, but it would be really nice to have some Python libraries with cert-verified SSL support. I've hacked SSL support on top of urllib2 for my projects at work (https://github.com/josephturnerjr/urllib2.VerifiedHTTPS), but it'd be really nice if libraries started prioritizing secure communications as a first-order requirement.

This will be in requests very soon. :)

Then I will be porting my projects over to Requests very soon :)

It's pretty easy to implement, as it already works at the lower (socket) level with the ssl module, it just hasn't made it up the stack into urllib2 (or Requests..yet!).

I note that you call gevent.monkey.patch_all() in async.

I'm still fairly new to gevent and I'm curious if others consider this good style for a library?

If you're using Twisted (or Tornado, ugh) then there are builtin HTTP clients which are async in all the right ways and provide similar high-level APIs for making requests. If you're using gevent, then this is what you want. If you're using eventlet, you should switch to gevent or learn Twisted.

So yeah, this is suboptimal, but not the worst thing in the world. It should really be documented though.

Trying to use the requests module yields:

  File "/usr/lib/python2.6/cookielib.py", line 38, in <module>
    from calendar import timegm

ImportError: cannot import name timegm

for me.

On two different machines, one OSX SL with Python 2.6 from MacPorts and the other Ubuntu 10.04.3 LTS with Python 2.6 via aptitude.

Can not find any solution for this via Google and its siblings.

I'm stuck :(

-jsl

I love this API, and it looks like the dev version is getting even better. r.iter_content is exactly what I needed! :D

Excellent library but they need to fix this piece of misinformation: "A project that is released as GPL cannot be used in any commercial product without the product itself also being offered as open source."

This is false. A project released as GPL can also be released on another (commercial) license.

Yes, it has to also be offered as open source.

If Python is going to include urllib or urllib2 in the standard distro, why not make it good? Http is foundational to (web) programming. I am shocked at how poor the standard httpclient implementations are in most environments.

Oh, what I would give for a "mechanize for humans" built on this! I love mechanize, but I dream of not having to go back to its one page documentation and the source code every time I want to do something.

There is also an addon for Requests that lets you make OAuth requests: https://github.com/maraujop/requests-oauth-hook

How does this compare (feature wise) to the poster library? Are they about the same? I've used poster some, but not requests... I'll have to try it now. Just curious how it compares to poster.

The current stable release actually has poster inside (and has for a while), but it's being taken out soon.

`requests` used `poster` at least in the past (if I recall correctly)

Given that there are a great deal of service APIs in the wild this is going to lower the barrier of entry for many less experienced developers looking to prototype their ideas.

It would be nice with something similar for Twisted.

What's with the "for human beings" phrase? Is that a way of saying the software is easy?

Some APIs are appropriate for programmers to target. Others are more appropriate for compilers or higher-abstraction libraries to target.

"Requests is an ISC Licensed HTTP library, written in Python, for human beings.

Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks."

It would appear so. Based off of testimonials and briefly looking at the API I'd say this is primarily aimed at a simple API + fewer "gotchas"

E.g., urllib/urllib2 both support urlopen(url, [data]) if `data` is supplied the request is executed as a POST, if not it is executed as a GET.

Issuing a GET w/ a querystring requires mutating the `url` and omitting `data`. This surprised me at first, and probably a number of others

It generally means it has an API which lends itself to writing readable, intuitive code.

Reply on news.ycombinator.com