Requests: HTTP for Humans | Hacker News Reader

Requests: HTTP for Humans

docs.python-requests.org

404 pointsDanielRibeiro66 comments

I don't know much Python, but I'd likely be correct in assuming there are quite a few proven http client libraries in existence.

This looks like another reason to believe that if you build a better solution, even when the space is well-covered, that you can still win adoption, so maybe, a lot of people can stop worrying about being second.

> so maybe, a lot of people can stop worrying about being second.

Indeed.

Being second can often be beneficial. You get to learn from the mistakes of the first without being locked into those mistakes.

Awesome stuff. How many hours I've wasted futzing around with urllib2 and creating my own Request handlers and all that other nonsense. This lends itself nicely to working with 3rd party APIs (Facebook, Twitter) in particular.

I used httplib2 (http://code.google.com/p/httplib2/) for the REST module in Bulbflow -- httplib2 is by Joe Gregorio of Google, and it's not bad.

httplib2 has a very nice API, but be warned that it's not thread safe.

I've noticed that the layout of the site is uncannily similar to flask's documentation (http://flask.pocoo.org/docs/), down to the "Fork me on Github" in the top right corner. Is there some sort of software producing this kind of documentation?

I don't have much more info than this, but it looks like they're both generated with Sphinx (http://sphinx.pocoo.org/), going by the footer from flask, and the common source (both have `<div class="sphinxsidebar">`).

It's pulling in a copy of the css file the flask site uses.

The "Fork me on Github" is pretty common on a lot of sites (Github made those graphics), but that aside I agree -- I was thinking it was another Pocoo project until I saw the author.

Looks like he just co-opted the theme (Pocoo might have it available somewhere, but I couldn't find it). Nothing wrong with that, it's a great theme.

My modification of the Flask theme is available here:

    https://github.com/kennethreitz/kr-sphinx-themes

The original is here:

    https://github.com/mitsuhiko/flask-sphinx-themes

Yes. They're both created using Sphinx: http://sphinx.pocoo.org/

Requests is hosted on a site called Read the Docs, which does documentation hosting for Sphinx projects: http://readthedocs.org/

Sphinx has become the defacto documentation tool for Python projects, and a lot of other projets as well. Notably a number of PHP projects, and Varnish the web cache use it also.

It should also be noted that Sphinx was developed by Pocoo, the same group that develops Flask, Jinja, Werkzeug.

http://pocoo.org

And also Pygments (probably the most popular syntax highlighting libraries) and they run http://paste.pocoo.org/ , an excellent pastebin.

http://www.pocoo.org/projects/#project-hub shows all of their projects. Pocoo significantly improves the lives of all Python developers. Thanks pocoo team!

For those interested in getting to know requests vs urllib2, I gave a talk @ PyOhio, covered some simple examples in each (and why requests is super great) http://python.mirocommunity.org/video/4388/pyohio-2011-consu...

Another useful Python library by Kenneth Reitz is tablib, "Python Module for Tabular Datasets in XLS, CSV, JSON, YAML, &c. ". See https://github.com/kennethreitz/tablib

That looks extremely useful. Thanks! Project webpage: http://tablib.org

Using this with DictShield (https://github.com/j2labs/dictshield) or Colander (https://docs.pylonsproject.org/projects/colander/dev/) would be very cool methinks

I'm almost frustrated to see this. I just learned to use urllib2, and found that it took a great deal of effort to get any reasonable behavior. I wish I had known about this before. It really looks great and I will try to use it in any future work.

You could be a contributor then ;)

A coworker introduced me to Requests a couple of weeks ago. It makes life a whole lot easier.

It does highlight my biggest gripe about Python, though. I love the language, but the class library is hit-and-miss, and anything to do with HTTP is a pretty big miss. httplib, urllib, urllib2, cookielib -- yuck. I like that Requests has a nice API, but it's still built on top of pretty bad class libs underneath. (And the Requests API does expose some warts, like having to use cookielib's CookieJars.)

I'd really love to see someone's take on a clean break from the standard modules with a simple, maleable API.

You are describing restkit. Check it out. Goes past the broken standard lib stuff and straight to socket. Very pythonic.

looks interesting! thanks for the pointer.

(Here's the URL for the project: http://benoitc.github.com/restkit/)

If you're looking for a "clean break", Google's httplib2 [1] might be what you want. It's somewhat lower-level than Requests, but adds a bunch of advanced stuff like caching support and keep-alives. As you might guess from the name, httplib2 is more like httplib, while Requests appears to be more like urllib2.

1: http://code.google.com/p/httplib2/

It should be noted that httpbib2 is not a Google project. A lot of people prefer it though.

httplib2 is part of why you should use requests: it's far more respectable as a client but not as well documented and it still takes way too much code for basic operations. I appreciate what httplib2 is trying to do, that there's a ton of hard low-level annoyances in building a modern HTTP client, but really, just use requests instead. Kenneth Reitz is very motivated and he gets the degree to which simple things should be simple whereas httplib2 feels more like an academic exercise than something people should use to build production systems[1].

Disclosure: I'm listed in the requests AUTHORS file but can claim credit for, oh, about 0.0001% of the awesomeness.

1. http://code.google.com/p/httplib2/issues/detail?id=96 is a good example: an annoying bug which affect many people, there was a fix available for months, which worked great when I applied it in a fork and pounded a couple TB of data through it, but it took over a year to make it into trunk and even longer to make it onto PyPI where any other project which required "httplib2" would get the working version.

Have been using this for a little while now. Totally awesome.

The simple example on the home page should have one more line of code that shows that you use r.content to get to the content that came back from the request. That would seem to be the most common use case.

So that's why I couldn't get it to work—have a look at this[1].

Thanks for teaching me what the documentation failed to—it's neither in the front page example, or in the API documentation for Response.

[1] https://github.com/kennethreitz/requests/issues/105

Fixing now :)

This library seems like a good idea. I hope it is or will be based on Joe Gregorio's httplib2, so it gets close to correct.

Also, I can't believe it includes PATCH support. Surprising, but good, to see that out in the wild.

Why not just include arbitrary verbs? The rest of the protocol doesn't change at all...

There is support for arbitrary verbs, just more direct API support for the common ones.

here's the api for a generic request, get is right under it: https://github.com/kennethreitz/requests/blob/develop/reques...

This looks neat. I have been using restkit[1] for a little while now, but I am certainly going to check this out.

[1]: http://benoitc.github.com/restkit/

+1 for restkit. It has a Pythonic interface, shortcuts and completely replaces urllib, urllib2 and httplib all of which I've had some beef with before. Love restkit!

Fascinating, I am really glad that "we" the new wave of programmers are not only conserned about code quality but are also deeply concerned with "usability". Everyday, as I go back and forth using old libs vs new libs, its obvious that the quality of old libs is pretty damn good but they have crappy "usability". New libs have a bit more bugs, being that they are also "newer", but they are a pleasure to use.

I wish someone would write an equivalent of this for Python's myriad of horrible process management functions. popen, popen2, popen3 etc were a nightmare, and the newer subprocess module still doesn't have an API that I'd call memorable or obvious.

I am planning on this actually! HTTP lends itself well to running a process.

It will likely be very call+result oriented at first (no interaction w/ a running process)

For such a new module, the API for subprocess is remarkably bad. I have to look it up absolutely every time.

Check out psutil (http://code.google.com/p/psutil/). I discovered it the other day when I needed to get a list of open ports used by an ssh client running in a subprocess, and it was as easy as this:

    sshc = psutil.Process(pid)
    sshc.get_connections()

It allows you to upconvert process ids, and it has its own wrapper of Popen that returns Process objects. It also seems to have well thought out windows compatibility, if that matters to you.

:D <- My face when I found it.

Wow I was just about to start going through the pain of using urllib2 for the millionth time right before I read this.

equivalent in node: http://github.com/mikeal/request equivalent in jquery: https://github.com/iriscouch/request_jquery

How do those handle SSL and user authentication? I skimmed your links but didn't see anything.

Python's urllib2 isn't that bad for the simple stuff shown in your links, but it's a mess when you have to try to log in somewhere. Seeing in the example that logging in is as straightforward as

    requests.get('https://api.github.com', auth=('user', 'pass'))

just blew me away.

Yeah the authentication parts of urllib2 remind me of the horrible 20-letter classes glued together in Java, just the thing I wanted to escape from by using Python. Seeing that I'm trying to make this script interface with a website that uses both cookies and http auth this library seems like a godsend.

Quick question on the code. I'm looking at this commit:

https://github.com/kennethreitz/requests/commit/773630b0109c...

and wondering what "from .status_codes import codes" does?

Specifically the fact that it's not "status_codes" and has a dot prepended. Is that something special in Python (guessing no) or some convention that is used? Just curious.

Just needed to look at the docs: http://docs.python.org/reference/simple_stmts.html#the-impor....

The relevant bit:

"When specifying what module to import you do not have to specify the absolute name of the module. When a module or package is contained within another package it is possible to make a relative import within the same top package without having to mention the package name. By using leading dots in the specified module or package after from you can specify how high to traverse up the current package hierarchy without specifying exact names. One leading dot means the current package where the module making the import exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute from . import mod from a module in the pkg package then you will end up importing pkg.mod. If you execute from ..subpkg2 import mod from within pkg.subpkg1 you will import pkg.subpkg2.mod. The specification for relative imports is contained within PEP 328."

Cool stuff.

Really cool stuff indeed. Thank you.

It's a relative import -- where "relative" here is like relative vs. absolute paths. See:

http://docs.python.org/tutorial/modules.html

This is a great library. I've used it in a couple of projects now and it is great.

This is the kind of stuff that I would love to see in the Python standard library.

Quickly judging by the look, I thought "hey, Armin Ronacher just released another library!", but this one seems as well thought out as if it was designed by Armin himself.

Does any one know of a good Ruby equivalent of such a library?

https://github.com/archiloque/rest-client

The example written in urllib2 instantly reminds me of Java and the classy query joke [http://ejohn.org/blog/classy-query/]. Just a thought: are most Java-related APIs over-engineered, and does Java have a bad influence on many API design work?

This seems to be a fantastic project. I wish more libraries were designed this way. Library's usability is just as important as performance or feature completeness.

This was much needed.

By the way, Common Lisp has Drakma for this: http://weitz.de/drakma/

From the title I thought this would be some kind of mechanical-Turk webserver.

Does anyone know of such an API in Java - HTTP for Humans in Java?

Restlet is a good attempt at writing an api that matches the http spec well. You can find docs for a client at http://wiki.restlet.org/docs_2.1/13-restlet/21-restlet/318-r.... It's still more verbose than it might be, would be a good starting point for writing a decent api though.

Restlet is a server framework.

I think the parent poster is looking for a client API that is nicer to use than java.net.Url and/or the various versions of HTTPClient

From the restlet home page "Do you want to blend your web services, web sites and web clients into unified web applications exposing and consuming RESTful web APIs?"

The link I provided was to an example of using restlet on the client side. I would agree that the api is not as human friendly as it could be but it does describe interactions in a restful which is why I suggested it would be a good starting point for writing a friendly api.

Jetty's HTTP client is pretty decent.

http://wiki.eclipse.org/Jetty/Tutorial/HttpClient

how does this compare with webob (http://docs.webob.org/en/latest/index.html)?

It's completely different.

It doesn't. Webob is a server framework, requests is a client.

People will rightly tell you that it does not. But I must admit that this did get me to thinking about a few small changes to WebOb along with the integration of WSGIProxy, at which point there would be some real similarities.

Not to start a language pissing war...but that syntax looks awfully like Ruby.

That means it's pretty elegant and nice :)

Reply on news.ycombinator.com