Requests, Python HTTP for Humans, reached v1.0 | Hacker News Reader

Requests, Python HTTP for Humans, reached v1.0

kennethreitz.org

386 pointsjorde86 comments

Congrats, Kenneth - Requests is a very nice library.

I've been using it a lot recently (though I've never had to go too deep into it). I'm impressed with the simplification and removal of code in this release. If only that was a goal for every project :)

A nice thing about requests is that you almost never have to go deep into it. Out of the box, it almost always just does what you want. That's harder than it sounds, as evidenced by the failure of the standard library's urllib and urllib2 modules.

The choice of Apache 2.0 for the license is interesting. It makes the library incompatible with GPLv2 software[1], but compatible with being integrated into the Python standard library[2].

[1] http://www.gnu.org/licenses/license-list.html#apache2

[2] http://wiki.python.org/moin/PythonSoftwareFoundationLicenseF...

I'd hardly characterize the ISC license as "esoteric". It's fairly compact, yet plain in its language. Apache 2.0 takes a lawyer's help in trying to understand.

Great work. It's a shame these nice libs weren't built a few years ago, they might have had a chance to become the default in python3. Would be a killer feature to push more people to it.

    "The entire codebase has been rearchitected"

I hope there are a few tests to check for regressions? ;)

This is the part that concerns me. Generally, I think of code tightening in on a release, which major swings early in the processes and minor tweaks later. While I all for everything that happened in this change, I wonder if a 0.2 (or even 0.9) would have been appropriate first. Version x.0 in open source software implies "this is ready for production" (as opposed to commercial software, where it seems to mean "let the beta test really commence).

(I just noticed it is at 1.0.2 now. I think I'm going to have to give this a couple weeks before I upgrade any of my projects.)

Look out for the change to the Request constructor that makes `url` no longer the first parameter. Rather poor backward compatibility there.

It's a major revision number change, that's the only time to do such a major api change.

While your point is valid, I didn't expect this to happen with requests. Just two weeks ago, I was telling my colleague that requests is such a nice library, we have nothing to worry about when installing a new version. Simply grab the latest!

Then 1.0 happens. I have now moved requests to the "thoroughly test before upgrade" pile.

This is what major version numbers are for.

It's an interesting challenge in the open source world to deal with. In closed source software, having x.0 break things us understandable. We generally never see all the effort between x.0 and (x+1).0.

In open source, it's harder because all of that is open, which means there is a conflict between x.0 means "release" and x.0 means "development is just beginning. It probably should mean the latter.

But then having a big "It's 1.0 and fabulous!" blog post is the wrong thing to do. It probably should have been "It's 1.0; be careful!" I don't see how you can have it both ways. Linus took care of that with odd/even versioning, but I think you need a huge project for people to pay enough attention to understand unless it became a de facto standard (which it hasn't in over a decade, so it's unlikely to).

(BTW, I am very grateful for the work you've put into this. Requests is truly a model API. My comments are much more meta about versioning and open source in general (using this project as an example) and should in no way be construed as saying I think you are doing something wrong.)

Version x.0 in open source software implies "this is ready for production"

I've never really had this impression before. Do you think people would have been holding off on using a library like this because it was at 0.14 before? I'm curious where this impression comes from.

No, but I think people absolutely expect version 1.0 to be more stable than 0.14. The impression started with the linux kernel many years ago and has included many other pieces of software since. Versions either seem to mean "something changed" or "some set of features is ready". When you make a jump from 0.13 to 0.14, it is safe to assume the former. 0.14 to 1.0 tends to imply the latter.

There's this interesting narrative that develops from time to time in Hacker News. Maybe it's just observer bias, but I had never heard of requests for Python until I read an article on HN about scraping, which in turn got me interested in Python, and now a week later a major release of Requests happens with a top story on HN.

Is the blog post the only direct documentation of what needs to change to upgrade to 1.0?

Details seem relatively scant. But maybe that's because response.json() is the only feature I'm using that changed...

This is great new - I recently integrated requests into my current project, and it's made many things a lot easier!

Used Requests in one of my projects. Definitely one of the best modules out there. Great news for all Python devs.

Is the verbose mode still around? Not having any configuration is nice, but what replaces this?

   session.config['verbose'] = sys.stderr

I couldn't find anything about it in the API docs.

I replaced it with real logging!

Need to show how to use it in the docs still.

Last week, I showed requests to a class of sysadmins to whom I was teaching Python and they were blown away. Great work.

This sounds great. I assume it's Python 3, right?

2.6–3.3

Woo hoo! If you don't mind a suggestion, you might add that fact to your feature list. It can hardly be taken for granted and is a very important feature to those who are already past their legacy upgrade hurdles.

If you can share some advice on Python 2/3 compatibility that would we very interesting. Especially things related to Unicode and testing. I see you have a single codebase for both versions. That is the same approach I took with my (much smaller) library, but several people advised against it. I also see you have very few "from __future__ import ..." Is that a conscious decision?

In any case, thank you! I will use Requests as an example to follow.

Using a single codebase for both Python 2 and 3 seems the most fashionable thing these days. It is, after all, the approach followed by Pyramid and Django. I also think it makes maintenance easier. The __future__ import is a consequence of it.

Congrats, Kenneth!

Any hints on how to solve these incompatibilities between HTTPie and Requests v1.0 appreciated: https://github.com/jkbr/httpie/issues/113.

Very nice, Thank you! I'm curious, why did you remove the json property?

I just started using requests and one of the first questions I asked myself was "Hey, why isnt't this a function rather than a property?" . I'm not that seasoned python developer but it feels wrong to me that you have to perform work in order to calculate a property, making it a function feels more explicit about that "there is work being done here".

Requests is great by the way, had already written some code to abstract away the urllib/urllib2 mess but this module is so much more complete (of course) and cleaner.

I actually disagree with this - the point of properties (in many languages) is to handle exactly the case of "there may be some transforms on this data, and they depend on internals, but really it is just a data view, it should be accessible like other data."

A good example is C#/.NET issues (and somewhat java) where you have .length .Length and .Length() and .Size() and .Count() and so on. It makes changing a data type an exercise in annoyance, and on the one hand there is a decent argument about semantic meaning of the name, on the other hand, tracking each and every subtle use of "I need to know how many things I'm dealing with" in several different ways becomes an exercise in yak shaving.

As for the specific of .json - A lot of python web frameworks provide a .json attribute/property to the request structure, so I presume the earlier version of Requests was mirroring that for conceptual continuity.

The issue is you don't want to raise an exception when returning a property.

I was dead wrong on what the issue was, but the actual explanation does actually hint that that doing error-prone calculation (such as deserialization) to magically calculate a property is not a good idea. I haven't seen the .json property used before but that only shows that I haven't looked into that many frameworks.

I'm not sure why java uses .length for arrays and .size() for collections ... but could it be that it was for similar reasons?

The whole point of the @property decorator is to compute properties. It's no big deal really, I'm just interested in the design decision in this particular case.

it's possible that it was a method all along, just callable as a property? I haven't looked at the source.

just a guess but if .json was implemented as a property and deserialization fails, you would get an AttributeError instead of the error reporting that there was a problem with deserialization. leaving it a method call would allow for the real error to bubble up as normal.

Bingo :)

i pretty much stopped using properties after I got bit by that. :)

Where does this happen? I'm not seeing it: https://gist.github.com/4329831 – the exception seems to come through just fine?

Feels weird. You shouldn't need to catch an exception when accessing an attribute.

Looks great! Unfortunately this broke one of our external dependencies which was careless enough to specify any version greater than 0.14.

This is why we should maintain our own pypi repository. Your external dependency did nothing wrong.

Wow, that's annoying. Which project was it? Could you let them know, perhaps pull request a fix?

Good job I guess (I use `requests` all the time and didn't even notice any issues, that's how neat it is). But the marketing-speak here makes my hair bristle!

"Requests is SEXY AWESOME!" "No wait, it's crap, complex, hard-to-follow code. But the NEW version is SEXY AWESOME!" (...at least until the next release, I suppose)

The developer interface was elegant, but the internal code left something to be desired.

That is no longer the case.

Dude you are my hero. I've printed requests source code and use it as an example of great python code.

You and guys like Armin Ronacher are the reason why I use and like Python so much !!!

(when jerking stops)

Question: I have a feeling that with Requests you're just hiding the problem and not trying to solve it (Urllib is still in the back). Shouldn't urllib(1,2,3...) be rewritten instead?

Probably, but rewriting a core module (this would be urllib4!?) is a lot harder (politically at least) than creating a 3rd party wrapper module.

Thanks!

Urllib isn't used at all, actually. Internally, I use a project called 'urllib3', but it's totally unrelated to urlilb. It's name is actually a joke.

It is essentially a light wrapper around httplib that provides connection pooling :)

By the way, I found a bug in httplib where, when sending a request body, if the peer closes the socket early with an error (HTTP/1.1 compliant behavior), the request will time out instead of returning the correct error information. To my knowledge, this is also present in httplib2. This is because the socket write in both libraries is blocking (which seems simply wrong). I'm hoping to send a patch sometime soon.

Regardless, httplib2 seems to have lots of other technical advantages over httplib - have you considered using it?

http://code.google.com/p/httplib2/source/browse/python2/http...

The entire process of sending a Request is wrapped in a 'try/except Exception' block. wtf. It also just aimlessly retries failed requests. This is not configurable.

Httplib2 is terrible. I actually created Requests partially in spite of it.

Makes sense, thanks. I'll have to work on patching up httplib, then.

httplib2 is not thread-safe. That pretty much makes it a deal-breaker, even if the API is a significant improvement over the standard library (though not over requests).

Well that's a different story then :) Good job!

I really respect both Kenneth and Armin for their work, but I find that Armin sometimes puts way too much trust in magic :)

Where did I do that?

http://dev.pocoo.org/~mitsuhiko/didntknow.pdf

"Custom Ducks" and "Cached Properties" terrify me a bit. Not Pythonic (as in, not easily readable).

But on the whole, I take that back. I was looking through your talks and realized that was the only place I got that feeling, and also found http://dev.pocoo.org/~mitsuhiko/badideas.pdf which takes a stand against magic.

Edit: I still think arguments would be better (more explicit/readable) than globals (request, etc.) in Flask, but it's your module and I know you've considered the reasons to make it that way. And globals don't exactly qualify as magic.

I love cached_property, and I don't see what makes it unpythonic. I don't consider it magic, it's just a small addition to the built-in property descriptor, and it makes the often repeated code easier to read. I'm sure it will be a common idiom very soon.

Are you perhaps confusing Armin with the creator of another Python web framework? One of the things I love about werkzeug and Flask is the relative absence of magic. Compared to many other popular web frameworks, his projects are extremely Pythonic.

No, I mean the right Armin. See the other reply though. I only got that feeling once when looking through his talks, and I otherwise take that sentiment back.

i just spent 20 minutes reviewing the code. it is indeed very clean compared to the last time I remember looking at it. great job kenneth

Leaves the following interesting question:

If the developer interface was elegant, and the library itself ran very well without any errors, is anything actually gained by making the internal code elegant?

What happens if the new version, while more elegant internally, has a new complex bug in it (such as a race condition, that only triggers once every billion requests)? Still worthwhile to fix up the inelegant code that was working correctly?

(Disclaimer: I feel it's better to clean it up regardless, but I have a feeling a study would show something different in terms of lost productivity...)

It does surface to the lower-level user interfaces, as shown in the blog post's connection adapter example. For advanced users, these changes will mean the world.

The new code is also much easier to debug and pleasure to work with. Bugs will be significantly easier to fix :)

> What happens if the new version, while more elegant internally, has a new complex bug in it (such as a race condition, that only triggers once every billion requests)? Still worthwhile to fix up the inelegant code that was working correctly?

Then it's easier to find. All software has bugs, readability is one important part of fixing them.

The "developer interface" includes at least three different parts: the API calls, the calling sequence, and the validity of the results.

That last part is borderline. Some wouldn't call it part of the interface, but I do, because if there are problems then interface users end up wrapping the interface with another interface to work around those problems.

I therefore argue that the "more elegant but buggier" internals don't have the same interface, even if the API calls look the same.

As to why it should be more elegant? There are at least four parts here . How does the developer, or any interface user, review the code to see if there is a complex bug inside? That is, your scenario about the bug in the elegant code also applies to the inelegant. And how easy is it to carry out extensions to the project in the old code base vs. the new, even if those new features aren't yet implemented?

Third, the type of person who strives for an elegant interface also probably strives for elegant internals. Fourth, there are people who don't like closed boxes, and want to understand some of the insides of a package before they use it. If the insides don't look good, then they might be adverse to using it.

A term for the negative aspects of this tendency to clean up internal code is called "gold plating." A positive term is "technical debt", as in the inelegant code incurs a debt which must be paid off in the future by cleaning up the code.

Testing becomes immensely easier. One of my main gripes with requests was that it had a fantastic API but had very unpleasant internals that made it complicated to test and mock http responses.

If this is no longer the case, then this indeed is a fantastic change - looking forward to diving in soon.

I've never used an open source library where I didn't end up having a question of "how exactly does this work?" (or related "If I do this thing, am I being redundant or interfering with normal operation?". Even with good docs. Requests is a good example of this, I didn't really fully understand what details were and were not kept in their sessions and what I needed to do w.r.t them and some testing I was doing (using requests, not on the lib itself). So I went code spelunking. I didn't think the code was bad to begin with, but if the code is cleaned up, and it would be even easier to do this, kudos on them. They made a future me or someone like me more productive!

It's a little bit naive to ask that question, isn't it? The decision to refactor internals is always related to how much technical debt the developers feel the project has incurred, and how much easier they think it will be to develop in the new codebase. So every rewrite is a subjective call, where future ease of development and attracting/on-boarding new developers is traded off against short-term lost productivity and potential for bugs, incomplete rewrites, and dropped support.

Congrats. Requests has been super useful to me. Awesome stuff.

Congratulations Kenneth! Well earned, well deserved.

Next step... Tablib v1.0? ;)

Actually, yes :)

Great! There is a problem with pip install tablib:

http://dpaste.com/hold/847350/

xlwt3 development stopped: http://pypi.python.org/pypi/xlwt3/0.1.2

Maybe xlwt3 could be replace by openpyxl? http://pypi.python.org/pypi/openpyxl/1.6.1

Going to do something similar. Remove all the formats. Pull them into separate libraries.

xlrd is going to get Python 3 compatibility: https://github.com/python-excel/xlrd/pull/21

Once that's done, someone has volunteered to add Python 3 support to xlwt as well. openpyxl isn't a direct replacement: it uses the XML-based xlsx format, whereas xlrd/xlwt use the binary xls format.

I was hoping Dates & Times for Humans would be next. ;)

What's the equivalent for Ruby?

httparty

Restclient as well from https://github.com/archiloque/rest-client

Can we all agree that it's a little weird to pause mid-article to fawn upon one's own code?

I love Kenneth Reitz and admire his skill but "come on, man."

No, we can't all agree that anything about that post was "weird". I didn't even see any "fawning". Just someone who's enthusiastic about his code. It's ok to love something you made and let people you think it's great. "Fawning" doesn't even mean what you think it means.

Yeah, it's not like I want to tattoo the source code all over my body or anything.

Well, now I kind of want you to.

Haha, on the other hand, one of the typical interview questions is 'mention something you built that you are proud of'. A developer's satisfaction with his own craftsmanship is, to me, a must.

So you can't just produce good, solid output without congratulating yourself in public? Interesting.

No, what I meant is that if someone is proud of their code and aren't afraid to share it with the world you should know probably lighten the hell up and let them be.

I wasn't trying to be mean, I just chuckled when you said, "Beautiful."

I'm grateful for the love and care you've put into libraries like 'requests'. When I've been toiling away at a program for hours and it winds up resembling something like spaghetti mixed with beer shits, I find that if I stop and study some of your code, I am instantly a better programmer.

Oh, I was just poking fun at the guy that responded. He has a testimonial on the requests homepage:

http://docs.python-requests.org/en/v1.0.0/

> I’m going to get @kennethreitz’s Python requests module tattooed on my body, somehow. The whole thing. — Matt DeBoard

Now that sounds like a great idea.

Reply on news.ycombinator.com