Python for Humans | Hacker News Reader

Python for Humans

speakerdeck.com

256 pointsColinWright73 comments

Wow, this is cool. Kenneth has started to identify and call out other libraries that are "barriers to entry" in Python. If this is a call to arms to scale the Requests experience, it could grow into something really powerful - a movement to systematically reinvent and rejuvenate the Python standard library.

Yep. I was looking for exactly this list.

Incidentally Python could use an more equivalent to djangopackages.com to help in identifying popular alternatives to standard lib packages.

While not a table of "stdlib package => 3rd party package", there is this: http://docs.python-guide.org/en/latest

[ Also by Kenneth Reitz ]

That link doesn't appear to work for me, so I think the same content is available at http://docs.python-guide.org/

Sorry. Looks like the trailing slash was omitted when I copy-pasted that from the location bar. ^^;;

That does seem to be the problem. But that seems wrong?

I'm of the leads on that project. We're working on upgrading the code base to support all of Python, and hope to have that ready by PyCon US.

Slightly off topic: in an effort to become a better programmer, and to improve the Python ecosystem, I've tried to write a "for Humans" style logging library. Would anyone mind giving me feedback?

http://peterdowns.com/lggr/

Looks great. Can't get to testing it today but I will get back to you soon.

Thanks! Is the API simple enough?

Some feedback from reading this page:

  - Why is your example logger named d? It might seem nitpicky but it's hard
    to read an example with a meaningless single character variable.
  - "d.close() # stop logging" - what is this? What does it mean to "stop
    logging" and why do I want to?
  - "Add a coroutine" - I would wager the lay Python developer doesn't even
    know what this word means.
  - lggr.Lggr() - Why not just name the class Logger?
  - The default format variables are inconsistent about when words are
    separated with an underscore.
  - I can't make sense of the example logging calls. In one you pass a 
    dictionary as the second argument, in another you pass three strings as
    separate arguments. Why would anyone pass a message like this instead of
    just using standard string formatting? Especially when there's other
    legit arguments like extra. It's not even really clear why you'd want to
    pass something in extra instead of in the message.

I fully support your goal, but at a glance this seems like a confusing alternative to logging.

EDIT: hot dang it's hard to format a bulleted list on HN

Hi jwpeddle, thanks for all of the feedback! All of the changes I've made are on a branch called 'hn-fix' (https://github.com/peterldowns/lggr/tree/hn-fix) if you'd like to take a look.

    > Why is your example logger named d? It might seem nitpicky but
    > it's hard to read an example with a meaningless single character variable.

No particular reason -- you're right, I will change this now.

    > "d.close() # stop logging" - what is this? What does it mean to "stop logging"
    > and why do I want to?

This is a way to "close" the logger and clean up all of its associated coroutines. An example of when you might want to is upon catching a fatal error to your program, you might want to log the error and then safely clean up all open files or network sockets to which the logger is writing.

The method name is clearly confusing -- what do you think of using "shutdown" instead?

    > "Add a coroutine" - I would wager the lay Python developer doesn't
    > even know what this word means.

Maybe not, but they should! The readme now includes a link to dabeaz's coroutines page, and I'll add a quick overview in a couple of minutes.

    > lggr.Lggr() - Why not just name the class Logger?

For consistency's sake. Maybe if I had to start again I would call the project `logger`, but I decided to be "hip" and use a vowel-less name instead :)

    > The default format variables are inconsistent about when words are separated
    > with an underscore.

The initial idea was to mimic the variable names from the default logging module (http://docs.python.org/2/library/logging.html#logrecord-attr...). You're right that it is confusing though! I think I will rename everything to be lower case, one word, instead of the default module's mix of camelCase and underscore_separated names. Thoughts?

    > I can't make sense of the example logging calls. In one you pass a
    > dictionary as the second argument, in another you pass three strings
    > as separate arguments. Why would anyone pass a message like this
    > instead of just using standard string formatting? Especially when
    > there's other legit arguments like extra. It's not even really clear
    > why you'd want to pass something in extra instead of in the message.

I should definitely clarify what formats are allowed and aren't. The log message format is using standard string formatting -- string.format(), to be exact. The 'extra' argument is a result of trying to imitate the default library (see http://docs.python.org/2/library/logging.html#logging.Logger... for a description of the 'extra' kwarg), and can be useful when you'd like to pass information to every single log message.

I think "close()" is the appropriate name, as it's analogous to closing a file.

Why not just add an atexit trap that automatically cleans up open logs?

http://docs.python.org/2/library/atexit.html

I am fairly sure that most of the streams and sockets would get closed when the interpreter exited regardless.

Also, in accordance to the principle of "explicit is better than implicit," it's generally better to avoid messing with global state (such as atexit) behind the scenes, and to provide a way to shut down the logs manually.

LeafStorm got to it before me :)

Maybe you want to support python's with statement instead of a close() method?

It's trivial to format a bullet list on HN:

* Leave blank lines between bullet points,

* Start each line with <splat> <space>

* Bob's your parent's brother.

Hope that helps. See also:

http://news.ycombinator.com/formatdoc

your library has no tests! it's a good effort but I honestly wouldn't use it yet.

a few nit picks: i can't imagine a scenario where i'd want my logger to close stdout (or any other file descriptor) for me.

there is a lot of missing error handling, which is really important for something critical like a logger. what happens when disk space runs out? no timeouts on network operations?

also your SMTP and Gmail loggers don't form valid MIME messages (I can't log non-ascii?). you also seem to just swallow exceptions which is totally not what I would want or expect from a logging library.

keep at it though, the logging module's API (inspired by log4j) is fairly painful.

    > your library has no tests! it's a good effort but I
    > honestly wouldn't use it yet.

You're right, and I'd love to have you help me add them!

    > i can't imagine a scenario where i'd want my logger to
    > close stdout (or any other file descriptor) for me.

By default it won't -- see https://github.com/peterldowns/lggr/blob/master/lggr/__init_....

    > there is a lot of missing error handling, which is really
    > important for something critical like a logger. what
    > happens when disk space runs out? no timeouts on
    > network operations?
    > [...]
    > you also seem to just swallow exceptions which is totally
    > not what I would want or expect from a logging library.

Lggr will fail silently by default, because your logging library shouldn't cause your code to crash. If you'd like, it's quite easy to stop it from suppressing errors. For more complex error handling, I think users should write their own coroutines for handling log messages -- the default file printing and network sending coroutines are merely the most basic case.

    > also your SMTP and Gmail loggers don't form valid MIME messages
    > (I can't log non-ascii?).

Please help me and add loggers that do format valid MIME message! The included loggers/coroutines are just a few examples I came up with, but I'd love if other people were to contribute more :)

    > keep at it though, the logging module's API (inspired by log4j)
    > is fairly painful.

Thank you, and thanks for your feedback :)

I like the idea of delegating logging to its own process in theory (in practice, it would depend on your configuration, I guess, it may mean a lot of context switching), but your built-in coroutines seem to rely on yield instead. Have you experimented with all three styles?

A big feature missing, IMHO, is file-based configuration. You definitely want this for larger projects.

Interesting. This is weird as fuck, though:

    d.critical("Someone {} us {} the {}!", "set", "up", "bomb")

See that seems ambiguous - does it actually use the string formating mini-lanaguage?

Most unpythonic module name ever. Change it to logger.

This keeps getting reposted, http://news.ycombinator.com/item?id=3419671 http://news.ycombinator.com/item?id=4117035 http://news.ycombinator.com/item?id=4651882

Are you sure these are all the same? I can't check from here (on my mobile, poor reception) but the URLs look significantly different. This one is a slide deck, one of the ones you quote is a video.

Added in edit:

It does look like this one - http://news.ycombinator.com/item?id=4651882 - really is a duplicate, but it got no votes and no comments, so perhaps a repost is in order.

It was reposted today because he gave the talk again this morning at Pycon Canada, I suspect.

Reminds me of a chat session I had with a dev at another office a few days ago:

  him: "have you ever used urllib2"
  me: "not if I can avoid it"
  him: "I've just spent 3 days trying to do X"
  me: "have you heard of Requests"
  him: "nope"
  ...send URL...
  him: "god damn it!  Why didn't I talk to you three days ago"

20 minutes later he's replaced three days work with a dozen lines of easy to read Requests code.

Was there today at PyCon Canada and not only was the presentation Kenneth gave very good, but he was very awesome to talk to afterwards. I randomly came across his 500px account, so we sent him a shirt, and he wore it on Friday just to say thanks :3 (I actually didn't get to see him on the Friday night party, probably because I know him best by his pixelated twitter pic, but it is super cool that he take the time to think of stuff like that).

While I appreciate the effort to make some more simplistic APIs available alongside their more complex counterparts - libraries like urllib2 are not expendable in the process.

In the act of making an API simpler and abstracting the details - you loose a lot of the detailed control provided by these lower level libraries. Perhaps you won't miss the 10% as outlined in the slides, but someone out there will.

Do we really need both? I think no, but then again for most of what I do, urllib2 just works; I'm not really the target for requests. Just don't expect that requests will ever replace urllib2.

If requests hits the 90% and urllib2 the 10%, then requests should be the standard library and urllib2 a module. I think his point is more to point out good modules that most people should use, and should become common knowledge of what people should turn to when trying to do something. Java, for example, has several date and time options but Jodatime actually does what most people want to do with dates and times.

For new projects the use HTTP at a high level, I believe that there is no reason to prefer urllib2 to requests.

Probably it won't replace urllib2, but it has attained critical mass and won't go away soon.

Anyone needing more power/flexibility should use httplib.

A bit off-topic, but I couldn't find this and was very curious: What font are the slides using in the code snippets? It looks very clean--I didn't realize it was monospace at first.

+1. Would love to know the name of that font.

Looks like Ubuntu Mono.

It is Monofur: http://www.dafont.com/monofur.font, IMO it looks better on slides in small doses than in an editor all day long.

And also, what's the main font used for all the text? It's a lovely serif

Looks like this one: http://www.azfonts.net/load_font/high_tower_text.html

Spot on. Thanks! http://www.myfonts.com/fonts/fontbureau/hightower/textroman/

I was at his talk this morning at pycon canada. It was a very good talk, the highlight of the conference. I like his approach, although I think it'll be a challenge to get there since not everyone has his "visionary" and api building skills, but someone has to start somewhere. Well done.

Also there- loved it. I think the key is what he said about writing the readme first. Write the docs on how you'd want to use the software, then hold yourself to that API when you actually write it.

As much as I know everyone dislikes PHP, there's lots of people still working with it. I love Requests in Python, so I built Requests for PHP: http://requests.ryanmccue.info/

I have to agree with Kenneth's points, in that pragmatism should outweigh the theoretical points. For example, in PHP, using a class as a grouping for static methods is a bad idea. I agree to a point, but there's something to be said for the ability to do `Requests::get('http://google.com/)` versus all the crap you have to do with cURL. (Plus, it's always >90% test coverage, which I stick to religiously.)

If autoloading namespaces or namespaced functions was possible then static classes wouldn't be needed and things like this would be commonplace:

    \Requests\get('http://google.com');

Another reminder of why in the fuck did they use the backslash character.

That's true, and it's something that's being discussed on the PHP core mailing list.

However, I support PHP 5.2 (pragmatic reasons; 5.2 is still used by a huge percentage of hosts), so I wouldn't be able to use that anyway.

I'll check this out. I use python at home but have to use PHP at work. Thanks :)

A little nitpick. In your tutorial you do:

    $request = Requests::get(...);

This should really be:

    $response = Requests::get(...);

As you get a Response object back.

Good point, thanks. I'll fix that up.

I think in the Python version they just use "r" so it's ambiguous.

Here's a video:

http://www.youtube.com/watch?v=Y97D8j6CAog

His requests module is an improvement, but he says being able to specify the outbound IP of an HTTP request is out of the scope of his module???

https://github.com/kennethreitz/requests/issues/394

That's why I am still using urllib2.

I'm not sure why you feel this niche requirement requires three question marks. The point of modules like this is to make it easier to do the tasks that 95% of developers are likely to need to do regularly, at the cost of not covering all rare use cases.

I think it is a fairly common requirement to use multiple IPs to avoid throttling or for APIs that restrict access to individual IPs.

The fact that he was confused by it does not help his credibility when he is trying to tell people how to design their libraries.

I have yet to ever have to use multiple outbound IPs from the same machine for an HTTP request from Python. It might be "fairly common" for whatever you do, but in terms of the general population, it's not.

Still, it's a fair point that it at least should be possible. It's fine by me if the uncommon 5% tasks are just as difficult to do in requests or even a little more so, so long as the most common 95% of tasks are a breeze.

It's a reasonable tradeoff for a specific library. But if he's trying to claim that the entire eco-system should put the API above all, then it's worth asking how the 5% use case integrates with this worldview. If I suddenly find I need multiple IPs, do I now need to switch my http client library?

The slides don't mention this, but he did touch on having layered designs so the 5% remains possible. However, the primarily exposed api shouldn't get sacrificed for said features. By conceding on a feature that makes the API less usable, you sacrifice more than not having it in the first place (confusion, complexity). Even the GH Issue linked shows a possible solution to the issue using urllib2, which is just a layer down.

I don't understand, what is not usable about being about do to this:

requests.get('news.ycombinator.com', sourceip='1.2.3.4')

No, you need to do what anyone who has fringe requirements has to do: patch the library yourself. Modify urllib3's (requests uses this as a go between for httplib/http.client) to accept/pass on the source_address that the HTTPConnection from httplib allows and then modify requests to accept/pass this on. This will take, at first glance, maybe half a day with some tests (likely much faster, but I'll allow that there might be something more involved than modifying urllib3's HttpConnectionPool and the accompanying changes for requests's Request model.

urllib2 is so bad (and I know how to use it) that I have at times used subprocess + curl, which says a lot, because subprocess ain't exactly pretty either. But the thing is, urllib2 is included and is the devil I know.

I remember writing raw socket stuff and I actually thought that was better than urllib2.

I was googling to see the status of the 'requests should be in stdlib' debate and discovered PEP411: http://www.python.org/dev/peps/pep-0411/

Basically this helps mitigate the downside of a module getting included in stdlib.

There is a video of him giving this presentation on the PyCon Ireland, in case someone want to hear it.

http://www.youtube.com/watch?v=IN5AlsEHdIM&feature=plcp

>lxml is awesome, but difficult to install

So true... I often need to write something that will run on a Windows machine and I just end up having to use the old version of Beautiful Soup (before it became dependent on lxml). Pretty frustrating.

lxml is great. The docs and the installation aren't the best.

For a windows install. http://www.lfd.uci.edu/~gohlke/pythonlibs/

Thanks so much! Just tried it and the setup binaries worked like a charm! 20 seconds for the whole experience including downloading and testing. Very different from how I remember my experience with lxml on Windows when I tried it before! They should definitely have a link to this website from http://lxml.de/installation.html#ms-windows

Absolutely love `requests`. First I've heard of `envoy`; will certainly be using that in the future. Thank you.

requests & envoy have saved me a huge amount of hours. I wish I met Kenneth when he was in Toronto this weekend, I would have bought him anything he wanted.

I'm still in Toronto :)

Haha, alright you name the place & time, I will bring said "anything you want" object.

Going straight to the Python Stockholm Meetup after yer flight? :-O Mmmmh, jetlag

I can't express how depressing dealing with text encoding issues in any language is.

spot on - if only I knew many others felt this way when building projects. lol

Maybe libraries should have a rating or something just for fun. Might help direct attention at problems like this.

Reply on news.ycombinator.com