This Is What Python Beginners Have to Deal With | Hacker News Reader

This Is What Python Beginners Have to Deal With

pythonforengineers.com

120 pointsreubano160 comments

> Now, compare that to what happens when I search for the same in Python. I get this:

> https://docs.python.org/2/library/re.html

I clicked and I saw the kind of technical documentation I love to read: precise, with examples. There is an introduction to Python's flavor of regular expressions, which highlights the potential problems (backslashes). This is not targetted at novice programmers, though.

> That’s the generic re library page. I have to scroll down to see which function I need.

Did you notice the table of contents on the left? You might want to try Ctrl+F. But you might have to scroll down too, I don't deny it. You have to read, too.

> And even then, all I get is a hundred words of generic description. No examples.

Ctrl+F "example" highlights the table of contents. There is an example for "findall".

> I then have to Google Python regex findall examples, and go through a dozen blogs written in 2008.

http://lmgtfy.com/?q=regex+findall+example#

Not really, there are plenty of recent posts (stackoverflow, etc). And having articles from 2008 would not even imply that they are incorrect. In order to see how old your post is, I have to look at the source, but even though it is quite recent, it contains factual errors.

Regarding the docs layout and "having to scroll down to see which function", I also don't quite get it. From my perspective, it's a very nice trait of Python docs compared to PHP docs, that the former actually explain how to use the various parts of the API together, before giving descriptions for every specific API. It's simply a much better way to digest it and learn to use it without making a mess (speaking of which and PHP...).

Coming from statically typed languages, I've really struggled with the Python docs, because they aren't really clear with what is returned from a lot of functions, and which arguments are allowed. I end up finding an example where it is being used, then deducing from what they do with the result what was actually returned.

Contrast this with the ugly-but-functional JavaDocs, which can be auto-generated, and easily shows all return types and arguments for every publicly exposed function and member variable. I understand why this can't be done automatically in Python, but at least they could put the time in to make it clear in their standard libraries.

Python also uses autogeneration for most of the documentation, but the documentation itself is not fully autogenerated. You write your documentation and you say to Sphinx: put the generated documentation for this module here.

IMO, this is something positive. You can decide the structure of your documentation and include only the most relevant references.

For example, this part of the documentation for the re module is autogenerated from code: https://docs.python.org/2/library/re.html#module-contents

Agreed, but my point is that argument and return types are badly documented, and in Python autogen doesn't help this.

Usually there are examples for everything. Sometimes you have to scroll down, though. What the function returns is usually stated in the description. But I know what you are saying, actually most 3-party libs do document types.

IMHO, python docs are ok. If you struggle with them as a beginner I don't see what would change as an advanced programmer.

I am an advanced programmer.

I shouldn't have to read code snippets to figure out if a function returns a tuple or a 'group' which can be iterated over, but can't be indexed. This is something which would be fixed if the documentation included types, because then I could check what all the allowable operations on 'group' are directly without searching for a code snippet which does what I want or just running it and praying I got it right.

To provide a counterexample, it takes me forever to figure out how to do something after reading Java/Cpp docs, but I instantly know how to do it in Python after reading the docs.

Typical workflow for Java: This function takes in these object types as inputs... Then you have to wonder how to get those object types so you scour through 10 more pages of docs to figure that out. You end up creating a factory of a factory of a factory to make that input object only to find out that it has to be further preprocessed by sending it somewhere else.

Typical workflow for Python: read docs, try out an example in the interpreter by passing a basic object type to the function (such as string or dict), copy code from interpreter into text editor. You are done.

I really love Python docs. It takes me seconds to find what I want and figure out how to use it. I hope they don't change anything.

That's more to do with coding style. I abhor reflection and factories, so if I write Java I don't use them. That said, you can end up writing a RandomNumberGeneratorFactory in Python just as easily if you are trying to achieve the same scaling and re-usability requirements that the Java program is.

The issue I run into in Python is that I need to call something which takes * args or * * kwargs, and the only way I know any of the arguments exist at all is because of some snippet of code on github, probably by the person who implemented the feature in the first place. Even if the argument is named I don't know if it should be a magic number which is defined in some enum-like class, a boolean, or a string to describe the name of what I want and pray that the library maintainers used the same capitalisation and localisation that I think is most logical. In fact, without looking at example code, I literally have no clue how to do anything.

As such, with Python I end up doing copy-paste programming which is dependant on my internet speed and how many other people had the same problem already. With C/C++/Java I tend to do a more build-it-from-scratch style programming, driven mostly I think by the style of the documentation.

> I abhor reflection and factories, so if I write Java I don't use them.

But factories etc are idiomatic Java. You can avoid it in the code that you write, sure, but how do you avoid it in the third-party libraries that you use?

I've been lucky enough to use Java for fun, not work, and as such I have the time to write things the way I feel they make the most sense.

The advantage of Python is that you do not need to think about type labels: "If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck."

For example, the builtin print() function accepts any object with write(string) method as the file parameter. There are ABCs classes that capture the "duck" essence but it would be an unnecessary pessimization to require a io.TextIOBase instance as the file parameter in this case. Don't write Java in Python. Go's io.Writer would work in this case (the type label as the documentation and the exact specification) but if you don't mind thinking about type labels and/or believe that the type of bugs that can be found by a compiler worth the trouble instead of just writing tests and expressing directly what you want the code to do then using a statically typed language make sense.

Though there are type hints (proposed by BDFL) in the recent Python (to enable better static analysis, IDEs code completion and refactoring) https://docs.python.org/3/library/typing.html If its usage is not limited (e.g., through Python culture convention); it could be a regression in Python evolution. If your API is so enormous that you need type labels; you should split it. Python code should communicate the intent to a human first over a compiler/static analysis tool.

Here's a static typing programmer perspective (how it feels) on dynamic languages "What is the appeal of dynamically-typed languages?" https://gist.github.com/non/ec48b0a7343db8291b92

Wholly agreed about not writing Java in Python, and I do write idiomatic Python. The problem is that I don't know what methods the returned objects have available, and as such whether they are a duck or a tiger. The examples show that it has a mouth and eyes, but I need to know whether it has wings or claws, and Python's docs don't make that clear.

Can I index the returned object, or only iterate over it? Is it mutable? Is it a dict or a tuple of (key,value) tuples? Did the author think it was worth it to make its own class for the return type, because they want the indexes where the value was found as well as the metadata?

Python, to a degree, brushes over these problems by being able to print out most types of data and then figure out where the problem is, but TBH I'd prefer to write the correct code to begin with, and the docs don't give me enough information to do so. As such, Python turns into a dodge-the-error-driven-programming, and I feel much less productive than I could be.

Yes, duck-typing gets you part of the way, but it doesn't let you pretend type doesn't exist entirely.

I feel like using Go's interfaces would be more powerful for a duck-typing language than having type-annotations.

"The problem is that I don't know what methods the returned objects have available, and as such whether they are a duck or a tiger."

I don't get this... Why not do the following:

    In [1]: import requests
    In [2]: resp = requests.get('http://www.google.com')
    In [3]: resp.
    resp.apparent_encoding     resp.iter_lines            
    resp.close                 resp.json                  
    resp.connection            resp.links                 
    resp.content               resp.ok                    
    resp.cookies               resp.raise_for_status      
    resp.elapsed               resp.raw                   
    resp.encoding              resp.reason                
    resp.headers               resp.request               
    resp.history               resp.status_code           
    resp.is_permanent_redirect resp.text                  
    resp.is_redirect           resp.url                   
    resp.iter_content

Now you can try each of those things and explore their methods.

    >>> import requests
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: No module named 'requests'

Now, what was your question?

Because I'm not always scripting - sometimes the work I'm doing takes 10 minutes of runtime to get to the end and I want to know what I'm writing before I run it.

If Python wants to remain a scripting language forever then the docs are sufficient. But if it wants to graduate, and it seems a lot of people think it already has, then the libraries need to be better documented.

If you don't know whether a method returns an iterator or a mutable/immutable sequence (regardless of the specific type) then the documentation is incomplete (it is a bug). If it is in stdlib; you could submit a documentation patch for a *.rst file on https://bugs.python.org/ See https://docs.python.org/devguide/docquality.html

The author has a vested interest in making people believe they can't figure it out themselves. The article is an advertisement for their book/videos.

The author also has experience teaching beginners (presumably), giving them a perspective that experienced programmers very easily forget and/or ignore.

On the contrary, I learned python starting with the docs, and I am very picky, if something is difficult to understand I am the first to complain, but I did find python docs to be okay, considering I started learning the language from the docs itself.

They might not be the best, but they aren't what the author says like 'we just can't understand shit'

I learned python purely from an offline copy of the official docs, back in 2001 or so. It was obviously doable, since I did learn Python, but it wasn't particularly easy and I remember starting to really like Microsofts documentation (eg their VB docs).

Python's docs have improved, but they're not amazing. Then again, neither are the docs for many other languages. For example, I'm a big fan of Clojure, but many of the docs there have me really appreciating Python's docs ;-)

I think once you get used to Pythons docs, they're pretty easy to follow, although I agree that type information on arguments/return values would be great and I also largely work from examples unless I'm already familiar with a library and am just looking for reference material.

That's what my point it, the author, in my opinion is wrong about the docs, since even I learned the language entirely from an offline html copy of the docs.

Yeah, I wasn't disagreeing with you. Just adding my own anecdote.

Yes, I know that :-) Sorry for being unclear in my comments!!

Googling about Python regular expressions still returns the “Regular Expression HOWTO” near the top of the results. https://docs.python.org/3/howto/regex.html

That’s pretty much the best succinct introductory page about regular expressions (applying to whatever language) online, and has been for 15+ years. After finishing that page, a beginning python programmer should be ready for the library reference and/or a more serious book.

(No idea why the author here thinks that examples from 2008 are such a terrible thing. Nothing has changed about Python regular expressions in ages.)

Overall, the Python docs are amazing. Not perfect by a long shot, but miles ahead of PHP or C++. Pretty much the best programming language documentation I’ve ever used (but I admit there are many programming languages I’ve never tried). Perhaps that’s partly down to the language itself making a lot more sense than many alternatives. The docs don’t have to do as much work to paper over brokenness.

His take on the community also bears no resemblance to my experience. People on python mailing lists, IRC, local meetups, Pycon, etc. have unfailingly been friendly and welcoming and willing to bend over backwards to help newbies.

I remember first learning about NLTK, not knowing the first thing about Python, or anything at all about natural language processing.

And I could just "get it". My lack of Python knowledge did not get in the way because it was like reading pseudocode. The language is so beautifully designed for the majority of the cases where you do any kind of algorithmic/math work.

In fact, it may have been too beautifully designed. :-) From what I can see from comment threads [1], most people seem to want to use NLTK even if there are a lot of complaints about how it is too slow for production, and how there are much better alternatives for NLP when it comes to performance. I think people are attracted to it because of its simplicity.

[1] https://news.ycombinator.com/item?id=7658977

Quoting myself from a blog post i wrote when i was learning Python two and a half years ago [1]:

Let’s say i want to know how to do the Python equivalent of PHP’s foreach. The docs about the for statement say this:

    The for statement is used to iterate over the elements of a sequence (such as a string, tuple or list) or other iterable object:

    for_stmt ::=  "for" target_list "in" expression_list ":" suite
          ["else" ":" suite]

Right, so what does that mean? What’s ::=? What’s an ‘expression list’?

    The expression list is evaluated once; it should yield an iterable object. An iterator is created for the result of the expression_list. The suite is then executed once for each item provided by the iterator, in the order of ascending indices.

“Ascending indices”? “Suite”? This might be clear for a CI professor, but it’s not for the mere mortals that read the docs and want to get stuff done.

Things haven't gotten much better since then.

[1]: https://www.haykranen.nl/2014/02/12/saying-hello-to-python-s...

You were reading the wrong docs for a newbie. The language reference is intended for people who want a very concise and precise technical document, not an introductory explanation. If you read the very beginning, the language reference is very clear about what it is and isn’t:

> This reference manual describes the syntax and “core semantics” of the language. It is terse, but attempts to be exact and complete. The semantics of non-essential built-in object types and of the built-in functions and modules are described in The Python Standard Library. For an informal introduction to the language, see The Python Tutorial.

You should have been looking at the Python Tutorial instead.

https://docs.python.org/3/tutorial/controlflow.html

https://docs.python.org/3/tutorial/datastructures.html#loopi...

Another nice example of what the author complains about, a "it's your own fault" reply whenever someone complains that the official language docs are hard to understand.

Personally I almost never find any use from the docs whenever they come up in a google search, its almost as bad as those expert-exchange hits that used to appear. The core issue being that pythons docs are only a language reference for the vast majority of the standard library, and the size of the stdlib bein one of pythons strengths.

Personally, as a 16 year old (or possibly younger, it was a long time ago) with no internet connection and little programming experience I found the Python user documentation very helpful. That possibly had something to do with me actually reading the tutorial rather than dismissing it as beneath me or ignoring it altogether and attacking the person who brought it up.

Well, the discussion is about the docs recently available online, so you finding a python book helpful in the times when no python shell autocompleted is a bit outside the topic at hand.

I'm talking about exactly the same documentation and the same tutorial that's available online (well, obviously it was a much older version of it). Only difference is that the offline version is a directory of local HTML files rather than a website. The user experience is pretty much the same with both. Still pretty handy even in the StackExchange era, though it'd be more useful if Google didn't keep finding the wrong version of the docs (2.x when I'm using 3.x or vice-versa).

It’s not “it’s your own fault”, but rather “you should read the document you’re looking at, and think critically, instead of expecting that every google hit is going to be precisely what you are looking for right off the bat, with no need to scroll or read”.

If the author had gone on the Freenode #python IRC channel or the Python Users mailing list (or..., or...) and asked “Hi, I’m a newcomer trying to understand Python loops, what’s the best resource?” someone could have linked him to the tutorial. Or if he’d done a google search and tried skimming a few hits, he would have found it.

It’s hopeless to expect that every single document in the world will be tailored for every reader.

There’s a real value to a concise and precise technical document aimed at folks who need something closer to a specification than to an introductory explanation. It would be net-negative to remove the language reference from the internet, or clutter it up with a bunch of information that can easily be written somewhere else.

Imagine someone wanted to understand the basics of how levers and pulleys work, and they came across a quantum mechanics textbook, which wasn’t comprehensible at their current level of understanding. Should the reader blame the book, or keep looking for a more appropriate source?

Sure, in a perfect world i would read the complete tutorial before i get started on a new language.

But it's not a perfect world, i want to get stuff done. I have twenty years of programming experience, i don't need to learn what a for loop is (the first link) and i don't want to read a whole document just to find a reference to the items() function (second link). I just want to know what the equivalent of foreach() is in Python.

Sure, a document with a 'very concise and precise' technical description might be useful to some, just for looking up things it's not. The current alternative is Googling StackOverflow posts.

Sure, in a perfect world the documentation would guess who I am and adapt accordingly, be both highly detailed when I need it or informal when I don't.

But it's not a perfect world, I want to get stuff done. I have twenty years of programming experience, I know how to find the information I need.

Sure, the whole language explained in one place for all kind of people from beginners to experts would be nice. But I have Internet. I type "foreach python" in Google and I see a StackOverflow post.

> I type "foreach python" in Google and I see a StackOverflow post.

That's kind of indicative right there.

Indicative of what?

Are you trying to say that a the docs should have "Python for X programmers" sections, foreach (no pun intended) X in {Perl, PHP, Haskell, Lisp, Go, Javascript, C++, ...}?

That would be nice, but what is even better is a totally different resource that aggregates idioms for all languages:

https://www.rosettacode.org/wiki/Loops/Foreach

That the first result is a stackoverflow question about the language rather than a documentation page, is indicative that getting started with the language is not so straight-forward, even factoring in documentation.

If you're googling 'python foreach', you're probably at the beginner level, and the fact that a documentation page is not the first result means that the beginners documentation is not very discoverable.

The fact that there's a not-closed, highly-upvoted, highly-ranked stackexchange question about this very basic issue in existence /at all/ is what one might call a "documentation smell".

The documentation contains "for-each" and "for each", but sadly not "foreach". Why should "foreach" be in the documentation? There is no "foreach" keyword in Python. If you are looking for "foreach", someone else will look for "python do until", "python fold", "python switch" and maybe another one for "python macros" or "python transducer".

Some of those searches are legitimate, because "foreach" is quite used outside of Python and "switch" should be expected to be defined in every language. However, the documentation states what the language is, not what it isn't.

The goal of a reference documentation is like a dictionary: you define your words w.r.t. the other words defined in your language; this is already quite a complex task. But you don't mention words from other languages because that would start to be overly confusing and a burden to maintain (now you have to check how other languages evolve). There are other resources to make connections between languages.

> Why should "foreach" be in the documentation?

To make it easier to find "for", "for each", and "for-each"?

How come you know "foreach" in the first place, this is not even a correct english term? You learn it from another language? Wonderful, but why should every keyword from every programming language ever designed be present in the Python docs?

I never argued for every for every keyword from every programming language. But like I said, that there's a stackoverflow question about this one, I think, is indicative.

And this one is special because?

With all due respect, if you have 20 years of programming experience and can't grok BNF, it might be worth stopping attempting to learn new languages and learn BNF instead. Given than BNF defines how Python (and many other languages) is parsed, it's well worth learning.

Actually, the folks with 20 years of experience in other languages are the ones who most need to read the Python tutorial.

The python code I read from PHP/Java/C++/... “career experts” tends to be worse than the python code I read from people who have only been programming for 2–3 years, because they never bother to learn Python idioms or style, but just keep on writing code in the same old style in every language.

It’s certainly possible to dive into a new programming language and start writing nominally working code right away, filling in with a google search whenever you get stuck.. That’s not a way to ever end up with any kind of basic competence in the language though. It also doesn’t give you much of a leg to stand on for complaining.

The Python tutorial is short and breezy, a fast read which will put you well on your way to understanding the language. I’d recommend it for anyone who plans to spend more than a day or two reading or writing Python code.

I think this is the exact type of comments that the author comes out against.

You aren't putting yourself in the shoes of a beginner.

Let's say I'm a beginner, and I want to just experiment with the language a bit (I've already read some tutorials or am not interested) - since I'm just trying to get the feel of the language and ecosystem, I'm not trying to get a deep understanding of the language and tools - I want to match a regex.

In the C++ and PHP examples he provided, the basic example is easily found - it's basically two actions - search in google and scroll to the example.

What you are suggesting takes several more steps - ctrl-f for a text that you might get wrong (you've searched for sample instead of example for instance), iterate until you get the right place, etc...

And that's assuming you even realize that you should run ctrl-f, why should you?

The main issue though is that if I'm a beginner I'm doing this kind of searches several times a minute - to have to stop and realize what to look for every time is extremely frustrating.

I've worked with Ruby,Python, Java and Javascript - python's (and django's) documentation is extremely verbose and rarely straightforward compared to the other ones.

> You aren't putting yourself in the shoes of a beginner.

How do you know if I am putting myself in the shoes of a beginner or not? And what kind of beginners are we talking about? Are they comfortable with a browser? Do they already know how to program something? You can't expect a single document to be relevant for all kind of people. The reference documentation has a clear target audience, which apparently conflicts with what you expect from it.

> And that's assuming you even realize that you should run ctrl-f, why should you?

> (you've searched for sample instead of example for instance)

Okay, so you are talking about absolute beginners, with no experience whatsoever with programming. And the C++ and PHP pages are somewhat useful for them?

> I've already read some tutorials or am not interested

> Let's say I'm a beginner [...] I want to match a regex.

Now, you are saying that you already have experience with programming? Google "python tutorial regex" gives plenty of results, the first of them being currently "Regular Expressions HOWTO" (https://docs.python.org/2/howto/regex.html).

Absolute beginners need a teacher, a mentor, or a step-by-step book, or a very gentle tutorial, or an online REPL. That's fine. You can't please everybody in the reference documentation.

And what kind of beginners are we talking about? Are they comfortable with a browser? Do they already know how to program something?

I'll add to that, "Are they even computer-literate in any way?" I'd worked at a college a few years ago, and it is sometimes shocking how little the newcomers know about just using and troubleshooting basic problems with computers, nevermind learning how to write programs for one... and IMHO it's only gotten worse. Knowledge of things like basic filesystem concepts ("What's a directory? How do paths work?") in particular seem to be disappearing.

You aren't putting yourself in the shoes of a beginner.

The real issue you and the author both run into is assuming that the wants, needs and learning styles of all people everywhere are identical. There's a good argument to be made for expanding the variety of styles and types of documentation available in order to cater to more ways of learning, but not a good argument to be made for suppressing or removing existing documentation since that boils down to "all right, let's cut off those people to focus on these people".

One big example that's not covered, but relevant, is how many (free, so potentially competing with the author's book) workshops or online complete guides to learn Python or Python-related tools have been popping up the past couple of years. Having an experienced coach who walks you through everything your first couple of days and gets you acclimated is a huge deal for some people (while others prefer to read on their own in an undirected way), but isn't accounted for in the article.

> he kind of technical documentation I love to read: precise

Nothing precise about it. Method signatures don't even include a clear description of what they can f return! I have to go and fish this infos from the text blob below. Even JavaScript gives you a damn return values section for every method: https://developer.mozilla.org/en/docs/Web/JavaScript/Referen... . I can't expect a type signature like in OCaml or Haskell (yeah, H sometimes gives you only a type signature which is even worse...), but still.

The cool thing about Python is not the high quality documentation. It's that you can still be productive in it despite it being a dynamic language with such a horrible and fuzzy documentation :) Guido really did a fine language design job.

The "text blob" always starts explaining precisely what kind of return value to expect under which circumstances. The return value's type can change depending on the context, and using plain text is somewhat easier to write and read than trying to come up with a dedicated notation.

See findall[0], for example. The structure returned depends on whether the regexp has capturing groups or not. A specialized notation could be designed to represent that, but as far as I am concerned, the textual string is good enough.

Just for fun, the return type could be expressed as:

      [string N]                 (A)
    | [(string M) N] : (M > 1)   (B)

    - Case A if the regexp matches N times without overlapping and without capturing any group, or capturing a single group.
    - Case B if the regexp matches N times without overlapping and, for each match, captures a tuple of M groups, as strings.

... where [type size] denotes a list of size "size" satisfying the type description "type". Likewise, (type size) is a tuple of "size" elements, all of which satisfying the same type. Or you could write t1 * t2 like in other languages to define cross-products.

[0] https://docs.python.org/2/library/re.html#re.findall

Your thinking is correct and precise, but I hope you never take a job of documentation writer for any project I will ever have to work with :)

My example of documenting re.findall all would be smth like this:

    Signature:
      re.findall(pattern, string, flags=0) -> [str] or [(str,)]

    Arguments: ...

    Returns:
    - [str] -- List of Strings matching the pattern if the pattern has no capturing groups, or only one capturing group
    - [(str,)] -- List of Tuples of Strings, one string for each captured group, if the pattern has more than one capturing group

    Examples:
      re.findall(r"h(\w{0,10}o)", "hello hoolo water")
      # -> ['ello', 'oolo']

      re.findall(r"h(\w{0,10}o)", "hello hoolo water")
      # -> ['ello', 'oolo']

      re.findall(r"(h)(\w{0,10}o)", "hello hoolo water")
      # -> [('h', 'ello'), ('h', 'oolo')]

    Details: ...put here some deatailed explanations about behavior with overlapping etc....

This is the kind of documentation that really spares me a lot of time. Your explanation may be correct but I'd have to read it 5 times and then open a REPL to do a small experiment and confirm that it means what I thought it means...

(I now I don't have the right to bitch about this though, the correct thing to do would be to donate some of my time and actually improve the existing docs instead of commenting on hn...)

> Your thinking is correct and precise, but I hope you never take a job of documentation writer for any project I will ever have to work with :)

I agree, that was terrible.

> My example of documenting re.findall all would be smth like this

The hard part is to have everybody agree on the documentation mini-languge.

https://www.python.org/dev/peps/pep-3107/

https://www.python.org/dev/peps/pep-0484/

I am a Ruby guy, but we had a project where we built a forking socket server in Python, I was really impressed with the documentation.

https://docs.python.org/2/library/socketserver.html

https://docs.python.org/3/library/socketserver.html

You know what puts off beginners?

Articles plagued with stuff like "this is horrible", "this is shit" and so on and son.

I see too often articles that just try to denigrate other technologies, saying that they suck, they are horrible, they are shit, they should be considered harmful, etc, just to try to make the author sound smart-assed.

The truth is that most stuff is the way it is for a reason. Maybe the situation has evolved and the reason is no longer valid, or maybe the people that built the thing had different goals than you. But understanding why it was valid for the other person and why it is maybe not valid for your use-case in an empathetic way will only make you a better engineer. We are all smart people trying our best to achieve something using technology. We are on the same boat. Let's stop insulting each-other.

If I was a novice and saw and article like this, I don't think I would want to hang out with such an overreacting crowd. Python is a libre project: instead of insulting other people's work to sell your book, try improving it. I can understand why proprietary software sometimes sucks -- it is disempowering -- but with libre software you are just kindly given a gift. If you don't like it, improve it and give back to the community, don't denigrate.

Many people have been and are contributing. But when there isn't critical mass or appreciation for changes they move on to other things. So the python community is not only left with the flaws but can't attract people to fix them.

I pick Python up maybe 3 or 4 times a year - usually to automate something or to explore some algorithm in a notebook - so I'm pretty much an eternal beginner (in respect of the language, rather than the development process). This means I'm hitting up the docs pretty much constantly and I do find them mildly frustrating. It's an organisation thing. As someone stated up thread, the docs are really comprehensive, which means they resist skimming. There's no distinction between - say - a language guide and a language reference. As someone coming at Python from a comprehensive background in other PLs I just want to see a damn example, I don't want or need to read a treatise on operator precedence or regex syntax. Fortunately this problem is easily solved by adding the word 'example' as a search term - and then going back to the super comprehensive docs to find out what I just did wrong with my regex syntax!

For a complete beginner, the documentation for most any language is just line noise at first, until you gain the appropriate context (for which there are a pile of easily googleable resources) and thereafter the more comprehensive the better because you don't know what you don't know.

Maybe if there weren't eleventy bajillion easily accessible 'how to X in Python' articles or SO wasn't a thing there would be some pressing need for differently organised docs, but there are and it is, so y'know, meh.

Edit: over opinionated autocorrect.

I think the Python docs are fine. I use them all the time. (The re docs in particular.)

It's not beginner-focused, but it'd suck if the current docs were replaced with docs for beginners.

The re page has over 7,000 words. I need those 7k words. They help me get things done.

Perhaps a solution might be something like a https://simple.docs.python.org type site: A simplified version, parallel to the existing documentation, consisting mostly of examples. I'd use it.

The issue with the Python docs is that you have to ACTUALLY READ IT. Some language have documentation that you can just sort of skim and pick out the bits you think you need.

The Python documentation, and the Django documentation have additional information in the text. Describing how a module or function works, what you can expect and how you should use it. Important information is often "hidden" in the text and if you're the sort of person that doesn't actually read the documentation you're going to get it wrong.

The Python docs are absolutely fine, because actually tries to teach you how stuff works and why it is the way it is.

I think the docs are often too verbose. Many pages start with a long winded intro that misses the main point. Some pages never get to the point. Docstrings are sometimes several pages long.

Example of not getting to the point: Extending/Embedding.

There is no fundamental difference. In both cases you're just calling a bunch of functions from libpython.a.

I like Django docs. They are lacking information in some places but generally about right. Python's docs are the opposite, they have too much information. I just want to remember the calls for using a reqex with capturing groups, and its about 5 pages of text into the document.

> if you're the sort of person that doesn't actually read the documentation

Is this a problem with the documentation, or the reader?

Perhaps it's just me, but I can (and do) skim the Python library documentation regularly. Due to the high level nature of Python, any given bit of code is doing a lot of work under the covers. I'd rather that complexity is properly documented than glazed over in the official language documentation.

This gap is very nicely filled by the Python Module of the Week series https://pymotw.com/3/ which focuses on examples rather than exhaustive docs. I believe the blog is coming out as a book called "Python standard library by example".

I've never encountered the "X requires unstated requirement Y, which fails to build because of obscure compiler error" issue, I would have liked if the author named X and Y.

However the documentation is totally a fair point. The "re" module is an excellent example - it starts off with a handful of relatively verbose introductory paragraphs and then dives straight into an extremely comprehensive list of special characters. The good news is that all the information you need is there somewhere but sadly it's hard to get there. Another commenter pointed out the Table of Contents, which for some reason my eyes just completely ignore by default. Maybe the font colour doesn't contrast with the background (aquamarine-ish on blue) or the line-spacing makes it tough. Certainly the numbered prefixes do not help, or the fact that the headings aren't actually particularly useful (sorry if you're responsible for the documentation, I like it overall I'm just searching for reasons why it's hard to read the side-menu).

I wish I had the time to help out, and the UX know-how to contribute to making it better. I'd feel awful if I pitched in and made things worse :(

>build because of obscure compiler error

Exactly what I had in mind, I never got such problems at least in installing libraries, how can that be an issue since we have pip?

yes, some badly maintained libraries will have this, but that's the library author's issue and not a problem with Python's pip, on the contrary pip is working exactly fine if it complains.

I tried installing OpenCV2. I needed to get Pip, then I tried to install OpenCV2, but it required a whole bunch of other stuff, which I had to compile, which failed.

Was a very frustrating afternoon.

If on Windows, google Christoph Gohlke's (spelling?) big list of pre-compiled Python modules.

How do I know to do that? Most projects just say "Install OpenCV" or, if you're lucky, give you the step-by-step for doing a pip install. Assuming you already have pip.

I remember the first time I sat down to really do something with python, step 1 was setup a virtual environment...

I never got to step 2 that week

>How do I know to do that?

The answer, unfortunately, is: you don't - at least not from the docs of all libraries you are trying to use. Maybe from some.

I found it by googling - so shared it, since it might help you.

A less than perfect system, I know.

Yes, I remember that installing pip itself was slightly tricky or involved, a while ago. I think they have made it simpler in recent versions, and it also comes pre-installed with some Python versions, likely 3.x and recent versions of 2.x.

Python's packaging system was and is still sometimes somewhat broken. However, it's getting better: compilation related issues are rarer nowadays thanks to the wheel pre-built package format (https://www.python.org/dev/peps/pep-0491/). Another great thing is the manylinux tag (https://www.python.org/dev/peps/pep-0513/) which allows to create portable packages which are compatible with many distributions + contain all required dependencies. For example, I didn't want to compile anymore OpenCV all the time by myself so I made it available as wheel package in PyPI: https://pypi.python.org/pypi/opencv-python

> I've never encountered the "X requires unstated requirement Y, which fails to build because of obscure compiler error" issue, I would have liked if the author named X and Y.

I maintain an open source Python project that tends to attract non-programmers or beginner programmers. This happens all the time. Case in point: https://github.com/BurntSushi/nfldb/issues/175

And that's a good case, because the error message is actually pretty nice compared to some others I've seen. But for a beginner user, learning to make the complete connection I wrote in my comment[1] on that thread is totally non-trivial.

[1] - https://github.com/BurntSushi/nfldb/issues/175#issuecomment-...

> I've never encountered the "X requires unstated requirement Y, which fails to build because of obscure compiler error" issue, I would have liked if the author named X and Y.

I've encountered this sort of thing plenty; usually it's not a python package that's required, it's a package you need to apt-get, and then another error pops up because that first one wasn't the only thing you needed.

The typical narrative of the mediocre duct tape engineer.

"I want to put something together fast by trial and error, without having to go through actual thinking or understanding. I don't care about details, just give me an example that I can copy, paste and duct-tape with another piece of code, so I can ship my product and charge some idiot a fortune for some spaghetti with meatballs code."

Documentation does not necessarily have to follow the format of a tutorial and can vary in levels of detail. Tutorials on this topic do exist. There, LMGTFY: https://docs.python.org/3/howto/regex.html

How did I find it? Google: regular expression site:python.org

Is it really wrong to learn something new with examples and without having to read a full book before typing the first few lines of code? I prefer to learn by trial and error.

It is a valid option to explore by trial and error. But once you have obtained something that seems to work, if you can't explain why it is working, it is a bad moment to stop.

If you construct an entire system by trial and error without an understanding of how and why it's working, you are exposing yourself and your customer to serious risks.

Even if things do not crash, if your code gets audited (e.g: when another person joins your team), you will have a bad time explaining your choices. As a professional engineer, it is expected of you to know what you are doing, and fudging code is the easiest way of getting sacked to the sound of a trumpet.

Right but the article was about a beginner learning a new language, I don't think a beginner is expected to have audit-proof code for quite some time.

I've been learning Python as it seems to be the preferred language in education at the moment in UK and I'm expecting my kids to be using it. I work in arts & craft but have a strong hobby background in computers.

I started with a Coursera course https://www.coursera.org/learn/python which was pretty easy (you can run the videos at 1.5 or 2x).

That course is based on a free gratis book by the lecturer (there a chapter on regex that covers re; with examples too).

I took a good few hours trying different IDEs and settled on PyCharm community edition, it has very easy doc lookup. (http://alicious.com/editors-ides-python-code/ is a brief summary of most of my IDE-for-python search.)

Currently I'm doing the Web Data course (using Beautiful Soup) and the only problem but is that by choice I'm using Python 3 when the course is written for 2. It covers regex pretty well.

I'm kinda surprised about the community argument. I've generally felt that people are nice enough, if a bit conservative. Though I guess this is the community that produced Python 3k, so something must have gone wrong

Especially to hear that someone went from Python to Ruby... I still feel raw for getting yelled at in IRC for trying to understand how blocks differ from anon. functions and about not doing things the "Ruby way"

I also found that part odd.

At least in stackoverflow, compared to R community, I found python community to be more helpful and more polite.

Agreed. The community is the part I like most about Python. I've largely stopped attending talks and such years ago, but still attend events every now and again (mostly the annual local conference) purely because I find the community friendly and interesting.

I feel a big problem with Python these days is that it's not one Python community. It really is becoming more and more fractured and efforts go into weird places.

For instance we have the core developers which do Python 3, then we have some separate volunteers that work on the packaging infrastructure which is largely based on cobbling systems on top of the horribly broken distutils/setuptools setup. For documentation people don't want to touch it that much because which version should it go in? Python 2 which everybody is using or Python 3?

This schism in the community is causing more and more issues as time goes on.

An odd article; the site is adware for a book, if the documentation was better he or she wouldn't have any sales!

2 out of the 3 negative points talks about how horrible the documentation is and yet I almost every day see blog posts, tutorials, books etc being posted online. If it really is so horrible why are these 'authors' apparently, not contributing to those horrible docs to make them better?

The gripe about packaging however is justified in a small manner, although we are way better than we used to be.

If the problem is fragmentation, there needs to be someone to project-manage the writing of the docs and, just as important, the user-testing of the docs. Much like the way the Django Software Foundation pays Tim Graham to be a steward of part of Django development, this probably means that the PSF should pay someone to be a steward of the user experience of the officially-sanctioned python documentation.

I am always astonished how the Django project accomplishes to create such a great documentation with their limited resources. Have a look at their fundraising page: https://www.djangoproject.com/fundraising/ .

> If it really is so horrible why are these 'authors' apparently, not contributing to those horrible docs to make them better?

Incentives (building their personal brand) is much more tilted towards blog posts, tutorials, books etc. than improving the standard documentation.

But if you're complaining primarily to further your personal brand (not because you genuinely want to see a problem addressed), why should your complaints even be taken with merit?

Once there was a discussion on HN about a specific command line tool, how easy command lines were and how even an idiot should be able to install and use it. I decided to test that theory by attempting to install it on a fresh copy of Windows, and documenting every step. I did get it installed, but did not succeed in using the command line tool for the desired application. There was also a number of difficult steps that nontechnical users would not know anything about and get frustrated by or turned away from. There were serious mistakes in the instructions. The biggest issue was dependencies which had further dependencies, all of which had their own difficult installation instructions. At least it didn't require compiling anything!

https://news.ycombinator.com/item?id=11453086

The trouble I had installing numpy, scipy and theano on python was over 3 days and only numpy & theano works.

3 days to go over the internet find out what the error... then trial and error.

Yes its difficult. And as the author says,"people get down voted"

Remember that Python is several decades old, and version 2 is about 16 years old. Like UNIX man pages, the docs (as well as best practices, idioms, etc.) were influenced by programming methods of the day. There was much more time to read the docs, think, and experiment. There weren't many answers to be found.

The methods of the modern developer are much different: 1) run into problem, 2) look for library to solve problem, 3) install library from package manager, 4) look for examples of library that fit problem, 5) copy-paste-modify example, 6) commit.

I can see why the docs are insufficient for today's beginners.

PHP is just as old and when I learnt PHP 14 years ago the docs were really friendly back then too, I loved being able to go to php.net/func_name and get a good description of how it worked. Of course this is for the standard library, which is what the article talks about, but PHP's standard library (at least then) had much more of the kitchen sink thrown in it.

If python documentation is bad then, god forbid, these poor users should try to use anything else such as node.

As far as technical documentation does, the python documentation is usually ranging from pretty good to very good.

I can't believe an advertisement has gotten so many up votes.

The only problem I had when learning python is that it took me a while to figure out that you have to put __init__.py in every folder. Easiest language I've ever learned and made projects with, and I had previously learned Java/C/C++/Erlang

1st problem, do pip install pip --upgrade in your virtualenv as the 1st thing. Another possible solution, install the corresponding .rpm/.deb package

2nd problem: go to this page first https://docs.python.org/3/py-modindex.html

3rd problem: yeah, kind of. Except for the logging module. Eff the logging module, it's a "java made in python" crap

Structlog FTW!

I only use the logging module these days when I've no alternative. The API for configuring logging is terrible.

>Installing libraries

I think pip and easy_install solves this problem. Or did I read the point incorrectly?

>docs are horrible

Also I never had seen the JS documentation of Mozilla, it is just beautiful. I have been using Python without any documentation related problems for that time, I actually am fond of Python documentation, for some reason it struck a chord with me.

The thing is, I learned Python by using the docs as the starting point, studied the tutorial and later the standard library, never really had a problem sorting through the docs ever. For everything there is a global module index, go to the a-z, click on a, go to the respective class/function and it has a good example, at least for the stdlib

I love Go language's docs more though.

Try "pip install pycurl". It'll fail. The solution is to install a C compiler and (on Ubuntu) run apt-get install libcurl4-openssl-dev. That's very different to other languages - they don't expect you to have a bunch of different development environments.

taking a quick look at the issue

the website says

PycURL is a Python interface to libcurl. and that would mean that we need libcurl installed. I am sure this is true with any other language as well, since this is a linker library, if that's a word.

also, surprisingly the website didn't mention that you need to install the libcurl4-openssl-dev :-)

so something is broken on this package.

Again, this is a package issue and not the issue of the language. We can find such examples in all languages. Doesn't mean Python's package system is broken or the language is broken.

I am waiting when the author will start teaching Go! just to read his "Problems everyone faces learning Go".

PycURL is a Python interface to libcurl. and that would mean that we need libcurl installed. I am sure this is true with any other language as well, since this is a linker library, if that's a word.

But this is just an example of the type of thing that is typical in Python using pip.

Anaconda is a bit better (see [1] for the details) but it is a real problem. It isn't uncommon to resort to Docker to work around this.

Pretending this isn't a problem for beginners (and even comparatively advanced users) isn't helpful.

[1] https://www.continuum.io/blog/developer-blog/python-packages...

I am curious, how do these libraries work for other languages, for instance Java interface to curl, how do they work around that fact?

This can be put forward as a PEP, I am not undermining the issue.

I am curious, how do these libraries work for other languages, for instance Java interface to curl, how do they work around that fact?

They typically ship a private precompiled library and link to without using the system LD_LIBRARY_PATH (or whatever).

Anaconda gets this mostly right - they ship precompiled libraries, but I'm not sure what the do with the linking situation.

This can be put forward as a PEP, I am not undermining the issue.

That isn't the most useful response for a beginner programmer.

exactly how many such precompiled libs do other languages ship?

pycurl etc.is not a part of stdlib, python or pip is not broken, this package's install mech could be broken

>That isn't the most useful response for a beginner programmer.

it was not for them, it was for those who are willing for a change and think this is a legitimate problem, jmdownvoting me win't make me shut up or solve the problem!

we can ask for a PEP since python is an Open Source project, noy sure if the author of this post tried that approach

That's not really true, lots of languages requires certain modules to be built from source with a C compiler.

Ruby and Node.js comes to mind.

Though in general the community behind them do a much better job of hiding this complexity from the user - to the point that most times you don't realize you've compiled it from source.

Worked fine for me (I had never used it before). Perhaps the problem is with your OS? Learning how to install dependencies should be one of the first things a programmer learns. I don't think it's unreasonable to learn that a message like "Error: X requires this library! Failed to install" means you should go and install that library. Plus it can be frustrating when something comes with its own version of a certain library because it makes tracking down bugs more difficult. For example Python ships its own version of libffi (the library it uses to call C functions). I've had issues with this library before, and it turned out the Python devs hadn't bothered to update it in a long time, because it "worked", but it didn't for me, and the newer version worked fine. So I had to end up trying to rebuild Python from source with a more up-to-date version of libffi, which was a giant pain. It may be "easier" to make everything self-contained like that, but it causes headaches too. Maybe it would be better in those cases if the library were just packaged in the distro's package manager? (Or brew for Macs). That would allow for more complete dependency handling.

Is there a good place to go for critiques on your project's documentation?

I'm currently working on updating the API docs for the company I work for, so we're paying people. However, what should open source projects do?

One thing I've seen that works poorly is encouraging new contributors should work on the documentation. The problem with this is that a new user only has knowledge of what they are confused by--not the broad understanding of the project that allows them to compose a document that teaches a coherent mental model. It seems like that can only be done by someone who is familiar with the project, but then they need a way to test that their writing actually makes sense.

I really like python, I use it all the time now and I don't start any new stuff in Ruby. But it was a struggle to get over the hump. I pretty much never use the Python docs, for the author's example, I would just google 'python array length stackoverflow'. I would only look at the official docs if something piqued my interest and I wanted to know more. Whereas with Ruby I go straight to chapter 4 of the pickaxe book and it answers the question 99 times out of 100. Also because its a PDF it works reliably on the train! I would dearly love to find a similarly written reference for python.

I'm not sure if this is what you're looking for, but Sphinx, which is used for generating the Python docs, is capable of producing docs in a number of formats, including ePub and PDF, and offline versions of the docs have been available for download in those formats ever since the changeover to Sphinx: https://docs.python.org/3/download.html

1) is really something that the maintainers of pip should deal with but unfortunately they don't see it as their problem.

pip's job isn't to specify dependencies, it's to resolve and install them. Specifying the dependencies is the job of setup.py, and that's not at all an unreasonable stance to take.

Really? Does it resolve and install libpq-dev or postgresql-libs when you try to install psycopg2? Or even tell you that you need to install them?

The problem isn't that this can't be solved in a consistent, clear way. The problem is that nobody views this as their problem to solve.

It also doesn't install Postgres itself, doesn't install a TCP/IP stack or any type of socket abstraction for communicating with a Postgres server, doesn't display an interactive SQL tutorial to ensure I'll know what to do once I get it running, doesn't install an operating system capable of running the psql client, and doesn't go out and buy me a computer or sign up a cloud hosting account with a provisioned database.

Would you fault it for those, too? Or would you find a point where you declare "not my problem", and then open yourself up to precisely this same criticism, just with the line drawn in a different place than where it currently is? I'm OK with the fact that the Python package installer installs Python packages. I'm OK with the fact that glancing at the documentation for psycopg immediately tells me up-front in its install instructions the things I'll need.

It also helpfully pre-empts the article here, suggesting a final debugging step for a non-functioning install:

Complain on your blog or on Twitter that psycopg2 is the worst package ever and about the quality time you have wasted figuring out the correct ARCHFLAGS. Especially useful from the Starbucks near you.

>It also doesn't install Postgres itself

Not a dependency.

>doesn't install a TCP/IP stack or any type of socket abstraction for communicating with a Postgres server

Because that's never caused a problem when installing psycopg2, ever.

>doesn't display an interactive SQL tutorial

Not a dependency.

>doesn't install an operating system capable

How exactly would you be running pip without an OS?

>Would you fault it for those, too?

I think you misunderstand what a dependency actually is.

I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager.

Nobody's done it because any potential solutions are much too brittle. On Debian derivatives, it's not too big a problem, but go outside of that, and even high-level tooling is a mess (the tooling on top of RPM varies from distro to distro, and even version to version). On macOS, if you're using a ports-like system, you might be dealing with fink, MacPorts, brew, pkgsrc, &c.

Any competently-managed project, OTOH, will list the dependencies for a number of major distros and OSs so that the person installing it will have an idea what to look for.

I'm all ears for a reasonable solution to this, if you have one.

I really don't know of anything like pip that can handle that case in any language

I've never ever had to think about it with the Java ecosystem. I just get a jar and stuff magically works.

If everything's native[1], whether you're talking about Python, Java, or any other language, you're golden.

The issue arises when you're calling out to something written in, say, C. You'll have the same pain in java if you start using JNI to call across the JVM boundary. It just crops up more in Python because C extensions get used a lot in Python.

Addendum:

[1] By this, I mean within the boundaries of the runtime environment and not using things like Python's C extension API or JNI. 'Native' is a poor choice of words.

It isn't really a question of native or non-native. It's more a question of "under the control of the package manager" vs. "not under the control".

You have a similar kind of pain when you have pip and apt trying to manage the system python installation.

If all you're doing is using pure Python code, there's no issue: you can install what you like, either with the --user flag or into a virtual environment. This situation is nothing like running pip as root.

No, there are issues. If you install packages that have dependencies on libraries which apt-installed packages depend upon then you can break them.

This is all about running code in an uncontrolled environment.

This is what virtual environments are for, and your solution still uses virtual environments.

So no, there aren't.

Now, I have a project I'd like to try with per-process mounts of dependencies using unionfs. It'd be a lot of work, and a lot could go wrong, but would solve a lot of problems.

>This is what virtual environments are for

Precisely. Virtualenvs solve part of the problem by isolating the python code and its dependencies - but not all of the code.

Apparently extending this behavior would cause security issues of an unknown nature...

My point was that I've never had to care if a jar contained C libraries or not. Some do, many don't, but I've never had to care. Maybe I've just been lucky.

I suppose it could have been a fat-jar thing: if the jar contains its non-java dependencies for my platform, then everything just works. If the native libs are separate, then sure, its a similar issue as described elsewhere here.

The solutions are only brittle if the different distribution systems doesn't talk with each other. Someone should gather a bunch of different language level package managers and a bunch of distro level package managers in a room and then not let them out until they have figured out a standardised API to declare, provide and require dependencies.

And who is that somebody going to be?

Also, what you're proposing boils down to asking them to all standardise on RPM. Have you asked yourself why all these different package management tools exist in the first place?

>I really don't know of anything like pip that can handle that case in any language, and I'm not entirely convinced that somehow resolving the plethora of naming conventions for packages between different distros (and even different vesions of those distros) is really the job of a language-specific package manager.

I actually started probably the only project to actually make a stab at solving this: https://github.com/unixpackage/unixpackage

However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this. Plus I know it would never make it into pip because "it's not pip's job".

>I'm all ears for a reasonable solution to this, if you have one.

The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself.

This will balloon the size of virtualenvs and increase installation times but it'll be worth it. Not only will it eliminate those horrible installation errors making everybody happy (not just beginners), it will eliminate a nasty source of brittleness (subtly different behavior caused by different versions of these packages on different platforms).

> However, I'm loath to continue working on it because there are few tasks more tedious, filled with annoying edge cases and thankless than this.

That, in a nutshell, is the problem.

> Plus I know it would never make it into pip because "it's not pip's job".

There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip.

> The only one I can think of is to packages like libpq-dev and libjpeg-dev and even the C compiler installable via pip and get the maintainers of pillow/psycopg2/whatever else to remove all their horrible hacky code to look for these dependencies on the system itself.

The 'horrible hacky code' is autotools 90% of the time, and while I'd like to see if die in a fire and for pkgconf to replace it as far as library detection goes, it's not going away any time soon.

Also, there is next to no chance of what you propose happening: you're essentially proposing a parallel system package management system, and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.

>That, in a nutshell, is the problem.

That problem can be solved with money. I've seen Jessica McKellar offer to shower PSF money over people who want to help make installation of python easier for beginners on Windows. I've not seen her offer to shower money over anybody wanting to solve this problem.

And, as I mentioned before I don't believe a solution like that unixpackage would actually make it into pip. They'd refuse it.

That, in the nutshell, is the problem.

>There are good reasons it wouldn't make it into pip: something like that is more the realm of setuptools, rather than pip.

Maybe. I'm never really sure where one ends and the other begins, to be honest. Perhaps that's part of the problem.

>you're essentially proposing a parallel system package management system,

I am, yes. And as ridiculous as that may sound it's utterly crucial.

>and at that point you'd might as well just propose that the Python packaging toolchain integrate, I don't know, maybe pkgsrc into itself.

That would be my other proposal, yes. Either pkgsrc, guix or nix.

>Also, there is next to no chance of what you propose happening

Want to know what's really shit about python?

No, this is a significantly bigger problem than simply getting things to work well on Windows.

Have you proposed this to anyone in the PSF?

> Maybe. I'm never really sure where one ends and the other begins, to be honest. Perhaps that's part of the problem.

In a nutshell, leaving out various details: pip downloads packages, asks them for their dependencies, downloads the dependencies, asks them for their dependencies, and so on, and so on. Then it does a topological sort and asks them to install themselves.

The process of installation is done by the setup.py file within the package[1]. That's setuptools.

The line between the two is pretty clear; this is similar to the split between dpkg/apt or rpm/yum, except setuptools itself has some built-in dependency resolution support.

[1] Assuming it's not a wheel.

> I am, yes. And as ridiculous as that may sound it's utterly crucial.

And a security nightmare.

> Want to know what's really shit about python?

Do you know what's shit about dealing with just about every language out there? Pick a language with its own package management, and the moment it steps outside of its own runtime environment, this problem crops up.

Want to know why I think there's little chance of this working? It's nothing to do with Python and everything to do this being a nightmare for sysadmins.

>Have you proposed this to anyone in the PSF?

I'm not interested in getting involved in their politics.

>And a security nightmare.

Absurd.

>Do you know what's shit about dealing with just about every language out there?

Not this. You might have noticed a variety of java programmers saying "nope, not an issue here".

>Pick a language with its own package management, and the moment it steps outside of its own runtime environment

The issue is not at all to do with stepping outside of the runtime environment and everything to do with running code in an uncontrolled environment. The solution is to control the environment as much as you can and fail cleanly if you can't.

>Want to know why I think there's little chance of this working?

No, I have no interest in what you think any more.

> I'm not interested in getting involved in their politics.

Then why did you even bring them up?

> Absurd.

Not if you're a sysadmin.

> Not this. You might have noticed a variety of java programmers saying "nope, not an issue here".

Because JNI is barely used. This isn't the case with Python, or Ruby, or a host of other lanuages.

> No, I have no interest in what you think any more.

Don't be an ass.

I see many people like python docs, and I agree, but the article talks about the difficulty for __beginners__.

That being said, maybe the python docs need something equivalent of wikipedia's 'Simple English' definitions.

Maybe by adding a 'simple' prefix (ie: simple.docs.python.org/3.5/library/re.html) that takes us to the very simplified version of the current documentation. Links to the in-depth docs are obviously necessary as well.

I don't think he's advocating for shortening the docs, if you look at the example he gave, his improved version was much longer: it explained each argument in detail as well as the return type, and gave an example.

The main problem with the python docs is not the verboseness. It's that they are entirely unstructured. If there just was a standard list of bold parameter - explanation and finally return value it would help immensely.

My own favorite is urllib.request.urlopen, the description mixes arguments, return values and exceptions in a blob of text two pages long. It's very confusing to read if you are a beginner just trying to fetch a file. The examples are way, way down on the page where you will never find them, the table of contents is so long you can't even see there is an example section unless you scroll down the menu.

On the bright side though, everything is explained in the text. It just less accessible than some other projects' documentation.

Here's a cached version (thanks, Google!) since the site seems to be down: http://webcache.googleusercontent.com/search?q=cache:3GQ0z6z...

>> I tried to install X. But when I ran it, it threw up a generic error, which meant (after Googling) that it needed Y. Okay. So I installed Y. But Y needed Z. And Z needs a C/C++ compiler. Okay, I’m on Linux, I have those.

If package maintainer does not specify dependencies, what can be done, really?

setup.py doesn't give a consistent mechanism to specify those kinds of dependencies.

How so? Any package in setup.py can declare a install_requires listing, including version matchers, of its (direct) dependencies. Those dependencies themselves should take care of listing theirs and so on completing the chain so that tools like pip can do their job.

That doesn't mean you can't run into a situation where a dependency cannot be resolved, but I don't see any problems around specifying them.

How exactly is pillow supposed to specify in a standardized way that it depends upon libjpeg and that libjpeg-dev is supposed to be installed on debian based distros?

I think the author meant packages output of pip, like this issue:

I want to install the pip package cryptography. I need to make sure Python.h is available. I need to make sure (on Debian) that lib-ffidev and lib-ssldev are available. This isn't said anywhere by pip (except Python.h, probably), and it's not the same on every system, let alone every Debian based system.

Ah, right, that makes sense. I don't think it's a fair criticism though, as you're crossing the boundaries between packages provided by your language ecosystem vs. those provided by your OS (distribution). There's no easy way to specify this in a way that would work on all platforms or you'd have to start adding logic to pip that knows how to apt/yum/pacman/port install those things.

It's the same if you look at Ruby, Node, even Go and many others. The second you need native libraries/binaries all you have to go on is the hope that the author has documented them along with the rest of the installation instructions. The current only way around it is hoping that your distribution has packaged up the library you need (and is up to date enough for your requirements) in their native format and declaring the required dependencies.

It's entirely solved with binary packages if you look at Java. You can even get a whole headless webkit (ui4j) setup from 2 maven packages without having to ever touch apt.

There's two ways around this:

* Provide a way to detect the package or lack thereof and fail in a consistent, easy to understand way. This part actually wouldn't be too hard.

* Provide a way to use the package via nix, guix or pkgsrc instead and enforce its usage so the OS package manager stops mattering. This has the additional benefit that you can specify versions (fairly important since different versions of those dependencies can have far reaching effects).

If pip can't find python-devel or a c compiler or whatever else, when it obviously knows that it's trying to run it, state that to the user and don't fail with "error 1".

0.02$ from a noob.

Much of this article should be taken with many, many large grains of salt.

I've never run into a package that I needed to install which both A) had mandatory dependencies and B) refused to specify them in a way automated tools would understand and resolve. I'm sure there are packages out there which have this problem, because a dependency list is not mandatory in setup.py, but I've not had the experience of encountering such a package and I've been using Python for a living for over a decade now.

oh please, don't compare python docs to php docs, php docs show one function per page, because their api is WRONG, python shows them often per module level, how to compose the api of a module and use them with actual examples, the doc is useful for experienced developers.

I feel this article is advertising their book, maybe the only point to consider is the somewhat broken status of pip that have to deals with system libraries dependencies, but whatever, I still use pip every day and with multiple production deploys.

Is there some good comment about python docs, packaging management, that isn't so much a rant?

This being said, I find the PHP docs' styling to be easier to read. The python docs are all over the place.

This being said, the PHP doc linked to in the article is 15% document, and 85% user comments (why do docs even have user comments?). Yuck.

While I understand the idea of not having user comments, back when I was doing php - and remember this was a time without StackOverflow - checking the docs user comments would get me going very quickly as some particular use cases would always show up there (and with improvements from other users too).

I'd suggest using ZealDocs or Dash if you want fast searchable documentation.

Regarding libraries, I often check unit tests for examples. But yeah, regex and similar docs suck.

I'd like details on the "installing libraries" comment. for python, most distros come with a package, otherwise compilation is rarer.

If compiling python, or even something like opencv that requires compilation on any random linux distro - yes, you need to be able to figure it out. linux distros aren't all standardised.

> I'd like details on the "installing libraries" comment. for python, most distros come with a package, otherwise compilation is rarer.

Installing python modules from a system's package manager is not something people should do. It causes more issues than it solves.

Can you go into more detail? So far, for my basic scripting needs, system packages have always been fine.

First of all, distro packages essentially always lag behind the released upstream versions, so you have to begin with figuring out which version the distro packaged, and find documentation for that, include that version in all your questions to the mailing list for the library, and hope that you're not on a version so old that it's no longer supported upstream.

Then there's the way distros like to make changes for "consistency" with the rest of their system. Command-line entry points get renamed or have options and behavior changed to reflect the distro's standards, but now the thing you're running isn't the thing upstream wrote or documented, so you're even more on your own if you hit trouble.

Also, it's not unheard-of for distros to take something that's a single packaged entity upstream, and split it into multiple distro packages, possibly without making it entirely clear that to get full functionality you'll have to install multiple distro packages. And, again, this is specific to the distro and their choices, so of course upstream won't document it and may be just as baffled as you are when you ask why something that's important to you doesn't seem to be in the package you got.

Finally, the Python ecosystem is built around the Python packaging and distribution toolchains. You're going to find plenty of expertise in things like using pip and virtual environments to develop and deploy Python, but all that goes out the window when you switch to using the distro package manager and packages. So you're giving up a lot of tooling built by and for Python engineers, for something which had to be a one-size-fits-all solution for the operating system it's bound to.

So while you can do all right with system packages and tooling for quite a few use cases, the dangers aren't that far away and are ready to hurt you, badly, as soon as you stray just a little bit outside of what the distro optimized for, or need help with a package beyond what the distro and its packagers are able to provide.

Hm okay, at least the first two things don't apply for me since Arch Linux updates usually within a few days, and likes to stay as close to upstream as possible.

This applies mostly to applications, which often rely on specific library versions which may or may not be available on the target system.

I'll go out on a limb and give general advice. I'm happy to receive constructive criticism in case I'm wrong.

The OS package manager is for system programs to administer the system. So if you have an admin tool it should run with `#!/usr/local/bin/python -E` (or wherever Python is installed). Note: It uses a hard coded path for `python` and it uses `-E` to ignore PYTHONPATH. This is so no matter how trashed your environment becomes, you can still run administration tools written in Python. The OS package manager should be installing programs that adhere to this policy.

Your applications that you develop should be using `#!/usr/bin/env python -S` This uses `env` to determine the correct Python version that your script should use. It also uses `-S` so it doesn't accidentally pull in an OS installed library which may have conflicts with libraries that you want to use.

People should do it in some cases because sometimes Python libs have to link against some other non-Python libs that are not part of the pip package.

Then if you just install the dependencies, sometimes the newer version in the PyPI requires a newer version of the dependencies, so you are stuck installing those things from source and building packages.

This is not true with Gentoo. With debian, ubuntu, or rhel - the vendor provided packages are frequently out of date.

Maybe instead of trying to teach others with a 'book project' he should try teaching himself and learn the actual language then he wouldn't have these kind of imaginary 'issues'!!

Has anyone made tldr pages for python packages?

Reply on news.ycombinator.com