Ask HN: How can I get better at writing production-level Python? | Hacker News Reader

Ask HN: How can I get better at writing production-level Python?

63 pointslegobridge53 comments

I'm looking for resources that could help me write better, more professional code in Python.

I've worked extensively as a software developer in Java before, so I am aware of the many things that go into writing "real" code (as opposed to what one writes for Leetcode or even an academic project), from little things like using the right annotations, to bigger ones like dependency injection frameworks.

I've used Python for Leetcode, scripting, isolated ML work, writing little games, etc., and I've read and practiced much of PEP8, so it's not that I write bad Python code. But I do feel that I could be doing so much better (since I've done better in Java) and I'd like to get to that level of proficiency as soon as I can.

In that vein, I'm looking for any resources that have worked for any of you and you think would be suited for me. Thanks!

View on HN

Assuming you have the fundamentals down, I’d recommend taking the time to dig in and understand the source code of a couple well written packages. Start with some simple libraries like datetime, requests, flask and move on to more complex, modern examples like pandas / fastapi.

Clone the package, run the tests, break the tests and try adding functionality. You’ll learn a lot - I know I did when I was starting out.

I’d also recommend checking out Fluent Python.

This is great recommendation, you can get very far by reading well written code. These have very well written Python, exposing much of the language expressiveness:

- https://github.com/python-attrs/attrs

- https://github.com/mahmoud/glom

- https://github.com/pytoolz/toolz

- https://github.com/Suor/funcy

- https://github.com/dabeaz/curio

  ps: dabaez has great educational content on writing idiomatic python, definitely worth checking out

An honorable mention of norvig's classic essays, which got me into python while I was in college, over 12y ago: https://norvig.com/spell-correct.html.

Aside from reading code, _writing_ something that you know it exist (e.g glom) and then comparing it to how others have done it is also a great learning experience.

Yeah, reading the Fluent Python book[1] and / or following along with their support files[2] would a good way to start

[1] https://amzn.to/3J48u2J

[2] https://github.com/fluentpython/example-code-2e

Just fyi to cheapskates like me, i was looking around for a free pdf, some are hosted on github, others on google docs. Putting the pdf urls into https://www.virustotal.com/gui/home/url , all were suspicious or malicious. I think ill search around a bit more, but will probably give up.

Might be of some interest to you: https://annas-archive.org/

Unfortunately it seems pdfs linked there may also be malicious ( https://www.virustotal.com/gui/url/7e73cc3d685c77d88ad9cf769... ). I checked on some other pdf links ive downloaded previously and they were safe, so I don’t believe the software just flags everything.

But it could be if the coding book includes working examples and exercises, that would probably flag any system looking for code execution.

I'm inclined to say that those are false positives, given the obscurity of the vendors who are flagging the URL.

Second or third reco for Fluent Python. Also, to be a purist for a moment. Explicit is better than implicit. Use this as a razor to decide all ties in your design.

FastAPI, while a very impressive codebase, is not approachable at even an intermediate proficiency in Python.

It is _staggeringly_ dynamic, and basically uses runtime reflection of "static" types to scaffold a really complex ast describing your application.

Fluent Python is fantastic.

Avoid writing java in python.

Don't stretch inheritance where where they are not needed - avoid factory classes unless you know for certain that it's called for.

Use pythonic stuff like @decorators and enjoy functions as first class objects.

Finally, try to avoid using an IDE. This keeps your files and folders structures simple and organized out of necessity. In Java it's almost impossible, but it's very possible in python as it removes so much verbosity.

> Finally, try to avoid using an IDE. This keeps your files and folders structures simple and organized out of necessity.

A good IDE will keep that structure simple and organized, too. The problem with python is more that its dynamic, duck typing type system hinders some of the great benefits that "modern" (as in, from the past ~2 decades) IDEs bring. Since the IDE can hardly infer any type, and since even the ones it could infer can dynamically change in shape at any time, the IDE cannot provide as helpful suggestions and as powerful navigation as IDEs for other languages can.

For the same reason, compile time almost exclusively tells you about crass syntax errors only, which further diminishes an IDE's helpfulness.

I'd be curious to know if there are python IDEs who integrate with mypy (or the underlying static typing PEP) to bring some of the lost magic back to python IDEs.

> "Since the IDE can hardly infer any type, and since even the ones it could infer can dynamically change in shape at any time, the IDE cannot provide as helpful suggestions and as powerful navigation as IDEs for other languages can."

I've felt this pain a lot. I finally took the time to learn how to make use of the type hinting that python does offer, and now I write those type hints without even thinking about it. It makes the development experience so much better for me and across the team, since we've all bought in. It doesn't do everything. It is kind of lipstick on a pig. But it's worth doing.

About using type annotations in Python: Do you get any push back from your team? I do. "Oh, it clutters the code." I don't know how to reply. And, they continue to write complex code without type annotations. It is hard to reason about. It is a never ending battle with people who come from weaker typed language. They struggle to see the value of stricter types.

Sure, a little bit. It is a change in peoples' workflows after all, and a lot of developers are pretty choosy about that. But I think you can overcome that if you make a respectful case.

I think you are spot on that people coming from weakly typed languages often don't realize what they're missing. "It clutters the code" is really just "I'm not used to seeing it that way" so if you can get people to try it out (maybe in a well-cordoned area of the code base or in some new dev) they will find out for themselves that it saves time and grief in the end.

Thanks for your reply. Unfortunately, your strategy has not worked for me. The mostly Python people are me continue view types as a burden, not a benefit.

I feel a similar way about typescript. It outsource the mental complexity into compile stage instead of directly on runtime.

100% yeah totally agree, I feel very similarly about TS (and can barely stand vanilla JS at this point)

Pycharm does, and so does any other jetbrains IDE with their python plugin installed. It's a large part of the reason I bother putting type annotations in my code, to help my IDE help me.

Good to know, thanks! I've warmed up to python again when I discovered mypy, having successfully used it for a personal project with rather complicated types (async and thus futures were involved, over more basic types but with complicated shapes). Knowing that there is also adequate IDE support makes it even more of a candidate again...

Personally, without considering languages, I think IDE subconsciously cause complexity and increase entropy for a project. When things are at the a single key jump it's easy to insert code wherever and eventually build a jumbled mess. I've seen this in too many projects. Going without IDE causes a short term pain but eventually the organization gets there and it speeds up because the code is simply better organized and where to look for things are either imprinted or self evident.

That may work for small projects[1], but for large projects you quickly run into a limit where manual navigation and lookup become very prohibitive to one's ability to efficiently write code.

Even then I'd rather avoid cumbersome navigation and typing (the fingers on keyboard kind this time, not the type system kind), as well as catching a lot of problems only in the compile step (if existing) in the best case, or at runtime in the worst case (common in python without static typing extensions).

[1] Or, I guess, projects with a small team, where all members are familiar with all or most of the code base.

My opinion is manual navigation is O(log(N)) for well organized projects. Which scales very well - especially as the package should be optimized for local changes/split into dependencies when it gets above a certain size. For a project that's not organized at all, it's O(N) which means it will soon become untenable.

Most project is somewhere in between but from what I've seen it tends towards the latter. As there are overwhelming amount of people utilizing the ease of IDE and do not care at all about the structure, or maintenance of the structure of a package.

I'm talking about pretty large projects. Try navigating, for example, an operating system kernel for its very many functions you encounter just when writing a device driver. You won't know where to find everything (most things maybe even), so manual navigation is certainly not O(log(N)).

Unless you want to mindlessly click through directory structures and files hunting for vague hints, what are you going to do? Full text search? Unless you're both careful and lucky, you will find a lot or all of the call sites as well. Which, again, gets worse the larger the project (and thus the number of call sites) gets.

Then, you better remember the arguments' type, order, and significance of the many functions you call, otherwise it's back to the navigation I've described above every time you want to use it.

This was one of the more extreme examples, but the whole problem starts to rear its ugly head pretty early in much smaller projects. The issue is however somewhat masked if you are the only or one of very few developers with high familiarity of the project. Of course then you know which file to click on and where to scroll to.

But then, even on a small project, that project rarely exists in a vacuum (the OS kernel example above is actually almost somewhat of an exception). You will use other frameworks and libraries, at the very least the language's standard library, usually, which is effectively a big project again.

> what are you going to do? Full text search?

Actually yes, I often have a grep tab open on the next terminal tab (whereas vim lives in the other one). Granted, I don't use it nearly as often as one would CTRL-B in intellij. grep (and other GNU utilities) is very powerful and can be used for more than just text searches. In fact for even project I develop using intellij I often find myself doing the same grep window in remote just because it's faster and I can make manual edits on the fly.

For navigation, I usually to tabe or vert sb and this allow the other call signature to simply exist on my desktop real estate or a gt away until I don't need it anymore.

You might feel like I'm simply proving your point, and you might just be right. But I do use the same method of editing on larger project outside of personal space, and delivers at speed if not just due to the fact that I need very minimal setup (the context switching is hefty when the package space is littered with various packages and not just a single one that you change all the time). I do use IDE for certain languages (yes, java), but I think it has more to deal with the verbosity and inherited complexity (chicken and egg here) then I would attribute to size alone.

The bottom line is, having been using IDE and not IDE for different projects for a long time, I would default to not using IDE if I could. It's not like I'm pathologically averse to the idea of IDE, in fact I use it daily. I just think not using the IDE has benefits both on speed for certain projects, and clarity and skill growth for others.

I still have nightmares of a previous job where there was a “god” module with a single file containing dozens of classes, many similarly named, over 7000 lines long.

Pycharm does a good job

    enjoy functions as first class objects

I roll my eyes every time I hear this trope. What do Python functions have that Java functions do not have? To the average developer, a lambda that maps to the Runnable interface _is_ a first class object.

    try to avoid using an IDE

Ugh. More purism. So now I need memorize more things? No thanks. The IDE can do that for me. Every trivial fact that I need to memorize takes a space in my mind that could be used for more valuable information, including other programming languages.

> a lambda that maps to the Runnable interface _is_ a first class object.

A simple python function also is. Just much simpler. On principal, every turing complete language can do everything that you make it do, including emulating concepts that are more natural in other languages - you can write OOP in C if you want to build the whole scaffold to support it.

I also happen to know quite a number of languages, up to 10 if we talk about everything I've developed something in, but more like 5~ if we talk about ones that I'm familiar with, this includes java and python. imo programming should be language agnostic - don't try to remember the syntax, all of it is available in 10 second via googling, but remember data structures that are common across all languages and specific useful patterns and choose the right one for the job.

Both python and java have their own advantages, and own ways of doing things. But writing python like classic OOP java results in hot garbage. It defeats the purpose of using python, should just use Java if one want to emulate all of that polymorphism and OOP patterns, at least the language is built for it.

Inheriting a python app written by java developers was one of the more frustrating code bases I've worked with

If you want to see an exercise of writing Java in Python look no further than the logging module source.

Avoid writing java... (fixed it for you)

Try new things. Simple over everything is the key to production python code. Too many layers of abstraction is a massive footgun, especially if youre passing objects around a lot, due to dynamic typing etc.

Constantly circle back and refactor. Ruthlessly.

Same as in any lang, focus on data structures and algorithms (business logic is algorithmic) rather than the implementation, at least to start with. Get it working (usually easy in python, it's so dynamic!), make it good, make it fast. A good data model makes coding a joy. If your data structures suck every little thing will accrue friction. This is a red flag that you chose the wrong data model.

Source: 1000s of python hours in engineering and research roles

RealPython.com

Learn all about pytest and how to use its fixtures well.

Mypy for use in conjunction with type annotations for static analysis.

Packaging: Python-poetry.org

Personally, I recommend Deal (design by contract) and/or Hypothesis (property-based testing) libraries, too.

Controversial opinion: Stay away from Flask and all of its derivations. That framework is badly designed. Learn from Django instead.

Another great course is David Beazley's Advanced Python Mastery; he just put it all up on github (PDF of all slides + exercises) https://github.com/dabeaz-course/python-mastery

It's designed as a four-day workshop. Lots of material around 'mature' Python code

> Stay away from Flask and all of its derivations. That framework is badly designed. Learn from Django instead.

I couldn’t of agree more. I’m a Python veteran, but hasn’t done web dev before a few years ago. I picked Flask for one of our web apps and now very much regret it. It feels like it fights a lot of the direction modern python is going. It’s also encumbered by an ecosystem of bloated and under-maintained extensions that only make it harder to use.

Regarding poetry for packaging, I’m 50/50 on it. The product is interesting, but I find myself almost always reverting to a structure where I have requirements managed by pip-compile. It just feels a lot cleaner. I also had a few unpleasant interactions with the poetry devs with regard of GitHub issues, which makes it easier to obviate on a personal level.

Going through Trey Hunner's Python Morsels [0] exercises is worth your time. Each exercise starts by describing desired outcomes and it's your job to make the tests pass. The accompanying screencasts are just the right length and level of detail. At the end you can compare your code to his. It covers all the right ways to write python.

[0] - https://www.pythonmorsels.com/

Anything like this for golang?

Start reading production level Python.

I mean, so many people want to write production-level code and yet never took the time to actually read the production-level code right in their faces!

Even the standard library is worth seeing. Next time you import pathlib.Path, right click it, select "See Definition" and go find out how the sausage is made.

Obviously you are not expected to understand _everything_. But you will be surprised you will understand a bit. And then a bit more. And you will start getting comfortable dealing with production-level code. Soon you'll start writing it yourself.

This little habit skyrocketed my Python game

The rule of writing better python is to forget most of what you have learned writing Java. Avoid boilerplate code . Avoid deep nested classes like factory.nextfactory.nextnext factory . Avoid premature over engineering abstractions .only use class if you have to . Make it simple. When you can achieve 20 lines of easy to read python code that probably need 200 lines of Java code that you used to write , then you succeed.

Use mypy if you can, and then as much as you can. python's dynamic type system is not only a pain for debugging and for catching problems (before they arrive as cryptic runtime errors way too late in production), it also makes it hard for other developers to know how data is shaped, and what functions and methods are doing without deep inspection.

mypy brings some of the sanity back.

Improving your Python skills for production-level code involves practice and continuous learning. Participate in open-source projects, review high-quality code, and read Python best practices. Collaborating with experienced developers and seeking feedback can help refine your skills. Stay curious, be patient, and celebrate progress along the way!

Some off-the-cuff tips, at least --

TDD is very easy with Python because the unit testing framework is built in -- I'd suggest writing tests for just about everything you do.

Additionally, the typing system is expanding all the time. Make sure you're adding type annotations where / when you can; even lightweight ones like TypedDict help.

When I used Python for Advent of Code a few years ago, I found that Python makes unit testing fun.

Coming from .NET 6 development, I'll do what I need to do when it comes to fleshing out unit tests, but the brevity of Python unit tests was real pleasant to work with.

Not a learning source per se, but a good tool to learn "best practices" is using ruff [0] as your linter.

I enabled nearly all the rules it has available. And I've learned so much from it.

[0]: https://beta.ruff.rs

A classic: https://www.youtube.com/watch?v=wf-BqAjZb8M (Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code - PyCon 2015)

All of Hettingers python talks are pretty great.

Write Python code and get feedback from a colleague.

If you don't have a colleague, GPT-4 is an acceptable substitute.

Learn good style (if you aren’t given one the google style guide is pretty common)and unit test your code. At google they call writing good code “readability”. When you get “python readability” your code is readable. Which makes it easier to understand, debug and maintain.

Type annotations, mypy, lots of documentation.

Where python falls down is the flexibility - if you aren't careful it's write only.

Search the channel of Arjan from arjancodes.com on Youtube. Watch his videos.

I find his stuff is oriented more to beginners. I'd recommend mCoding instead perhaps.

Will have a look at mCoding. But I think Arjan's videos are on an intermediate level.

Reply on news.ycombinator.com