Hacker News Reader

Comment by Yoric | Hacker News Reader

On the one side, yes Python is incredibly versatile.

On the other side, I have worked on many Python projects, some of them fairly high profile, and I have seen exactly two kinds of Python codebases:

1. a few were written by extreme professionals, plugging at every single hole, with ~100% coverage, plus considerable maintenance because every dependency upgrade tends to break something;

2. many that feel cobbled together, with undocumented (and often inconsistent) invariants "because something will eventually throw a TypeError if I make a mistake", undocumented metaprogramming, and heroic levels of maintenance because this stuff works until it doesn't.

Option 1. feels like "writing Python like it's Rust", but of course without any of the benefits of Rust either on performance or on safety.

Option 2. feels like "experiments running out of control". That is the unfortunate price of this incredible versatility.

I don't have a clear conclusion to this, except perhaps that Python is really good for the initial experimentation (when versatility actually gets you to initial results much faster), but really bad for... well, let's call it industralization.

View on HN

To be fair, a division between hell and heaven will happen with any language.

The question is: is this particular hell worth the result?

There is no generic answer to that of course, it just happens is has been the case for me during those 20 years.

First, you have to get to the industrialization phase. And of course, you have to get there, with the constraint of time, budgets, and talent.

Second, Python does have less benefits for that phase than rust or haskel, but it's not impotent. Industrialization is just hard, no matter the language.

Because we are an industry where a lot of self-taught people get a career, we tend to forget this last point. Creating a serious computing system is engineering, and this includes the project management part, and being rigorous about a lot of thing.

Granted, the rigor needs to be higher with Python once your reach that scale, and at a certain point (which I wish everybody would reach), you may want to ditch it. It's a good problem to have though.

Yet we have to remember not all industrialization attempts are equal. Most are really tame. I'd argue you could replace half the website code base out there with a bunch of bash scripts on a VPS and they would still make money. So even in this context, Python can deliver a lot.

This is roughly my experience, too.

Static type safety has interesting YAGNI characteristics. For the bits of the code that must work, and where defects and regressions due to type errors may be subtle and difficult to detect, it's indispensable.

But it can also be an impediment to iteration. Sometimes the code you're working on is still so experimental that you don't really know what the best structure and flow of data will be yet, and it's easier to just bodge it together with maps and heterogeneous lists for the first few iterations so you can let the code tell you how it wants to be structured. Having to start with static types subtly anchors you to your first idea. And the first is typically the worst.

Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment, and be selective in where I spend my industrialization tokens. Here's a list of the languages where I've found selectively rewriting the important bits in Rust to work well: C, C++, Python. I know which of those I'd rather use for prototyping, scripting, and high level control.

Relevant Fred Brooks quote: "Plan to throw one away. You will, anyway."

> Something I really like about Python for this sort of thing is that I have a lot more ability to delay this industrialization stuff to the last responsible moment

This is one of my favorite aspects of Python. I can start with every module in prototype form and industrialize each module as its design firms up. I can spend my early development getting an idea fleshed out with minimal overhead.

> I can start with every module in prototype form and industrialize each module as its design firms up.

That is definitely the theory. And this flexibility is indeed very precious for some types of work.

However... does it actually happen as you describe? I can count on half of the fingers of one hand the number of Python codebases that I've seen that actually feel like they've properly been reworked into something of industrial quality. All the other codebases I've seen are of the "I guess it works, maybe?" persuasion, quite possibly because there is always something higher priority than quality.

Something that just about seems to work is something that has often met the threshold for "solves a real problem" while also meeting the other requirements that a product needs to be successful.

It's easy to make something that works, is well designed, and either doesn't solve a problem someone has or nobody knows about it.

I realize that this is the common wisdom these days: write code that works just well enough that we have a chance to fix before it does too much damage whenever it breaks. I suspect that this approach is strongly fueled by the unlimited VC money available in tech, since it means that any company can employ an unlimited number of full-time developers (and PR) just to handle catastrophes.

We'll see how that wisdom holds if/when/as VC money dries up and/or moves to other sectors.

> I suspect that this approach is strongly fueled by the unlimited VC money available in tech,

Well, I suppose we could trade anecdotes and counter-examples, but my position largely comes from my own experience rather than received wisdom (though there's plenty of that).

Instead I'll just say that I disagree, largely because because a business is a complex and shifting arrangement of various factors competing for limited time and resources. Even in a software business, software is only one of those.

I definitely agree with your premises. Just not with your disagreement :)

Wandering offtopic, perhaps, but I've noticed that this kind of behavior seems to strongly correlate with Scrum.

The work starts getting rushed toward the end of the sprint. Every two weeks, people start furiously cutting corners to meet a completely artificial due date. And then there's basically zero chance that you'll be able to get the PO to agree to cleaning it up in the next sprint, because they can't see the problem.

Scrum of course prescribes all sorts of adornments you can add to try and counteract this effect. But I'm a firm believer that an ounce of prevention is worth a pound of cure.

Oh, interesting observation. Would you say it is scrum itself or just the existence of arbitrary deadlines?

I think it's the application of Scrum to work that doesn't primarily fall inside the (Cynefin) "simple" domain-type work that Scrum was designed for.

It's fine if the work is straightforward and easy to estimate. But, if it isn't, things get problematic. There are three variables that interact with each other when working on a project: time, scope, and quality. If you pin down both your acceptance criteria and the time you have to implement (which is basically what happens in a sprint planning meeting), then quality is the only remaining variable you have to manipulate when things aren't going according to plan.

The two week rush + everything is always broken failure pattern is solvable.

Given dubious project management schemes, do the refactoring and cleanup first. Then the functional change is easier to review as it's against a sane background, and there's minimal planned cleanup after in case that phase gets dropped. Call writing the tests characterisation if that helps.

The thing is type hints in Python are less a code quality feature and more a quality of life feature for developers. As long as I've got descriptive argument names and docstrings I can just tell you how to use a method. Your IDE can at least tell you argument names.

Type hints help reduce cognitive load when someone else (or you in the future) is trying to use some code. If you have strict type requirements you're testing that inside a method or with a decorator or something (and verifying with tests).

Even a big project can hum along happily without type hints. They're also something you can add with relative ease at any point.

> The thing is type hints in Python are less a code quality feature and more a quality of life feature for developers.

They are absolutely a code quality feature.

> As long as I've got descriptive argument names and docstrings I can just tell you how to use a method.

Yes, you can, but that doesn’t seem to be germane to the argument, since “it is possible to communicate intended use without typing” doesn’t support your QoL vs. code quality argument.

With typing, the type-checker can statically find potential errors in your description, or in my attempt to follow it—that’s a code quality feature. (Of course, that it does provide a description, and a better chance that the description is correct, is also a QoL issue.)

> Yes, you can, but that doesn’t seem to be germane to the argument, since “it is possible to communicate intended use without typing” doesn’t support your QoL vs. code quality argument.

Jillions of lines of quality Python were written before type hints. They're not strictly necessary for writing quality code. If you find modern code that's high quality it probably uses type hints but type hints don't automatically make high quality code.

> Having to start with static types subtly anchors you to your first idea. And the first is typically the worst.

Not just that, but dynamic typing conveniently allows you to try multiple ideas in parallel. In static languages, you tend to refactor The One Representation for a thing and try multiple ideas sequentially, which may or may not be better.

Of course, none of this is really inherent to the type system -- plenty of Python folks try various shapes for their data sequentially, and you can have multiple representations of the same data in a static language too. But I feel like the languages encourage a particular kind of experimentation, and sometimes either is more helpful than the other.

That is a very good point.

However, the other side of the coin is that in many Python codebases I've seen, people keep using multiple antagonistic ideas/paradigms in the same code, way past the point where it should have become clear that these ideas are actually incompatible and no amount of heroic hackery will solve the issues.

One of the core values in the Python community is, "we're all adults here."

I think, though, that a lot of Python programmers - particularly less-experienced ones - fail to realize that that typically functions as more of an expectation, perhaps even an obligation, than a liberty.

All this discussion is kind of interesting to me since in the beginning Python seemed to be focused on being the anti-Perl.

Rejecting "There's More Than One Way To Do It" in favour of "There should be one - and preferably only one - obvious way to do it" (IMO they did not find the right balance there in terms of python3 strings)

Ultra minimalism on syntax to the point of introducing invisible errors...

I suppose this greater leniency on approaches now is simply due to broad adoption.

> But it can also be an impediment to iteration. Sometimes the code you're working on is still so experimental that you don't really know what the best structure and flow of data will be yet, and it's easier to just bodge it together with maps and heterogeneous lists for the first few iterations so you can let the code tell you how it wants to be structured. Having to start with static types subtly anchors you to your first idea. And the first is typically the worst.

That's a very interesting observation.

Let me add a counterpoint, though. Indeed, you will refactor everything. But by having strong, static typing, you have a much clearer idea of what you're breaking along the way. Cue in hundreds of anecdotes by each of us, when we broke something during a refactoring because the dependency was not obvious and there were not enough tests to detect the breakage. I've seen two of these in Python code just this week.

That was what I originally thought, as someone who grew up on static typing and then migrated to Python.

But what I've discovered in practice is that, during those early iterations, I don't really need the compiler to help me predict what will break, because it's already in my head. The more common problem is that static typing results in more breaks than there would be in the code that just uses heterogeneous maps and lists, because I've got to set up special types, constructors, etc. for different states of the data such as "an ID has/has not been assigned yet". So it kind of ends up being the best solution to a problem largely of its own making.

I'm also working from the assumption here that one will go through and clean up code before putting it into production. That could be as simple as replacing dicts with dataclasses and adding type hints, but might also mean migrating modules to cython or Rust when it makes sense to do so. So you should still have good static type checking of code by the time it goes into production.

As someone who has primarily gone in the other direction, my anecdata supports the opposite conclusion: static typing tends to help me to prototype over more dynamic languages, even if it takes (slightly) longer to physically write things down. I think this is because of two things:

1. If I'm prototyping things, I find I spend a lot of time trying to figure out what sorts of shapes the data in my program will have - what sort of states are allowed, what sort of cases are present, what data will always exist vs what data will be optional, etc. If I'm doing that in my head, I may as well write it down at the same time, and voila, types. So I'm not usually doing much extra works by adding types.

2. If I change my code, which I often do when prototyping (some name turns out to be wrong, some switch needs extra cases, some function needs more data), then that is much easier in typed languages than untyped ones. Many times my IDE can do the refactoring for me, and if that isn't possible, I can start making the change somewhere (e.g. in a type declaration) and just follow the red lines until I've made the change everywhere. One of the big results of this is that statically typed prototypes are often immediately ready to be developed into a product, whereas in dynamic languages, the prototype already bears so many scars from refactoring and the natural back-and-forth that comes from prototyping, that it needs to be rewritten over more from scratch. (The corollary to that being that I have never once had a chance to rewrite a prototype before releasing it to production.)

I can imagine that some of this comes down to programming/architectural style. I tend to want to define my types up front, even in dynamic languages, because types and data are how I best understand the programs I work on. But if that's not how you work, the tradeoffs might not be the same. The other side is that the type systems I regularly use are almost exclusively modern ones, with decent support for things like product and sum types, so that I can use relatively simple types to model a lot of invariants.

> But what I've discovered in practice is that, during those early iterations, I don't really need the compiler to help me predict what will break, because it's already in my head.

Reading this, I have the feeling that you're talking mostly of single-person (or at least small team) projects. Am I wrong?

> The more common problem is that static typing results in more breaks than there would be in the code that just uses heterogeneous maps and lists, because I've got to set up special types, constructors, etc. for different states of the data such as "an ID has/has not been assigned yet". So it kind of ends up being the best solution to a problem largely of its own making.

There is definitely truth to this. I feel that this is a tax I'm absolutely willing to pay for most of my projects, but for single-person highly experimental projects, I agree that it sometimes feels unnecessary.

> I'm also working from the assumption here that one will go through and clean up code before putting it into production. That could be as simple as replacing dicts with dataclasses and adding type hints, but might also mean migrating modules to cython or Rust when it makes sense to do so. So you should still have good static type checking of code by the time it goes into production.

Just to be sure that we're talking of the same thing: do we agree that dataclasses and type hints are just the first step towards actually using types correctly? Just as putting things in `struct` or `enum` in Rust are just the first step.

> I have the feeling that you're talking mostly of single-person (or at least small team) projects. Am I wrong?

Small teams. And ones that work collaboratively, not ones that than carve the code up into bailiwicks so that they can mostly work in mini-silos.

I frankly don't like to work any other way. Conway's Law all but mandates that large teams produce excess complexity, because they're basically unable to develop efficient internal communication patterns. (Geometric scaling is a heck of a thing.) And then additive bias means that we tend to deal with that problem by pulling even more complexity into our tooling and development methodologies.

I used to believe that was just how it was, but now I'm getting to old for that crap. Better to push the whole "expensive communication" mess up to a macro scale where it belongs so that the day-to-day work can be easier.

> Small teams. And ones that work collaboratively, not ones that than carve the code up into bailiwicks so that they can mostly work in mini-silos.

Well, that may explain some of the difference between our points of view. Most of my experience is with medium (<20 developers) to pretty large (> 500 developers) applications. At some point, no matter how cooperative the team is, the amount of complexity that you can hold in your head is not sufficient to make sure that you're not breaking stuff during a simple-looking refactoring.

Sure but at that point we're probably not on the first iteration of code anyway. Even at a big tech company, I find it most effective to make a POC first-iteration that you prove out in a development or staging environment that uses the map-of-heterogeneous-types style development. Once you get the PMs and Designers onboard, you'll iterate through it until the POC is in an okay state, and then you turn that into a final product that goes through a larger architecture review and gets carved up into deliverables that medium and large-scale teams work on. This latter work is done in a language with better type systems that can better handle the complexity of coordinating across 10s or 100s of developers and can generally handle the potential scale of Big Tech.

There's something to be said that the demand for type systems is being driven by organizational bloat but it's also true that large organizations delivering complex software has been a constant for decades now.

Do you work in an organization that does this? Because most organizations I've seen who don't pick the approach of "write it like it's Rust" rather have the following workflow.

1. Iterate on early prototype.

2. Show prototype to stakeholders.

3. Stakeholders want more features. At best, one dev has a little time to tighten a few bolts here and there while working on second prototype.

4. Show second prototype to stakeholders.

5. Stakeholders want more features. At best, one dev has a little time to tighten a few bolts here and there while working on third prototype.

6. etc.

Of course, productivity decreases with each iteration because as things progress (and new developers join or historical developers leave), people lose sight of what every line of code means.

In the best case, at some point, a senior enough developer gets hired and has enough clout to warrant some time to tighten things a bit further. But that attempt never finishes, because stakeholders insist that new features are needed, and refactoring a codebase while everybody else is busy hacking through it is a burnout-inducing task.

> Do you work in an organization that does this?

Yup! I'm at a company that used to be a startup and ended up becoming Big Tech (over many years, I'm a dinosaur here.) Our initial phase involved building lots of quick-and-dirty services as we were iterating very quickly. These services were bad and unreliable but were quick to write and throwaway.

From there we had a "medium" phase where we built a lot of services in more strictly typed languages that we intended on living longer. The problem we encountered in this phase was that no matter the type safety or performance, we started hitting issues from the way our services were architected. We started putting too much load on our DBs, we didn't think through our service semantics properly and started encountering consistency issues/high network chatter, our caches started having hotspotting issues, our queues would block on too much shared state, etc, etc.

We decided to move to a model that's pretty common across Big Tech of having senior engineers/architects develop a PoC and using that PoC to shop around the service. For purely internal services with constrained problem domains and infrequent changes, we'd usually skip this step and move directly to a strictly typed, high performance language (for us that's Java or Go because we find them able to deal with < 15 ms P99 in-region latency guarantees (2 ms P50 latencies) just fine.) For services with more fluid requirements, the senior engineer usually creates a throwaway-ish service written in something like Node or Python and brings stakeholders together to iterate on the service. Iteration usually lasts a couple weeks to a couple months (big tech timelines can be slow), and then once requirements are agreed upon, we actually carve out the work necessary to stand up the real service into production. We specifically call out these two phases (pre-prod and GA) in each of our projects. Sometimes the mature service work occurs in parallel to the experimentation as a lot of the initial setup work is just boilerplate of plugging things in.

===

I have friends who work/have worked in places like you describe but a lot of them tell me that those shops end up in a morass of tech debt over time anyway and eventually find it very difficult to hire due to the huge amount of tech debt and end up mandating huge rewrites anyway.

That's nice! Feels like your company has managed to get Python to work well for your case!

Most of the shops I've seen/heard of don't seem to reach that level of maturity. Although I'm trying very hard to get mine there :)

>impediment to iteration

The "aha!" moment that converted me from a Clojure guy to a Haskell guy was realizing that types aren't an impediment to iteration, they are an enabler of rapid design iterating. Once written, code has a way of not wanting to be changed. Types let me work "above the code" during that squishy beginning period when I'm not sure 'what the best structure and flow of data will be'. Emotionally, deleting types is a lot easier for me than deleting code.

This is in no way saying that types are the One True Way™ ^_^ Just that I've found them, given the way my brain is wired, to be a great tool for iteration.

Yeah, types are a type of (usually) easy to write, automatically enforced, documentation which I find endlessly useful during experimentation, if sometimes constraining. I'm sure that there are other means to achieve similar results (some variant of TDD, maybe?) but I haven't experienced them yet.

Wholeheartedly agree. I'm a self taught programmer who programs to get things done rather than just doing programming for fun (I started in ML/data science field). That way, my code often resembles the caricature of these hacky codebases alluded above. And I'm well aware of it.

I often have arguments about utility of Python with my more technically solid engineer friends who keep telling me "Python doesn't scale" etc. And I often come back to the same point you highlighted: is the particular hell worth the result? For 85-90% of the cases the answer is resounding yes.

For nimble teams, startup projects, internal tools : most of the code is often thrown away eventually. Python's beauty is that the language quickly gets out of the way. Everyone then is forced to focus on complexity of the business domain, "does the code solve the business problem". The architecture/scale/speed problems will eventually arrive with Python. But most purist engineers overestimate how soon it will come. At most of the failed startup attempts I was part of, the PMF problem was more pressing to solve than software constraints.

Early days of YouTube, Dropbox, Instagram (and maybe OpenAI too) are big testament to this. I've made my peace these days by not fighting to prove Python is the best language etc. If someone tells me "Rust beats Python" kind of argument: I say I agree, wish them luck and focus on shipping things.

tl:dr; Python is still the best choice to "quickly deliver value and test your business hypothesis".

Python scales fine as long as you know microservices.

The issue is people assuming that they don't need to learn anything new to use Python.

You got people stating that using static typing in a dynamically typed language like Python is a good or reasonable idea. It's not.

But people don't want to put in the effort to learn things like dynamic typing and microservices.

How do microservices factor in this conversation? Do you find that they actually reduce complexity?

I think you have to grade code in your project, i.e not every piece of code has the same value, high value => more strict, low value => less strict.

Less strict code is something that you can throw away and rewrite without too much effort.

This has the implication that you have bulkheads (like in a ship) to separate high value and low value code, thus you have to avoid code that is toxic (i.e magic) that spreads between them, e.g. ORM entities.

When using the type system correctly you don't need 100% coverage, now I'm not a Python programmer so I don't how feasible that is with that type system, but in PHP I use the type system extensible and I don't write that many tests anymore, except for legacy code or when I need to test something specific, like an algorithm. By using the type system you can catch most programmer errors, and combine that with routinely throwing exceptions for any unwanted state, you will detect most bugs.

The way unit testing has been interpreted and implemented it has become more of a nuisance than a help in fast moving projects. Usually what happens is that the team refactors some unit and then multiple test breaks and then time needs to be spent rewriting these test, but what that means is that the tests were actually useless after they where committed if you can change them as you like, when you change a test you are no longer testing the same thing.

Thus tests can also can be graded, from high value tests, i.e stable, and low value tests. Depending how you organize your code, high value tests will usually test a larger set of functionality in one run, like a system (or integration) test from call to finish, these should never change regardless of much you refactor. High level tests should only change when your business logic changes.

Low value test typically involves a lot of mocking and stubbing, those are pretty much pointless to commit after you implemented your unit. Too much mocking and stubbing is code smell.

This is my experience as well. There are very few code bases with 100% coverage and strict development practices. Moreover, if the average developers get their hands on one of them, they'll degrade soon.

The problem comes from the very "top", since CPython developers are in camp (2) and celebrate their development practices.

Other languages also have this divide of course but not to the extent that Python has.

As for maintenance: This is also true. I have rewritten a Python code base in C++. Even in C++ it is easier to maintain, because the C++ compiler does not break between versions and there are no new bugs. And modern C++ can look quite like Python.

>Option 1. feels like "writing Python like it's Rust", but of course without any of the benefits of Rust either on performance or on safety.

...but with the benefits of Python. You might not see them/need them but they are there :)

Fair enough.

I would like to see a codebase of the first kind. All larger python projects I have seen are of kind 2...

The Firefox build system feels to me like kind 1. The Synapse Matrix server also feels pretty close (although I don't understand why it doesn't use Pydantic or something like Pydantic).

I think this is a form of selection bias. Certain other languages wouldn't even allow a project like your point 2. to get started. In Python, the bar for something to start kind of working is quite low, even if you're a beginner, a scientist of another discipline who needs a little script to manage some data, a teen who's learning to code, etc. So of course you see a lot of half-baked stuff. It's important we distinguish between 1. and 2., but 2s might sometimes be better than nothing.

Absolutely. In some domains, the choice is between 2. and nothing – and 2. is better. That's true for Python or JavaScript, for instance (in each their domain).

I believe that we agree that there are actually many domains in which you really want 1., and if you can't have it, then "nothing" is generally preferable to 2.

How are any of these points exclusive to Python among the commonly used languages?

Everything you said applies equally to many C++ projects I work with as well

You are right, this is probably not specific to Python. It just happens that:

1. we're currently discussing Python;

2. anecdotally, the C++ projects on which I have worked were all case 1. ("code it like it's Rust") – I make no claim that this is representative, though.

I see exactly this with Ruby. In that regard, Ruby and Python too, are very alike.

Reply on news.ycombinator.com