Opaque Types in Python | Hacker News Reader

Opaque Types in Python

blog.glyph.im

131 pointslumpa58 comments

You're holding it (Python) wrong. Python OO was a counter reaction to the bondage and discipline that languages like C++ had with private members and protected inheritance.

If you have members that users probably shouldn't touch, you prepend them with an underscore. This is just a hint; It doesn't actually change anything. We're all adults here and we know the consequences of reaching into implementation details.

I agreed with this 100% for a long time. Then I started working on a library at $WORK with dozens of downstream users abusing the hell out of my idiomatic underscore usage, especially in the context of lazy tests with folks writing endless mocks. When I’d “break” their test suite (blocking some time sensitive release) I’d get all kinds of shit. But _they_ were breaking the contract. Unfortunately I had little (if any) control on the path of application code making it to production (yeah yeah not great engineering org, but it’s the world I lived in). Strategies like this post would be helpful for said situations.

Something similar happened to me. I told those groups to pound sand because they knew they were relying on something which they should not. Manager had my back, they whined a lot but they had to change and improve their processes.

There’s always the extra idiomatic __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED for coworkers that can’t take a hint.

https://github.com/reactjs/react.dev/issues/3896

You can technically "enforce" this at runtime with __getattribute__ or decorators[1].

Maybe be evil and add 1000 "Private access not allowed: {name}" with 1 second delays between each.

Aaaand, this flexibility is exactly why python is slow.

[1] https://pypi.org/project/accessify/

My secret for test-only code is to provide functions whose first argument is a value that only exist in a test scenario. There are also static analysis rules for the linter in my language of choice that disallows test-only functions being called outside of test files. This gets you closer to "we're actually adults here" but still a tiny bit cheatable.

I'm torn. On the one hand, this is not too uncommon of a problem to run into. On the other, poor practices from coworkers are not going to go away thanks to a language filter.

So, the question will come down to which causes more grief, people abusing this convention, or people that overly use the language features that combat it? It is the standard optimization question between poor practices and enforcement that you have in any question of enforcement.

I would be delighted if we could get some empirical data on this.

I generally agree with you (and disagree with the parent comment) that I think this seems like a useful technique. But just note that if someone really wants to break the contract, they can just use the "private" class. Just like "private" members, it is only private by convention.

> Then I started working on a library at $WORK with dozens of downstream users abusing the hell out of my idiomatic underscore usage [...]. When I’d “break” their test suite [...] I’d get all kinds of shit

I can understand this happening once. But, after that, management should have been involved and addressed the root problem. It would be no longer a technical or communication issue, but a project management one.

> We're all adults here and we know the consequences of reaching into implementation details.

I wish you were right but, IMHE, it requires a lot of communication once teams grow and many team member do not fully understand the consequences of what they do. It is nice to have something that helps when reviewing code.

> If you have members that users probably shouldn't touch, you prepend them with an underscore

Well, this is precisely what TFA does. It prepends the constructor with an underscore.

I think you missed the issue at hand:

> even if you keep all your fields private, the constructor is still, inherently, public.

ShippingOptions and the literals / enums are part of the public API, so the user would just be writing

    ShippingOptions(Carrier.USPS, Conveyance.Air)

with no hint that they're doing anything wrong.

Dataclasses do have a `kw_only` option, but I'm not sure how well underscore prefixes would be understood as private parameters / a private ctor, whereas wrapping a clearly "private" type should be clear to everybody.

Glyph is not entirely correct on the "any class" bit as you can always break the default init path:

  class ShippingOptions:
      _ship: Literal["fast", "normal", "slow"]
      __init__ = None


  def shipFast() -> ShippingOptions:
      opts = object.__new__(ShippingOptions)
      opts._ship = "fast"
      return opts

however that's a pretty ugly pattern, and unlike the one they propose I doubt tooling would understand it.

"Glyph" knows. He has been in the Python inner circle for decades back to when the circle promoted "spam and eggs" and "consenting adults".

Like the rest of that circle, he moves with the times, supports public shaming of Tim Peters and others and now promotes poorly implemented information hiding so Python ticks a few more boxes for the industry.

Information hiding in a language that allows changing the values of small integers at runtime via ctypes is doomed anyway. And there are plenty of better languages that do it out of the box and in a straightforward manner.

I don't know who any of these people are. This seems very gossipy.

Are we, though? No offence but if an external module starts calling an internal function they better prepare a PR that changes the internal (yes, I know, by convention) to a public one. This is a signal to the developers of that module that they have to maintain the behavior.

That PR might well be rejected. And you have to work with the module owners to get your case supported.

Anything else is not responsible and I would not call it "adult".

or they could just test and maintain the hack on their own.

Yeah, good luck with that. I've had too many libraries change underneath me by clearly violating the previous contract that I know it doesn't work.

nothing is stopping adult users from disabling the type checker and using your internal type directly. the newtype is just a private class mechanism that comes with better tooling to validate that you aren't breaking the intended contract.

The conversation here is surprisingly devoid of comments on name mangling, which _almost_ enforces private properties. https://docs.python.org/3/reference/expressions.html

An alternative to consider might be to accept a Literal[“fast”, “slow”] or an Enum FAST or SLOW, and then decode that into shipping options inside the shipping code.

Only then are you truly putting a solid boundary between your library and the folks using your library. Everything else is just praying that you and only you have an underscore on your keyboard! :)

And of course another alternative is to accept that there is no true private in Python other than defdef*, so you allow your ShippingOption to be publicly visible while also documenting that the helper-constructors are what should really be used.

*”defdef” as in function definitions inside other function definitions — closures if you will, although I prefer to write mine as taking most if not all their parameters explicitly:

  def public(foo):
    def private(foo):
      …

    class Private:
      …  # less common

    …

Funny, I ran into the same pattern just a few months ago!

In practice, I found it difficult for coworkers to read and understand so I dropped the idea.

Another limitation I found is that it breaks down when you start using inheritance. For example:

```

class _A: pass

A = NewType("A", _A)

class _B(_A): pass

B = NewType("B", _B)

def foo(a: A) -> None: pass

b = B(_B())

foo(b) # Mypy is not happy: Argument 1 to "foo" has incompatible type "B"; expected "A"

foo(A(b)) # Mypy is OK

```

(On Hacker News you can do code blocks by indenting each line with two spaces.)

Aaah nice! Thank you!

Just use a generic and make it bound to (A, B):

    from typing import *
    
    
    class _A:
        pass
    
    class _B(_A):
        pass
    
    A = NewType("A", _A)
    B = NewType("B", _B)
    
    def foo[T: (A, B)](val: T) -> T:
        return val
    
    a = A(_A())
    b = B(_B())
    
    _a = foo(a)
    _b = foo(b)
    
    reveal_type(_a)
    reveal_type(_b)

Playground here: https://mypy-play.net/?mypy=latest&python=3.12&gist=36573363...

This does seem like an abstraction leak though.

The abstraction gets leaky once you expect the distinct NewTypes to adhere to the original inheritance property. I think that's a wrong assumption from the get-go.

OP could just do:

    def foo(val: _A) -> None:
        pass

...and it'll accept both NewTypes just fine. I guess it depends on whether foo is designed to be public or private.

This works but it does not let other modules define such methods.

Java made opaque types possible from the very start by private and package-private constructors.

It's sad to see that many features regarding object-oriented programming and static typing are implemented worse in Python than Java. Various examples: __str__() vs. toString(); underscore vs. private; @staticmethod/@classmethod vs. static; generic types are so clunky in Python; types are not shown in the official Python standand library documentation; __init__() doesn't force you to call super() whereas it's mandatory in Java; @override (Python 3.12; year 2023) copying Java @Override (JDK 1.5; year 2004) very late; convention changing from duck typing (always available in Python) to structural typing (optional in Python, mandatory in Java).

I agree that these are implemented "worse" than Java but that's because Python wants these things to be optional! You can complain about it as much as you want but that just means it's the wrong language for you, if that's important to you.

(I'd like to note that I wrote a lot of code in Java and Python, and continue to use each language in its respective strong areas. This isn't meant as a drive-by attack of "Java r00lz, Python sux"; this is an experienced take.)

My real problem with the evolution of Python is that initially, the language and the community was positioned as anti-Java, anti-big-OOP-like-C++, and then it changed into the thing that it was against, but in a roundabout and suboptimal way. To me, the initial vibe of Python was, "write a 100-line script, don't worry about explicitly documenting types, don't worry about grand architecture, don't worry about creating custom classes, don't worry about encapsulation and public/private". I've been with Python since year 2007 in the 2.x days, and Java since 2002.

Initial examples: Why go through the ceremony of `public static void main(String[] args)` when Python just executes the script line by line at the top level? Oh wait, now you have things like `import` actually executing code instead of simply being a compile-time namespace convenience, and you need weird techniques like `if __name__ == "__main__"`. Why `System.out.println()` when `print()` is so much more concise? But now you're polluting the global namespace, and `print(file=sys.stderr)` isn't that elegant either.

Static typing in Python is the biggest hypocrisy ever. As I understood it, Python scripts were meant to be lightweight and free of the tyranny of enterprise OOP which was epitomized by Java. But people found out that keeping track of types in your head is laborious and error-prone, and getting a compiler to check {that the shape of your objects and function calls match} is a huge productivity boost. And so Python 3 enabled static type hints... which, like I said before, Java had from day zero. To make matters worse, static type hint features were introduced progressively over the years, leading to things getting deprecated from the `typing` module and moved to things like `T|None` and `list[T]` and `collections.abc`.

IIRC the old practice in Python was that you specified some kind of interface in prose or in code (e.g. `class IoStream: def read(); def close()`), but you didn't need to explicitly use that interface as a superclass; you can just duck-type your way around things. But this completely goes against static typing, so I'm pretty sure the new preferred way is to explicitly use abstract superclasses... just like Java did all along (and is mandatory).

I really don't think having top-level (module) variables and functions in Python is a good thing, especially because then they are duplicated as fields and methods in classes. In Java, fields and methods (whether static or instance) can only be placed in classes, and I think this particular straitjacket is a good thing.

> because Python wants these things to be optional

We can both agree that Python gives multiple ways to do things (e.g. no static type hints vs. static type hints). This flies in the face of:

> Readability counts.

> The Zen of Python / There should be one-- and preferably only one --obvious way to do it. -- https://peps.python.org/pep-0020/

Probably the most tragic example is the ways to build up strings in Python: `+` and str(), `%` operator, `str.format()`, f-string.

(To be fair, I have a laundry list of complaints about Java too, such as: .class files and the JVM being an intermediate layer that needs to be understood which is actually different from the Java source language, lack of in-place structs so `new Point[]` is very painful on the memory system, awkward string interpolation/formatting compared to Python's f-strings, very awkward JDBC compared to for example Python sqlite3 API, kinda clunky for web server programming, very awkward JSON handling, enterprisey libraries and APIs that are perfectly documented but are impossible to actually understand.)

> so I'm pretty sure the new preferred way is to explicitly use abstract superclasses... just like Java did all along (and is mandatory).

typing.Protocol is a good fit for this use case

  from typing import Protocol
  
  class HasMessage(Protocol):
      def get_message(self) -> str: ...
  
  class A:
      """Implicit (duck-typed)"""
      def get_message(self) -> str:
          return "A"
  
  class B(HasMessage):
      """Explicit"""
      def get_message(self) -> str:
          return "B"
  
  class C:
      def get_message(self) -> int:
          return 1
  
  def print_message(m: HasMessage) -> None:
      print(m.get_message())
  
  print_message(A())
  print_message(B())
  print_message(C())  # fails type check

Are you ever supposed to inherit from the protocol though(unless you’re defining another protocol)? One of the great things about protocols is that your class doesn’t even have to know about the protocol explicitly. What this code looks closer to doing (style wise) is an abstract base class

"To explicitly declare that a certain class implements a given protocol, it can be used as a regular base class. In this case a class could use default implementations of protocol members. Static analysis tools are expected to automatically detect that a class implements a given protocol. So while it’s possible to subclass a protocol explicitly, it’s not necessary to do so for the sake of type-checking."

https://peps.python.org/pep-0544/#explicitly-declaring-imple...

> Static typing in Python is the biggest hypocrisy ever

Yes, agreed. I used to work on a large python codebase and tried to add type hints where I could. The issue is that python was not the right tool for the job - except that switching to the right tool was a non-starter. So type hints were the best I could do.

It is indeed a significant undertaking. But... it is doable. I've worked on a code base that converted several functionalities to golang. It did take a lot of effort and quite a lot of planning.

Let it out co-internet programmer. It’s ok. The pain is real. We both love and hate our languages. It’s natural and healthy.

> Why go through the ceremony of `public static void main(String[] args)` when Python just executes the script line by line at the top level? Oh wait, now you have things like `import` actually executing code instead of simply being a compile-time namespace convenience, and you need weird techniques like `if __name__ == "__main__"`.

I don't think this is a fair criticism. Python is a scripting language, it makes sense that the code is executed line by line at the top level. This is also how other programming languages from its time like Perl or Bash does it. Even newer scripting languages like Ruby does something similar.

> Why `System.out.println()` when `print()` is so much more concise? But now you're polluting the global namespace, and `print(file=sys.stderr)` isn't that elegant either.

Another criticism that I don't think it is fair. Lots of other languages "polutes" the global namespace. I actually can't think another language other than Java that doesn't. Python at least still allow you to manually `import builtins`, but Go for example AFAIK has no mechanism for you to reference builtins if you end up shadowing them.

Also I find `print(file=sys.stderr)` pretty much elegant, it works exactly how I would expect, it also means I can open a file and write to it using `print`.

> And so Python 3 enabled static type hints... which, like I said before, Java had from day zero.

Again, I don't think this is fair. I find Python 3 type-hints much more powerful than whatever type system Java has, especially because Python has Option types that actually make the type system useful (Java is infamous for its NullPointerException for a reason).

You'll note that even Java now recognises that "public static void main(String[] args)" was pointless ceremony.

> Python scripts were meant to be lightweight and free of the tyranny of enterprise OOP which was epitomized by Java. But people found out that keeping track of types in your head is laborious and error-prone, and getting a compiler to check {that the shape of your objects and function calls match} is a huge productivity boost. And so Python 3 enabled static type hints... which, like I said before, Java had from day zero.

Java didn't have "hints", it had mandatory types everywhere. Which, again, they've now recognised was a bad idea and gradually removed. Having a type system is a good idea, and maybe if Python had taken more inspiration from OCaml (or more people had used OCaml instead of either Java or Python) we'd all be better off, but the early-2000s-Java cure of mandatory types everywhere was worse than the disease.

> To make matters worse, static type hint features were introduced progressively over the years, leading to things getting deprecated from the `typing` module and moved to things like `T|None` and `list[T]` and `collections.abc`.

Which happens in any language that evolves over time. Java, despite having a type system built into the language from day 1, now has about ten different deprecated sets of nullness annotations with overlapping functionality.

> I really don't think having top-level (module) variables and functions in Python is a good thing, especially because then they are duplicated as fields and methods in classes. In Java, fields and methods (whether static or instance) can only be placed in classes, and I think this particular straitjacket is a good thing.

Nah. No other language has adopted Java's approach, for good reason. Functions and values are easier than classes and the language shouldn't force you to complicate your code when there's no need to.

> Probably the most tragic example is the ways to build up strings in Python: `+` and str(), `%` operator, `str.format()`, f-string.

It sucks, but what's the alternative? Either a language dies young or it lives long enough to have a bunch of deprecated crap in it.

> You'll note that even Java now recognises that "public static void main(String[] args)" was pointless ceremony.

There is still the part of the ceremony that actually mattered: having a single entrypoint instead of the option to litter side effects throughout the file and having those side effects execute automatically on import.

> It sucks, but what's the alternative?

3.0 was a big missed opportunity to kill a lot of the deprecated cruft.

> the option to litter side effects throughout the file and having those side effects execute automatically on import.

You can actually do that in Java too with static initialisation blocks.

> 3.0 was a big missed opportunity to kill a lot of the deprecated cruft.

Maybe. 3.0 already came damn close to killing the language. If they'd made the changes any more radical it could easily have been another Perl 6.

> My real problem with the evolution of Python is that initially, the language and the community was positioned as anti-Java, anti-big-OOP-like-C++, and then it changed into the thing that it was against, but in a roundabout and suboptimal way. To me, the initial vibe of Python was, "write a 100-line script, don't worry about explicitly documenting types, don't worry about grand architecture, don't worry about creating custom classes, don't worry about encapsulation and public/private". I've been with Python since year 2007 in the 2.x days, and Java since 2002.

It really wasn't that anti-Java. Late 90s / early 2000s had a huge branch that was very dedicated to Java-style OOP. E.g. anything to do with Zope, and GvR himself worked on that at the time. Zope even has had its own ABC / interface system, specifically modeled after Java interfaces. stdlib logging is a reimplementation of log4j in Python and so on.

Python is much older than Java, and Java is a big OO-first language. It's a bit like saying Python doesn't do functional as well as Erlang.

Much older? Wikipedia says[^1][^2] java appeared in 1995 (started in 1991), while python appeared in 1991 (started in late 1980s). 4 years doesn't seem too far apart, considering both language are >30 years old now.

[^1]: https://en.wikipedia.org/wiki/Python_(programming_language)

[^2]: https://en.wikipedia.org/wiki/Java_(programming_language)

Not only that, but Python had the benefit of doing a very painful break in version 3 (2008), when they had the option to cleaned up almost anything they wanted.

(Some changes in Python 3 I can recall: bytes/str/unicode being the biggest one; fixing mutable variables in nested functions; changing some obscure behavior in class hierarchies and overload resolution; changing things like range() and map() to lazy evaluation.)

For better or for worse, Java has maintained very good (not perfect) compatibility throughout, even with painful changes like generics in 1.5, lambdas in 8, modules in 9, eventual removal of applets and SecurityManager, etc. This also contrasts with C#/.NET, which I think had some breaking changes over the decades.

The main problem with such approach is that `class _RealShipOpts:` is very ugly to write unit tests for. You need to import a private entity in tests. I would slightly change the presented approach, and move the "public" `ShippingOptions`, `shipFast`, etc., into a new module that is a public API, for my users to use something like `from my_lib.shipping.api import ShippingOptions`.

That way, I can use "normal" naming in `class RealShipOpts:...`, and be explicit that it's not really public for the end users (they should use the `.api` module instead).

Why wouldn't you just write the tests against the public api?

Why not just use a dictionary, or why not just leave the type unannotated? If you really can't (or don't want to) say anything about the type, then don't. Python is dynamically typed!

average python script writer

The point is to mark the constructor as "private" so that it is easy to spot unintended use during code reviews (or using linters).

The blog post does want to share some type information with users. They just want to prevent users from relying on a specific implementation of that type.

They are basically describing a public API backed by a private type that they can extend, rearrange, or otherwise modify without breaking the public contract.

I'm sorry but if you write Python functions/methods in camel case I can't take you seriously.

Just fyi, the author is https://en.wikipedia.org/wiki/Glyph_Lefkowitz, creator of Twisted.

Seems like this is even more reason to set a good example and follow the style guide.

Twisted has had its own style guide for decades: https://docs.twisted.org/en/stable/development/coding-standa...

(Earliest mention I could find for camelCase methods was 25 years ago: https://github.com/twisted/twisted/blob/d7c19cd40d07c8c37f85... )

> get_someAttribute

Hell no

Maybe, but the position to not listen to him at all because he doesn't do that is still a much worse look in my book.

Wow, that's a much better reason to not take them seriously.

Do not ever do this Javaesque nonsense in Python or anywhere really. Use ShippingOptions directly.

Reply on news.ycombinator.com