Show HN: Koda, a typesafe functional toolkit for Python | Hacker News Reader

Show HN: Koda, a typesafe functional toolkit for Python

pypi.org

110 pointskeithasaurus58 comments

A similar "framework", very robust and used in production systems, is dry-python's one[0][1].

The approach taken is a bit different I think, since they rely heavily on `mypy` plugins to reach type safety and functional constructs otherwise impossible to get, without runtime inspections.

0: https://github.com/dry-python 1: https://github.com/dry-python/returns

dry-python has a lot of cool stuff. I encourage people to check it out. Koda has a slightly different approach in that it's meant to be a little simpler than returns, and to work within Python's constraints.

Am I strange for disagreeing with the very first example of dry-python? They suggest the second version is "much better" than the first but I sort of hate it.

===================

user: Optional[User]

discount_program: Optional['DiscountProgram'] = None

if user is not None:

     balance = user.get_balance()

     if balance is not None:

         credit = balance.credit_amount()

         if credit is not None and credit > 0:

             discount_program = choose_discount(credit)

===================

user: Optional[User]

discount_program: Maybe['DiscountProgram'] = Maybe.from_optional(

    user,

).bind_optional( # This won't be called if `user is None`

    lambda real_user: real_user.get_balance(),

).bind_optional( # This won't be called if `real_user.get_balance()` is None

    lambda balance: balance.credit_amount(),

).bind_optional( # And so on!

    lambda credit: choose_discount(credit) if credit > 0 else None,

)

I agree with you that the second is less readable and less pythonic (feels more JS/Node-ish).

The second example is a pretty standard monadic composition pipeline. In languages like Haskell or F# there can be special syntax to make it easier to read (do notation or computation expressions).

However if you’ve never done it will be hard to read. It’s easy to read to someone who has the background, and it is easier to refactor and less error prone because the context of a nullable value (the if statement checks) is factored out from the core logic. So if the core logic needs to be changed it can be done so without touching the if statement checking for None.

Especially given the first example is tailor-made for the walrus operator.

    if (balance := user.get_balance()) and (credit := balance.credit_amount()):
        discount_program = choose_discount(credit)

(Omitting the credit > 0, and without getting into other fundamental issues with the implied design, since it's just a readme example.)

Really nice project! I’ve been looking for something like this, and typically end up building a very minimal version of it for projects. Are you looking for help or contributors? Would love to contribute if possible.

Yeah, happy to have contributors. I'm not 100% certain on what the next features / changes will be, so also happy to have input, requests, questions, etc.

I want a similar toolkit that lifts over the Ruby/Groovy/Kotlin style functions like countBy, groupBy, sortBy, collate, etc etc. Programming in Python is such a chore without these higher level collections APIs and for some reason (I guess, list comprehensions being considered idiomatic, bare bones itertools being considered "good enough", and the lack of real inline closures?) it doesn't seem like they'll ever be part of Python proper. But I still want them.

With full acknowledgment that it would be better to have at least some of those in the stdlib, Pandas covers all those needs pretty well in practice, with the bonus of workarounds that sidestep CPython’s usual performance limitations.

There is https://github.com/lauriat/funct which has some of those higher level functions.

Maybe the toolz[0] family would cover your use cases? There is also a Cython implementation if you need better performance.

[0] https://github.com/pytoolz/toolz/

The `collections`, `itertools` and `functools` packages have most of these.

Nice. One question I have is does the compose operator preserve error messages, allow you to pinpoint which function raised an exception etc? That's the one issue I have w/ the `pipe` function from dry-python/returns; if you build a complex pipeline, the errors are somewhat inscrutable.

So far I've been able to follow them. If you `raise` an exception during one of the functions that's used in a compose the traceback will mention the function that failed (I'm looking in 3.9 at the moment).

Can't you 'compose' together the various errors inside the `Failure` and inspect it at the end of the pipe?

I was pretty puzzled by the method name "map" in the examples. Why would you call it "map" when there are no arrays or lists involved?

Because of the mathematical "map" [0] perhaps?

[0]: https://en.wikipedia.org/wiki/Map_(mathematics)

Are you thinking about Maybe.map in particular? It's the same name because it does the same thing: run this function on the value inside if there is one. It's a Functor :)

A Maybe is effectively isomorphic to an immutable list whose cardinality can only be zero or one.

x.map(f) doesnt mean "apply function f to each element in this list that we called x"

x.map(f) means "apply the function f to the value within x, where x is a container of some sort"

[].map(f) would do nothing, because there is nothing to pull out.

['adam', 'bill'].map(f) takes 'adam' out of the array, applies f to it, and puts it back in the array, then does the same with 'bill'.

Just(5).map(f) takes 5 out of the Maybe, applies f to it, and puts it back in

nothing.map(f) returns nothing.

So we can map functions to values in Arrays, map functions to values in Maybes and even map functions to values in Eithers. Each time the function doesnt care that the value is in a Maybe or an Array or an Either.

The only requirement for something to be mappable is that it be iterable

Brian Lonsdorf has a fun course where he explains these things: https://egghead.io/courses/professor-frisby-introduces-compo...

Because the Result and Maybe types in the examples are functors: https://en.wikipedia.org/wiki/Functor_(functional_programmin...

Why would you write Python like it’s Rust? I really fundamentally do not understand the purpose. Ain’t `is Null` checks and try except good enough?

Came here to say the same thing...

Yeah this sort of thing is necessary in statically typed languages like Rust and C++ but in Python (and e.g. Typescript) you can just use anonymous type unions like `str | None`.

There's no need to construct a static type to hold both possibilities because variables can already hold all types.

Good luck checking for `None` equality and indent your code like 5 times... a sort of railway programming[0] avoids the awkward checks for values, making functions work with higher order types.

0: https://www.greenbird.com/news/railway-oriented-programming-....

The original Railway:

https://fsharpforfunandprofit.com/rop/

And his 'Against Railway-Oriented Programming':

https://fsharpforfunandprofit.com/posts/against-railway-orie...

How is this different from match case ok case err or from if result.val?

Well, the most basic pro is to avoid indentation and have pure function which are clear and have less assumption to take into account; if I know that my pipes always return me a result such as the one I want, and the "bad path" is discarded automatically, code is way cleaner.

Imagine a code without `if result is not None` basically!

  if a is None:
    raise Exception()
  if a.b is None:
    raise Exception()
  if a.b.c is None:
    raise Exception()

One of the advantages of Python/Typescript style type hinting is that it avoids the mega-nested `if`s that you often have to resort to in Rust. In Rust the above would not work:

  if a.is_none() {
    bail!();
  }
  if a.b. // nope! a is still Option<>.

It may be possible to fix that when enum variants are distinct types but as it stands you often have to do lots of indenting.

  foo | None

is not a tolerable replacement for

  Maybe(foo)

When None is a valid value of type foo.

Which is obviously true when foo is NoneType, but more to the point is true when foo is (bar | None).

Maybe composes, “... | None” does not.

a) do Optional[Union[foo, bar]] b) why do you expect this from a duck typed language? None of options you listed or the library is offering saves you from mismatched types. Mypy is still as good as it can possibly be using only vanilla type hints

> None of options you listed or the library is offering saves you from mismatched types.

Even if true, so what? The utility of Maybe/Result datastructures isn't limited to their use in typechecking, though with a proper static typechecker, that is part of their use.

I've definitely hacked my way around this with sentinels. `foo | sentinel | None` is a passable replacement for Maybe(foo), and it arises naturally from dict.get("key", SENTINEL). Absolutely not fun to need to check, but if you're using it within a single function e.g. an implementation of `deep_get` or something like that, the code is readable.

Yeah, there's lots of things that work good enough for localized use to approximate something like Maybe or Result and a lot of times that's all you need, and (in Python) more convenient than the real thing when that's all you need.

Wanting something that's ergonomic for free composition where you aren't anticipating all the potential uses where it is defined is where real monadic Maybe/Result types shine, and once you pull them in to a code base it sometimes makes sense to use them for the local cases, too.

Because you'd like use the Python ecosystem, for example for writing ML code, but you'd also like to write typesafe code (or rather typechecked, using mypy)

I let myself do a little bit of thinking and I guess this might be helpful when you are doing functional style pipes/chains but you can wrap the whole chain into a try except block. But the question is still there: why would you do that? Why the one would write functional code in a language that ain’t very good with maps and filters, has no lazy calculations and has no constants on language design level? Why would you set `Maybe[int]` type for a function argument? That’s an obvious overcomplication imo. Go and write some F# or Haskell. Well, the variety of options and flexibility is always good, but the language is fundamentally not meant to do it. From what I can see, it only makes sense in recreational programming, making some eye-pleasing code, but it all is just a workaround on a language that would not benefit from that code style.

All above is strongly imo of course since I only write Python and am just learning Zig.

> Why the one would write functional code in a language that ain’t very good with maps and filters, has no lazy calculations and has no constants on language design level?

Python is fine with maps and filters, and has lazy calculations.

Please provide links so I can read on this

The map() function is already lazy (in python 3 it returns a generator, and not a materialized list).

You can also have a look at "generator expressions", which are the lazy counterpart to a list comprehension (hint: use " ()" instead of "[]").

> Please provide links so I can read on this

https://docs.python.org/3/library/functions.html#map

https://docs.python.org/3/library/functions.html#filter

https://docs.python.org/3/howto/functional.html

Python syntax is very very well suited for functional programming. A python map and filter is done with list comprehensions and even flatmap is doable within the list comprehension ecosystem. Additionally the syntax transfers over to inputting values into n-arity functions and even dictionary comprehensions. Note that this feature Isn't even part of the python standard library... map and filter and flatmap are an inherit part of the CORE syntax. It makes it much better suited for FP then javascript imo.

   x = [i*2 for i in range(5)] #[0, 2, 4, 6, 8]
   y = [i for i in range(8) if i%2 == 0] #[0, 2, 4, 6, 8]
   z = [[1],[2],[3],[4]]
   w = [j for i in z for j in i] #[1,2,3,4]
   t = {str(j):j*2 for j in w} #{"1":1, "2":4, "3":6, "4":8}
   
   #reduce (note the generator (i for i in range(20))... that's a lazy list essentially)
   reduce(lambda acc, x: x + acc, (i for i in range(20)), 0) #sum(0 ... 20)
   
   
   #get max value from a bunch of numbers:
   f = max(i for i in range(30) if 30 % 3 == 0) #max(0, 3, 9, 12, 15, 18, 21, 24, 27, 30) = 30
   
   #basic recursion
   def myreverse(x: str) -> str:
       return x if len(x) <= 1 else x[-1] + myreverse(x[:-1]) #O(N) on each slice, (However x[-1] is O(1) unlike the haskell list). 

   #qsort haskell style
   def qsort(l: List[int]) -> List[int]:
       return l if len(l) <= 1 else qsort([i for i in l if i <= l[0]]) + qsort([i for i in l if i > l[0]])

Python also supports pattern matching shown here: https://www.python.org/dev/peps/pep-0636/

Additionally lambda syntax, aka python anonymous first class functions HAVE to be functional.

   f = lambda x: x + 1

There is no procedural concepts like variable assignment allowed in a python lambda. This is done because they wanted to force the python lambda to be properly functional and short.

The type hints that python uses are also quite good with support for generics sum types and even type level programming (however your type checker needs to support it too and basically none of them do right now) You can literally assign a type to a variable in python. See: https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html

I would say the main weakness in python is that the IMPLEMENTATION is not well suited for functional programming (but the syntax VERY much is). Tail slices on lists like x[1:] cost O(N) and that's the biggest issue right now imo. It makes some valid solutions on leetcode exceed the time limit because it adds and additional N complexity on every slice when the head and tail functions in haskell cost O(1)

I'm sure some library can easily provide the underlying functional primitives python needs to be faster. Actually on that note, does anyone know of a library that solves the head tail slicing problem I described above? Probably should use the python deque, but the api for that is inherently not functional.

> There is no procedural concepts like variable assignment allowed in a python lambda.

Just a small note that the walrus operator gives you some wiggle room here:

  add_5_to_square_if_greater_than_10 = lambda x: squared + 5 if (squared := x * x) > 10 else x
  assert add_5_to_square_if_greater_than_10(2) == 2
  assert add_5_to_square_if_greater_than_10(5) == 30

yeah my bad. Not fully up to date with all the newer python 3 tricks. That walrus operator is a trap though imo.

> Additionally lambda syntax, aka python anonymous first class functions HAVE to be functional.

They have to be a single python expression, but that’s not the same as the usual understanding of “functional” (i.e., referentially transparent). Consider:

  d = {"foo": 1}
  f = lambda m: d.update({k, v+d.get(k,0) for k,v in m.items()}) or d

> Tail slices on lists like x[1:] cost O(N) and that's the biggest issue right now imo. It makes some valid solutions on leetcode exceed the time limit because it adds and additional N complexity on every slice when the head and tail functions in haskell cost O(1)

Python lists are not lisp-style lists, which is more efficient for most uses, but not for a lot of functional-style algorithms which leverage cheap head/tail splits.

But it wouldn't be hard to write a typed sequence class that wrapped a lisp-style immutable list implemented with nested tuples with the deepest nested one an empty tuple in Python, there's just not a lot of demand for it.

>They have to be a single python expression, but that’s not the same as the usual understanding of “functional” (i.e., referentially transparent). Consider:

True, yeah

>But it wouldn't be hard to write a typed sequence class that wrapped a lisp-style immutable list implemented with nested tuples with the deepest nested one an empty tuple in Python, there's just not a lot of demand for it.

A doubly linked list with the same slice syntax would be fine. You can use java style classes as it's just the underlying implementation. There's no demand (currently) but it's a good idea as python syntax works extremely well (better then even javascript) for functional programming.

> ain’t very good with maps and filters

Isn't a comprehension just a map/filter by another syntax? You can re-create them from a list comprehension pretty simply:

`map = lambda data, map_fn : [map_fn(x) for x in data]`

`filter = lambda data, filter_fn : [x for x in data if filter_fn(x)]`

Then, why even have the lambda-translations above since the comprehension is already "Pythonic"? Also, you have functools (https://docs.python.org/3/library/functools.html) which you can use for several functional programming paradigms as well.

Functional programming isn't what you call the techniques (map/filter/etc.) it's how you are programming that makes it functional. You can still use functional techniques in Python to make things a bit easier to reason about. A lot of the things people thing about functional programming aren't inherent to the actual functional programming paradigm: like laziness or immutability.

Nit: in py3 map() and filter() are lazy (they return iterators instead of lists). You can better mimic their semantics using generator expressions "()" instead of list comprehensions "[]".

Say you're given a Python function in a large codebase that doesn't have documentation but does have type annotations, say, to indicate it returns Optional[str]. Does this tell you that you just need to check for None, and otherwise it's a string?

Nope, it could throw. What could it throw? We don't know. There are many applications where this magnifies the dangers of any existing technical debt. One could desire strong exception-proofing of results without needing to go "all in" to a borrow-checked regime like Rust.

The Koda approach is lightweight and doesn't enforce this, but one could imagine a static checker that says "any function that returns a Result, and calls any function that does not itself return a Result or does any other risky operation, must have an exhaustive top-level try-except block, or be wrapped by a decorator that does the same."

Then, you could guarantee that all code explicitly handles exceptions, while still accessing the full Python library ecosystem - you can use non-Koda-wrapped code at any time, you just need to handle its exceptions.

Is this overkill? Possibly. But I can think of a handful of bugs offhand that this would have caught in our codebase if enforced.

Now this is an answer I was waiting for. Thank you for explaining the real benefits of sticking to koda style. I guess this would be obvious if I used statically typed languages since there is no try except clauses in most of them

I'll add that caveats do apply to these benefits. Many of them are outlined in https://fsharpforfunandprofit.com/posts/against-railway-orie... - which is a response to an excellent functional programming talk (see slides or video linked at https://fsharpforfunandprofit.com/rop/) that walks through why you'd want a Result-type thing to exist in the first place... and eases you into the larger world of monads!

> and eases you into the larger world of monads

Yes, just begin reading at page 29

https://www.slideshare.net/ScottWlaschin/railway-oriented-pr...

I wanted to ask the same thing: Couldn't we just use Optional and None for this? So the point of using Koda is to communicate intent (functions that don't throw exceptions), and some utility methods?

You can also nest `Maybe`s. Just(Just(nothing)) could convey that two levels of a computation "succeeded", but the third did not. This may not be something you see often, but it's a case that Optional can't convey.

There are also cases where None is a valid value, and then Optional doesn't make sense. Maybe[None] can be a valid, if rare, use case -- Just(None) is different from Nothing. But Optional[None] doesn't make sense -- since it's None | None.

You can also do other stuff with Maybe, like map, flat_map, apply. And there's more that can be added.

I think the discussion is far enough along here to cover a lot of the bases. Just to add a bit of clarity, Scala's APIs were a bit more influential than Rust, but, yeah, it's similar to both.

No, they're not. And I wouldn't say it's writing it like Rust; rather, it's writing it like you can sleep at night. These concepts weren't born in Rust, they were born in functional languages and category theory and they're all about making sure you don't get away with sloppy code and billion-dollar mistakes[0].

[0]: https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retra...

Reply on news.ycombinator.com