File System Interfaces for Go – Draft Design

The article is literally: "Hell thing" Step 1 -> monads step 2 -> ???? step 3 -> profit?

This is apparently the monads curse where people that get them can't effing explain them to save their life.

I just skimmed but IIUC, the article is a little more of an in-joke than an explanation and does seem to expect its audience to already understand (or maybe be intrigued enough to learn more from other sources?).

That said, I can try to explain here:

Haskell has a bit of syntax available called "do notation". You can write Haskell without it, but it makes some things read better (as a matter of common but not universal opinion).

There's a simple, purely syntactic translation from do notation into regular application of functions. "Syntactic sugar causes cancer of the semicolon." There are four rules, none of which is complicated, and only two of which are relevant here:

First, a single expression is just that expression, nothing magic happens.

    do
      expr

simply becomes

    expr

Next, the arrows:

    do
      x <- m
      ... more stuff, which might use x ...

becomes

    bind m (\ x -> do
      ... more stuff, which might use x ...
    )

or in a few other syntaxes:

    bind(m, x => do ... more stuff, which might use x ...)
    (bind m (lambda (x) do ... more stuff, which might use x ...)
    m.bind(|x| { do ... more stuff, which might use x ...)
    bind(m, [] (auto x) { do ... more stuff, which might use x ...)

That internal do is then expanded recursively.

So to translate the whole block that's repeated throughout the article:

    do
      a <- getData
      b <- getMoreData a
      c <- getMoreData b
      d <- getEvenMoreData a c
      print d

becomes

    bind getData (\ a -> do
      b <- getMoreData a
      c <- getMoreData b
      d <- getEvenMoreData a c
      print d)

which becomes

    bind getData (\ a ->
      bind (getMoreData a) (\ b -> do
        c <- getMoreData b
        d <- getEvenMoreData a c
        print d))

which then becomes:

    bind getData (\ a ->
      bind (getMoreData a) (\ b ->
        bind (getMoreData b) (\ c -> do
          d <- getEvenMoreData a c
          print d)))

and then:

    bind getData (\ a ->
      bind (getMoreData a) (\ b ->
        bind (getMoreData b) (\ c ->
          bind (getEvenMoreData a c) -> (\ d -> do
            print d))))

and finally

    bind getData (\ a ->
      bind (getMoreData a) (\ b ->
        bind (getMoreData b) (\ c ->
          bind (getEvenMoreData a c) -> (\ d ->
            print d))))

Which is "just" a bunch of chained functions combining lambdas.

So... why bother? and how does it do so many different things? and how does it know which to do? and what does it even do?

The key is that we're overloading "bind", picking the behavior that we want. You could do this in most languages by passing in a choice of function. We could have "do" take a parameter, like

    do(bind)

or in an OO language you might hang bind on the objects involved. We often see this for particular instances - for instance, .then for promises/futures.

Haskell does it with a mechanism called "type classes", where you can specify an implementation of an interface for a given type, and the compiler will figure out which implementation to provide. This is very similar to Traits in rust, implicits in Scala, etc. You can usually avoid specifying the types manually because of inference.

So in Haskell, Monad is an interface providing two functions:

    class Monad m where
      bind :: m a -> (a -> m b) -> m b
      pure :: a -> m a

We've already come across `bind`, which lets us operate "inside" a context in a way that combines the contexts.

The other function included, `pure`, takes a value and gives it the minimal possible context. What that means is implied by the "monad laws", which tell us how `bind` and `pure` must interact.

For each type that "is a monad", we tell the Haskell compiler how that type implements the interface:

    // optionality:
    instance Monad Maybe where
      bind Nothing _ = Nothing
      bind (Just x) f = f x
      pure x = Just x

    // state:
    newtype State s a = State (s -> (a, s))
    instance Monad State where
      // pure gives the State action that produces x and leaves the state unchanged
      pure x = State (\ s -> (x, s))

      // bind threads the state through the chained actions
      bind (State f0) f1 = State (\ s ->
        let (x, s') = f0 s
            State f2 = f1 x
          in f2 s')

    // lists:
    instance Monad [] where
      pure x = [x]

      // bind for lists is concatMap
      bind [] _ = []
      bind (x:xs) f = f x ++ bind xs f

... that got long. Please ask any questions you're left with :)

I think what they have in mind is a Reader monad.

    newtype Reader r a = Reader { runReader :: r -> a }

    instance Monad (Reader r) where
        pure x = Reader (\ _ -> x)
        m >>= f = Reader (\ r -> runReader (f (runReader m r)) r)

which allows us to define:

    getContext :: Reader r r
    getContext = Reader (\ r -> r)

and which spares us threading the context manually.

I don't think that's such a big deal, so long as there's only one thing we're threading.

In some languages, we can be generic over "contexts which provide the thing I want, whether they do other things or not", which is sometimes a much bigger win.

Context is like Schrödinger's Cat, you don't know if the Context you're working in is alive or dead (i.e. cancelled, timed out, etc.), but you have to keep passing it forward, one intermediate network call after the next, as long as its status is uncertain. Only when the request fully returns, or is actively cancelled, do you know whether the request is alive or dead.

The Maybe monad deals with the same issue. You have a series of function calls, each of which might or might not be legal, because you never know if you actually have the parameter for the next function call. Maybe you do, maybe you don't.

If you take the naive approach to solving the issue, you pass along Schrödinger's Cat each step along the way. You intertwine the concern of the algorithm you're trying to write with the uncertainty of whether you're carrying a live cat or a dead one. It can work, but it's ugly.

The monadic approach allows you to separate these concerns. You write your algorithm as if you know for a fact that the cat is alive. If the cat were ever to be revealed to be dead along the path, it doesn't matter, the monad separated the consequences of dealing with the live or dead cat from the rest of your algorithm. The rest of your algorithm simply doesn't get run.

Context is the same. You get to write code as if the context is always valid. You don't have to worry about context being cancelled, or timed out, or anything else. You take all those concerns and relate to them separately, in one place, where they can be neatly dealt with.

The whole point of the monadic pattern is to propagate state in such a way that it doesn't interfere with the pure algorithm which you're trying to write. You write the pure algorithm separately, and then use it within the monadic context of Context, so to speak.

And the draft specifically mentions that those methods are the minimum interface - a library that supports contexts could always do this either way.

OK, I'm convinced. Not a great interface, should just return an iterator.

You're welcome!

It is still a ton of time, about an order of magnitude more than optimal if the real/sys time split is to be believed.

"ORM is great because it means I have all the data avaliable!"

You can also use io_uring and at least do all the stats in a single syscall (or none, depending on what flags you create the ring with, but for normal use it would be one call).

Let’s not do participation trophies though, Pike and Thompson were the original authors of Go. Both of them should know better.

It’s fine to include an inefficient version if you also have a ‘proper’ way to fall back to when performance does matter. And if performance does not matter, why are you even using Go? Funnily enough Python Ruby and Node do expose performant ways to do this in the stdlib, so...

So just what the fuck are you supposed to use on a directory with 1000 entries?

I think the 'In favor of Go' argument is that the Go authors defer decisions they don't know the answer for yet. They would rather leave something unimplemented, then do it the wrong way.

But maybe you are right and they didn't anticipate these issues! I have no idea either way :)

You can compare golang's package/version management with .net's. It's obvious that the latter's designers got kicked in their delicate regions enough in the 90's that that they made it a first class priority from day one.

Probably doesn't help that at google when a package breaks they'll just force some faceless drone to make it work. And since they don't have customers in the traditional sense that works for them.

I did. Contrary a point above, I vendored dependencies in Go in 2012 the old school way: copy them into a vendor directory. It worked fine for end-user binaries, and you didn't have to worry about painfully long builds like you can get doing that with C++.

The Go authors have explicitly stated they don't do things until they figure out the right way to do them. No language gets it all right out of the gate, but Go got pretty close with e.g. its standard library.

So he complains about stuff that is alread fixed? Shall we now go back to the first version of ffmpeg? Or php pre php5? or pre java 8? or C++ pre C++11?

Come on.

He should try golang and then come back instead of relying on blogposts to form his opinion

Colossus doesn't have directories either.

http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

Page 9

What are the pains when treating prefix as directories?

I am not familiar with S3.