Hacker News Reader

Comment by noelwelsh | Hacker News Reader

How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.”

Preventing this is one of the basic motivations of functional programming. If a function is "referentially transparent" or "pure" it can't do anything unexpected. It seems to me this is the only reasonable way we can reason about software at scale.

View on HN

> If a function is "referentially transparent" or "pure" it can't do anything unexpected.

Sure it can; at a minimum all functions consume compute resources. A tiny change in a "pure" JavaScript function could lead to cascading deoptimizations that destroys performance and compromises service availability. For example, changing 0.0 to 0 could do it.

A slight change in performance characteristics can also lead to cascading failures. Pure functions don't help with that. That aside, even functional programs need to have side effects (other than heating up the CPU). Referential transparency doesn't help when the state of your monad is a little bit weird and that causes some corner of your code to kill the DB.

Anecdoctally this happened to me the other day in my Haskell backend: had a recursive function (in the IO monad) that every ten second would take a connection of my connection pool (and since it was recursive) would not give it back to the pool. Of course it was a stupid bug, but it was hard to figure out because it would take a while before it got to that point and once it was there every call to the backend would block then timeout. I like functional programming, and yes: it was in the IO monad so not pure, but I think most applications ends up in some stateful monad even if your stack is built for functional stuff... This was 'not at scale' but it as a Haskell project it is fairly big. Point being: yes, pure apps will not havr this predicament, but even with a stack built for pureness I can see it happen quite often. Running software is messy...

This makes another argument for Haskell, because the language allows you to isolate the non-pure code into a thin top layer. That makes debugging much easier than it would be if the complexity of IO extended all the way down through your app's subfunctions.

While I generally agree that functional programming does help immensely to make secure software, in this case, the function wasn't pure. It was meant to return data that didn't exist from a database, but the error was silenced. Also, the reason why the servers were "burning" was because this somehow resulted in a memory leak that consumed all the memory available to the system, which is why they also had trouble pushing fixes. Even in a pure function in Haskell, memory leaks are possible and the same scenario could've resulted.

While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand internally, so multiple levels of black-box abstraction are required to integrate/interact with the sub components. Additionally, both the client systems and the sub components may have huge amounts of persistent state information.

Could you suggest any resources that might help me understand how this can be accomplished in a pure functional paradigm? Something that assumes only a casual understanding of functional programming would be especially helpful.

> Do you have anything that explains this complex topic?

> Something that assumes I don't know anything about the pre-requisite topic would be helpful

Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms.

Me saying "Monads can help with state abstractions" doesn't help you.

The fact that large non-trivial applications have been written in haskell should be enough evidence that this is possible.

The fact that the majority of people involved in such endeavors have claimed that it made their code safer and easier to refactor and had relatively few bugs should be enough evidence that it's a good idea.

The fact that those who decry it more often than not do not know the subject should give their criticisms no weight.

I don't see why you need any more evidence that you should simply learn functional programming so that you may first-hand answer your own question.

> Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms.

> I don't see why you need any more evidence that you should simply learn functional programming so that you may first-hand answer your own question.

It is possible to interpret the GP comment as a request for good resources to learn FP. You seem like you could probably point one or two out. Since they've already expressed a desire to understand, that might be more beneficial for them than the equivalent of "just do it".

Thank you. Basically I was looking for the functional-programming equivalent to the OOP factory-equipment analogy, or asking if such an analogy does/can exist (if it can't, the why might also be a valuable explanation). That analogy can be used to explain the main ideas behind encapsulation, interfaces, and internal state, and why they are valuable, before having any understanding of how those are implemented in practice, and even without any programming knowledge at all. So far everything I've found on functional programming starts with implementation details and provides no initial framework for understanding the broad-strokes of organisation and value proposition.

You might like Rich Hickey's talks, especially "The value of values", "The language of the System", and the 2018 conj key note. The Talk in transducers also has a nice analogy: the conveyor belt.

Unfortunately the only way to gain a deep understanding of the powerful tools functional languages do give you is to gain a deep understanding of functional paradigms.

If I didn't know what 'functional' was in this context, I'd be entirely convinced you were tying to convert me to your religion with that statement. This attitude from the functional-programming community is the main reason I have avoided learning more. I was asking a legitimate question and instead of providing something useful you decided to assume I don't know anything about the pre-requisite topic and spout verbose dogma at me that does nothing but attempt, and fail, to make you appear intelligent.

Maybe the blogpost "Functional architecture is ports and adapters" by Mark Seemann may be of help to you. "fsharp for fun and profit" by Scott Wlashin is really approachable. Cant post links since on mobile.

Thanks, I will take look at both of those.

Before you spend some time doing mostly functional it's easy to ignore just how much meaningless state you might have floating around. You're right that any non-trivial endeavour will necessitate quite a lot of information being kept around. But in my experience most of the state you produce in languages that don't support functional patterns isn't even related to the domain. You're often forced to put thing into variables, objects and all sorts of things that might not make a lot of sense.

The original article wasn't talking about issues caused by storing too much program state in in-memory structures (variables/objects) though. It was talking about issues introduced by interactions with an external stateful system--a database.

Bugs related to in-memory state that gets out of whack are definitely a hassle--especially in multithreaded situations--but are, in my experience, only the tip of the iceberg. Once you have that kind of issue under control (either by choice of platform, discipline, linting, or unit testing) there is a huge category of problems that can arise from the remaining surface, categorized loosely as "as soon as your code leaves its own memory space, it's part of a distributed system, with all the hassles that entails". Proclaiming that FP techniques resolve or even significantly ease that category of issue seems debatable at best, and a false promise at worst.

By definition, pure functions can't read from or write to disk, can't read from or write to the database, and can't read from or write to a socket.

If you aren't able to do any of those things, there is no "at scale" - you're trapped in a single process on a single computer, spinning away but not able to communicate with anyone else.

Pure functional programming helps within an individual component by letting you quarantine the statefulness at defined areas (e.g., in Haskell, at the program's entry point), but it doesn't let you make it disappear. Sooner or later, you're gonna have to deal with it.

At scale, you've got tens or hundreds or thousands of different processes, and each of them has its own statefulness that it ultimately has to deal with. When you're looking at the big picture like that, I'm not sure it makes sense to worry about whether it's been quarantined, Haskell-style, or not. It's still there, and, in the big picture, it's still distributed more-or-less homogeneously throughout the system. Even a process that doesn't use disk or talk to the DB or anything like that is still not referentially transparent, because it's still subject to side effects by virtue of the network being unreliable, the OOM killer being unpredictable, and all that fun stuff.

> I'm not sure it makes sense to worry about whether > it's been quarantined, Haskell-style, or not. > It's still there

True, but the quarantine makes the bug much easier to find.

Reply on news.ycombinator.com