I Believe Zig Has Function Colors
gavinhoward.com
gavinhoward.com
Structured concurrency has nothing to do with function colouring. It is how you organise and spawn tasks, so that you can have certain guarantees e.g. the group responds to cancellation as you'd expect, etc.
I don't see how async/await plays a part in this. You could have structured concurrency with a os threading model, as opposed to a userland threading model.
You can argue that the ergonomics of Zig's async story needs better documentation, especially if you're playing with function pointers. But please don't mislead readers that structured concurrency solves this.
If so, then it does solve the color problem. The alternative to structural (ie., scopy-syntaxy) flow primatives, is to reuse existing control flow primtives (ie., function calls) and just layer on coloring.
By declining to reuse functions this way, you are dropping the need for color.
> But please don't mislead readers that structured concurrency solves this.
It absolutely does. The reason is that, in structured concurrency, when a function spawns a thread, you can expect that that thread is joined before the function returns.
For callers, this means that they don't have to worry about whether the function they are calling spawns threads or not; they just call the function, and the function does its thing.
It is the same principle as standard control flow constructs: they use `goto` underneath, but you don't need to worry about the use of `goto` because they guarantee that execution will always end up at the same place afterward (barring early returns and things like that).
So when I say that structured concurrency does not suffer from the colored function problem, what I am saying is that callers, and programmers, do not need to care what functions do underneath when they are called.
Also, structured concurrency solves the function coloring problem because you can use any function, whether it spawns threads or not, as first-class functions, or function pointers, and it all works without needing any special code, or work, or compiler trickery.
Thanks for clarifying what you meant with your original comment in the article. I completely agree that this is a very big advantage, as opposed to not using structured concurrency.
However, it seems that you are conflating two things with this assertion. For example you say:
> The reason is that, in structured concurrency, when a function spawns a thread, you can expect that that thread is joined before the function returns
While structured concurrency gives you this guarantee, this is orthogonal to whether a function the thread runs is "red" or "blue".
"Red" functions exist in a language because you want blocking to happen on userland so it's cheap to unschedule the task that is blocking.
Structured concurrency doesn't remove "red" functions from a language. They do make the ergonomics of using such functions much easier on the caller e.g. python's trio library [0].
To actually "remove Red functions from the language" you can make the user think they're just using plain "blue" function techniques for blocking.
Go achieves this by only allowing use userland threading.
Java's project Loom [1] achieves this by making common suspension/yielding points (e.g. IO) compatible with both userland and OS threading.
To me, this is what solves the colouring problem i.e. you only have one colour and keep using and writing your code as you always have been.
Structured concurrency, much like you said, is the added bonus that helps reasoning about the lifecycle of tasks a function may or may not spawn.
[0] - https://trio.readthedocs.io/en/stable/index.html
[1] - https://wiki.openjdk.java.net/display/loom/Main
(edit note: my phone messed up quoting)
But then I realized this is exactly what Go does (with the scheduler intercepting system calls), just minus the structured concurrency point, strengthening your point that these are orthogonal.
I think OP saying that structured concurrency solves the coloring problem is a very similar argument to the authors of Zig saying Zig is colorblind; it might be true for a portion of the userbase, mainly those who are callers, and mostly not for those who are library implementers.
I also think that the structured concurrency feature of "you can expect that the {thread, task, fiber, goroutine} is joned before the function returns" is weaker for the purpose of colorblindness than "you can expect that the {thread, task, fiber, goroutine} is joined before the next instruction is executed" which is what I think you'd need as a single tool for colorblindness
> While structured concurrency gives you this guarantee, this is orthogonal to whether a function the thread runs is "red" or "blue".
I disagree with this.
Making sure that threads join before return is a big reason why I can pass functions around easily as function pointers and not worry about it.
An example is in my up-and-coming build system. [1]
In that code, I have a function that opens a threadset (my term for trio's nurseries) and executes a build. Once my build system is out of alpha, the direct call to that function will be replaced by a function pointer to the build type of the user's choice. (For example, you might want a "quick build" to just rebuild a file saved by your editor, or a "full build" regardless of what is already built, or just a normal build.)
Despite using function pointers, I don't need to worry about whether threads are spawned or not. In Zig, you do need to worry about if a function is async or not when using function pointers.
This includes if I make a build type whose function will not spawn threads at all, by the way. (For example, if building on a platform where you don't have the memory to spare for extra threads.) I can use that build type's function as a function pointer the same way I would use any function that does use a threadset.
In other words, while Zig is colorblind at compile time, structured concurrency is colorless at both compile time and runtime.
[1]: https://git.yzena.com/Yzena/Yc/src/commit/a7f535f8d0df45120e...
The compiler still needs to stamp out colored function bodies because the generated code for a function with async support needs to look different - the compiler needs to turn the code into a state machine instead of a simple sequence).
It's a bit unfortunate that red and blue functions appear to have a different "ABI signature", but I guess that's needed to pass an additional context pointer into a function with async support (which would otherwise be the implicit stack pointer).
https://github.com/kristoff-it/zig-okredis
I thought using "colorblind" in the title of my original blogpost would be a clear enough hint to the reader that the colors still exist, but I guess you can never be too explicit.
fn foo() void {
bar();
std.log.info("bar is finished");
baz();
std.log.info("baz is finished");
}
Specifically, examine this:
fn foo() void {
bar();
std.log.info("foo() is finished", .{});
}
Not in a trivial way either. A good async language will never let std.log.info print until/unless bar resumes.
The difference between a "blue foo" and a "red foo" is also subtler than we pretend it is. Both will finish eventually, unless they don't, and both may be paused for as long as the OS wants in any preëmptive operating system.
bar() could contain an infinite loop. bar() could call rand() and try to reference the null pointer 1 in n times. bar() could issue forth nasal demons to haunt you. bar() could call os.exit. bar() could recurse until the stack is exhausted. bar() could suspend and not resume. The OS could kill bar() from outside before it returns.
Who knows what shadows lurk within a function's stack frame?
Run it and find out.
The invariant which needs to be preserved here is that if bar() returns, the log will print, and equally important, if bar does not return, the log will not print.
That's what a good async system can preserve without knowing what's inside bar.
foo();
std.log.info("main", .{}); foo();
_ = async foo();
They could instead require me to explicitly write `await bar();` within foo(), and I could find comfort in the "no hidden control flow" tenet of the language once more. I recognise now that this would break the colour-blindness of their async implementation and that this is a pretty significant trade-off, but my perspective remains that this breaks one of the rules they've laid out on their homepage.
The details are not fresh, but it went something like: if a function called an async function like so `the_frame = async foo();`, you couldn't really reliably `resume the_frame;` unless you were responsible for all suspend points hit inside `foo`. If `foo` tried to call some other function `bar` you had no idea had a suspend point, you couldn't resume the frame anymore since you'd need to resume the inner frame instead.
So, yeah, they were colorless as in you could call any function you wanted and they don't advertise suspend points or async usage, but they are actually colorful inside and calling the wrong color breaks things.
ISTM this could be fixed in the type system, possibly at the cost of breaking some existing working code.
./src/main.zig:42:32: error: expected type 'fn(*u64) void', found 'fn(*u64) callconv(.Async) void'
const fns = [2]@TypeOf(blue) { red, blue };
After that, changing the array to be an array of @TypeOf(red) gets you a working program. That is, it's apparently fine to store blue in a value of red's type and use blue with @asyncCall as if it was red's type at runtime, but trying to store red in a variable of blue's type is a compile error.
here's a gist demonstrating this stuff that a user posted in the Zig Discord: https://gist.github.com/sharpobject/49bd88416e68606d7812d609...
The zig "colourless" innovation was informed by my work with cqueues in Lua[1]; which I gave as an example of good colourless-ness.
[1] http://25thandclement.com/~william/projects/cqueues.htmlThe issue that the function color concept illustrates is how subtle function incomposibility can seem perfectly workable for a while, but sometimes leads you to a deep dead end that takes a lot of reworking of otherwise good code to resolve.
The key limitation is that “red” functions can only be called from other “red” functions. So that if you’ve developed a bunch of code composed of “blue” functions, possibly with deep, complex call chains, and then find you need to call a “red” function at a low level, you suddenly find yourself in a dead end with no good way forward.
This article seems to drop that key point from function colors. (I say seems because it only speaks of its own definition indirectly. Maybe it’s in the deleted post mentioned at the start?)
I think that makes this whole article pretty much pointless.
The definition that I use is the bolded one near the beginning:
> some functions cannot be used without special work.
As-is, bringing it up just muddies your argument.
Critically, function colors are an issue of the language design and here you’re finding limitations in the language implementation.
I heavily disagree, but near the end of the post, I also show that Zig does fit the original definition:
> Third, async is, in fact, viral. The language reference says,
>> Zig infers that a function is async when it observes that the function contains a suspension point. Async functions can be called the same as normal functions. A function call of an async function is a suspend point.
> (Emphasis added.)
> In other words, if you call an async function, your function gets a suspend point, which means that your function becomes async.
> In fact, if you look at Bob Nystrom’s point 3, “You can only call a red function from within another red function,” this means that Zig fits point 3!
> What Zig does differently than Bob Nystrom’s “hypothetical” language is that if it sees you call a red function from a blue function, it shrugs its shoulders, makes the blue function red, and does not tell you. This gives the appearance of being able to call red functions from blue functions while still only allowing red functions to call red functions.
In zig async functions can call async and non-async functions, and non-async functions can call async and non-async functions. Thus, no “color” distinction.
It’s just pointless to ascribe the existing terminology of function colors while fundamentally changing the concept to something else. You end up making no point at all.
But Zig's language reference is clear: when needed, blue functions are turned into red functions transparently at compile time. What that does is make it seem as though there are no colors. That's why the dichotomy can be exposed at runtime: the compiler can't intervene and turn blue functions into red functions.
No matter how much you say that changes the definition, it still means that blue functions cannot call red functions. The fact that functions are turned into red functions does not magically make the colors disappear; it just hides their existence until it cannot be hidden anymore.
If color meant function signature, why define color?
Now, I don't know much about Zig's async story to comment on this post, but this is exactly my problem with Zig's async.
Zig's top self-promoted feature is language simplicity. "You don't have to debug your knowledge about the language". But with async, that's exactly what I would have to do. I tried to read the articles about Zig's async several times but the idea never clicked in my head.
Maybe I'm just dumb.
> This difference of opinion is probably what caused most of the conflict I alluded to earlier, so I thought that I would lay it out, then make my case for it, along with the case that Zig has function colors according to my definition.
You're right, but it's not like they don't address that.
Besides where a sibling comment says I mentioned it, I also mentioned it near the end:
> Third, async is, in fact, viral. The language reference says,
>> Zig infers that a function is async when it observes that the function contains a suspension point. Async functions can be called the same as normal functions. A function call of an async function is a suspend point.
> (Emphasis added.)
> In other words, if you call an async function, your function gets a suspend point, which means that your function becomes async.
> In fact, if you look at Bob Nystrom’s point 3, “You can only call a red function from within another red function,” this means that Zig fits point 3!
> What Zig does differently than Bob Nystrom’s “hypothetical” language is that if it sees you call a red function from a blue function, it shrugs its shoulders, makes the blue function red, and does not tell you. This gives the appearance of being able to call red functions from blue functions while still only allowing red functions to call red functions.
So I argue that Zig actually does fit point 3.
That is what I mean by viral: if a function low in the call chain is turned `async`, every function above it is too.
You can call an async function in a sync context. You just can't unwrap the value from the result container, as it's not available. But you do get a result, which is Promise<T>.
Any language that has concurrency, pointers or any mechanism that allows the value to change from under your feet has colored functions, doesn't it?
Also IMHO: Not being able to use "await" or not depending on the context, and wrong usage being a parse error, is much better than black magic happening at the compilation.
On the face of it those 2 things seem incompatible. Why are they compatible?
If Zig followed the C philosophy, then your async code would basically look like nginx, i.e. state machines and callbacks. Not saying this is desirable, but it does work (for some definition of "work").
Where is such blog post?
The Zig part has a number of updates and clarifications that contribute helpful balance while not disrupting the original text. Quite a few of the points have ended up being superceded by developments and evolution in the language.
It was discussion on lobste.rs, not a blog post. I would rather not link to it because I'm not proud of what I did.
But you could find it if you try.
Is that really much different than how python abstracts away types?
As an aside, while Zig really does end up with some detectable differences between functions that are async and functions that are not, various other implementations of similar ideas like greenlet or Lua coroutines do not have these issues. The author maybe did not look into those implementations because they don't call themselves `async`, but this is a superficial detail. In Lua for example, `suspend` is written `coroutine.yield`, `resume` is written `coroutine.resume`, and `await` is written by calling `coroutine.resume` in a loop until the coroutine has errored or returned. The author could conduct a similar investigation of Lua coroutines and not run into any of the same issues because there really is only one calling convention in Lua (as far as call sites written in Lua are concerned, anyway). C primitives like ucontext_t or those exposed by libco also expose the same operations without introducing additional calling conventions.
The current system of calling conventions in Zig is imo somewhat troublesome because "async/not async" and "inlined/not inlined" should be independent concerns. This has caused some issues in my experience where using `suspend` to jump out of a search when a solution is found causes much worse codegen than the alternatives because the compiler ends up emitting many more distinct functions.
I think async should be reserved specifically for stackless coroutines runtimes.
Edit: I guess I mean to say that Zig could in the future remove the async calling convention (but keep the fact that each function is either async or not) and switch to a model like libco without breaking very much code. If this does happen, I don't think it's necessary to rename the keywords at that time.
The original article contains an anecdote that describes a limitation specific to JS, and a couple of opinions that aren't universally applicable. Sometimes it's nice to have magic that sweeps away implementation details, sometimes it's good to have full explicit control.
Maybe Zig could add something to these pointer to mark them async?
void green(int p)
void yellow(char k[10)
Under his definition, these are colored functions. The compiler will create colors for them, and you can only pass them to places that expect these colors.
I don’t think it’s a useful definition of color.
color != function signature.
Put in other words, what is the Structured Programming Theorem equivalent for structured concurrency?
I have no proof of it yet (I want to create the proof), but I believe the structured concurrency equivalent of the Structured Programming Theorem could be something like this:
> For all possible programs in CSP style or in async style, there exists a program that uses structured concurrency that will accomplish the same result.
Maybe, but later in the post, I show that Zig does, in fact, fit Bob Nystrom's definition:
> Third, async is, in fact, viral. The language reference says,
>> Zig infers that a function is async when it observes that the function contains a suspension point. Async functions can be called the same as normal functions. A function call of an async function is a suspend point.
> (Emphasis added.)
> In other words, if you call an async function, your function gets a suspend point, which means that your function becomes async.
> In fact, if you look at Bob Nystrom’s point 3, “You can only call a red function from within another red function,” this means that Zig fits point 3!
> What Zig does differently than Bob Nystrom’s “hypothetical” language is that if it sees you call a red function from a blue function, it shrugs its shoulders, makes the blue function red, and does not tell you. This gives the appearance of being able to call red functions from blue functions while still only allowing red functions to call red functions.
Not only does this mean that I can express arbitrary constraints on callers, it also means that all constraints and stateful expectations are explicit, verifiable, and clear to the reader.
The post is interesting because it highlights that, colorless or not, functions have constraints and you can try to hide them and handle things automatically, via inference as is demonstrated here, but they're always there. Async is probably one of the less interesting constraints, but I find it so much more confusing when languages have colorless functions... I can't see suspension points, I can't tell wtf is going on. It's extremely frustrating.
> However, before I continue, let me say a little bit of praise for Zig: this is innovation. This is a piece of good work that, most of the time, async functions can be called the same as normal functions.
It's definitely cool and I like the definition of async they have. I think suspension is probably one of the less interesting things that a function can do, vs, say, IO, which is similarly "colorless" in languages. I just wish there were colors for all of these things.
Another thing that you could do is in the command pattern for undo/redo, to make sure changes to your data model only happen from command objects. In Python, I had a decorator that said "when this property is set, assert that Action.do is on the call stack". It would be really powerful to be able to do that at compile time, too.
There’s also Haskell’s system, which IIRC uses an IO monad.
The advantage of something like JavaScript's async/await over Haskell's "do" is the fact because async/await is more limited than `do` (or yield and coroutines in JS) it's a lot easier to create tooling for.
I really don’t get how is Haskell “behind”.
What kind of tooling are you referring to?
https://www.academia.edu/71720554/A_Modular_Verifiable_Excep...
So here you go, http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p070...
As for being useless, well that is the usual argument from no exception, no RTTI folks, that would be better off using another language instead of messing up with C++'s design.
I recently tried Go to speed up some Cloud Functions that use Firestore, it's very difficult to know which of the chained method calls of the Google-provided SDK is the one to actually "blocking" the green thread, perhaps they are all are unnecessarily? Without studying the source code, it's hard to know. I prefer explicitness over absolute simplicity. It also makes it easier to optimise code in my opinion as it's easier to identify sequential async calls that could actually happen concurrently, either by merging SQL queries or by using Promise.all() or the equivilent.
JavaScript does a good job with the await keyword in my opinion, you know exactly when something could take a significant delay, when it's doing something more behind the scenes. Rust does one better by making ".await" a suffix, allowing for nifty call chains. It communicates that this is an explicit yield point and could take a while.
Then you keep pushing stuff to other threads "just to be sure", which may hurt performance but the worst part becomes reading that code.
Well it also sucks if unpure code has to add a bunch of boilerplate annotations to every single function and call.
> I recently tried Go to speed up some Cloud Functions that use Firestore, it's very difficult to know which of the chained method calls of the Google-provided SDK is the one to actually "blocking" the green thread, perhaps they are all are unnecessarily?
If you have to mark enough of your functions as "possibly blocking" then you can get just as lost about which one is actually blocking in a particular situation.
Since this could be read in a very abstract, purist way, I'd like to add that this is also very much an architecture concern for many pieces of code.
At work, I regularly encounter issues with a piece of code that is written in one of the worst imaginable spaghetti styles. And, since it changes things in a database (not even in proper transactions, btw), and is written to pretty much prevent being tested, no one wants to touch it.
Yet, what the code does would really match an "FP" approach:
- Read in all inputs. You might even read during computation if you're simulating FP through transaction isolation
- Compute the outputs and the new desired state
- Pass that to someone who writes it back to the database and commits the transaction
Outputs in our case are messages that get stored in the database, too (in order to allow recovering after transmission issues), so "output" in effect also just means "state". You don't always have to read "FP" in the strictest sense -- there's a spectrum to this.
Now, if that code was written in such a way it would be easily testable and thus refactorable. In fact, most ideas in the clean/hexagonal architectures or "functional core, imperative shell" are to achieve exactly this. But since the author of said code never had any pushback, they just kept making an already grim situation worse with every change. For years.
I see the same issue with the "ice nine" argument for async. If async infects all your business logic, chances are your abstractions suck.
Edit: Also note that goroutines do effectively the same thing as synchronous code in a larger asynchronous function call: You group the code you can sensibly do in one go into a single goroutine. Then you pass the result to someone else to perform the next step.
So, it's very often just another way to express the same process. CSP will invariably be a better abstraction for some problems, and async/await for others.
Now you will want to blow your brains out because you will have to refactor all of your function calls down the chain.
It could get even worse if at some point, you are passing a function to a library/framework and for whatever reason it refuses to tolerate a function of your color (assuming you can't lambda-ize the color). Now you have to rewrite the library. Or use a library that does tolerate it. Which might lose other features or ergonomics.
If we really need to be explicit, we begin to need different libraries for each combination of effects/capabilities/whatever, sort of like how Rust users need at least 4 separate sync/async/nostd-sync/nostd-async libraries just to speak websockets. This seems like a waste to me.
Zig's error system in which errors must be handled and functions can opt to have their error sets inferred by the compiler (again, this includes specialization to different error sets for instances of the function that are parameterized by functions with different error sets) sort of gives us the same thing for errors that we have for async-ness, but I can sympathize with a perspective that feels that special casing these two things and not allowing any others is inelegant or whatever.
> Async is probably one of the less interesting constraints, but I find it so much more confusing when languages have colorless functions... I can't see suspension points, I can't tell wtf is going on. It's extremely frustrating.
Zig's view on this, which I endorse, is that suspending to your event loop is not substantially different from suspending to the kernel. Zig sort of assumes that we don't want to manually mark each function with which syscalls it and its callees are allowed to make, but maybe it's worthwhile to make a language that does do that.
Based on what I've seen in Rust's async, you don't actually want this in a low level language. Those calling scenarios are fundamentally different. If you magically make something async, other code can now suddenly mutate state that your function is referencing on its own stack and really mess things up.
In Rust this isn't an issue because you're not allowed to pass references into an async function that outlive the future. If you take a synchronous function that holds references and add the async keyword, you probably will get compile errors about this. I don't think Zig has anything like that.
>Zig's view on this, which I endorse, is that suspending to your event loop is not substantially different from suspending to the kernel.
No, that doesn't make sense, not even in the context of Zig. Suspending to the event loop allows other coroutines to run. Blocking in a syscall stops the whole program and all the coroutines.
But to address your comment directly, no it's not, not when you're actually building a different state machine with different constraints each time. I actually wish we could put this protocol code in one C library in a way that only handles the protocol logic and not the I/O part, and then we'd be done with it forever. But you know that wouldn't be good enough for some, those libraries exist and people still don't use them and instead they go and write their own in the new fancy language with their cool new concept for a state machine.
Yes, this is a great approach. Unfortunately people in e.g. the Rust world tend to write crates that assume they can call malloc, recv, and send, rather than one crate just implementing the protocol logic and another that does all that other stuff.
Zig's way of doing things causes the library people naively write and use to call malloc, recv, and send to also be a library that just implements the protocol logic, reads from and writes to static buffers, and doesn't call malloc.
Edit to reply: Yes, Rust has no_std, but most libraries that implement a protocol are not divided into a no_std library that implements just the protocol logic and a separate std-using library that wraps it, nor are most libraries that implement a protocol asynchronously wrappers around a no_std library that is not async and only implements the protocol and makes 0 syscalls. A consequence of the fundamental non-composability of naively written Rust libraries is that users who want the wrong combination of asyncness and malloc-calling very frequently need to reimplement existing Rust libraries. A consequence of the fundamental composability of naively written Zig libraries is that they never need to do that.
You can have one Python library at least[1], h11 and h2 in particular make a pretty convincing case as the ultimate in Python HTTP. (You will always have wrappers, though, because not all levels of detail are appropriate for all uses—similar to how imagemagick/libgd/etc. wrap libtiff which wraps libjpeg and wrap libpng which wraps zlib and so on.)
The problem, as far as I can see, is that to avoid calling out to I/O you need inversion of control, and either you convert your code into CPS and state machines manually, which almost impossible, almost unreadable, and makes layering libraries a distant dream (not that people don’t do it anyway), or you use some sort of generic inversion-of-control mechanism like async/await or call/cc, which really wants dynamic memory allocation.
It seems like being able to accurately track stack usage (per module, not per program) could solve this, but that means (we must abandon arbitrary recursion and) the compiler must be able to track arithmetic equations, which is liable to make Gödel and Turing rise from their graves just to shout NOPE! at you. There’s also ABI: what do you do when a new version of a protocol library requires a dozen more bytes on “the” stack, and you were putting these on your stack or in static storage?
(As far as I can tell, Rust’s no_std answer to ABI of this kind is “no.”)
To that end, Zig does have another kind of function coloring that's useful there: functions which take an allocator parameter. I like that about Zig, I wish C++ and Rust were better about doing it. Sadly you can still hide this by referencing a static allocator in Zig so it's not totally there yet on the language level.
The problem is that the library author must then code for the worst case and the compiler won’t help them; injecting preemption into synchronous code can expose things about it that used to be unobservable, but now can make the difference between success and deadlock. (Similarly to how injecting mutation into pure code can expose—and bar the optimizer from changing—things that were previously unobservable, like “in which order does map() visit elements?”.)
There still will be async-compatible and async-incompatible libraries, except now you can’t tell which one is which except by looking at the documentation—look at the lengths R5RS goes to in order to define things like returning multiple times from inside a letrec while R6RS just flat out prohibits that.
Yes, this seems to be a pro-colouring comment while elsewhere in this discussion I make what seem to be anti-colouring comments. I really don’t know what the answer is.
But continuing on your idea, one colouring I would really like to see everywhere is pureness. C++'s `const` comes close, and perhaps Haskell does it right by having everything be (mostly) pure by default and using the type system to encode other properties. And yeah - different monads (like IO) are likely a good way to handle this whole topic in a user-definable way, but not many people like them (at least when they know they are using them, because unknowingly many programmer do use it, eg. most async syntactic sugar is basically a do notation for a single hard-coded Monad)
the async/non async functions in javascript are good and well designed. the whole colored functions thing is nonsense and it's weird that it gained so much traction on here
Those two groups are minorities in the set of all people that are even aware of monads.
Imagine you have 2 singleton objects in your 'void main()', redkey and bluekey of types Redkey and Bluekey.
Somewhere else, you declare your function: 'int foo(Redkey redkey, int x, int y)' that needs a Redkey object of which only one exists: in your main.
This by itself forces every call on the path between 'main()' and a call to 'foo()' to also include a parameter Redkey.
In the extreme cases where a pattern like this is useful (tracing IO calls, you named it) it can really help cut through a codebase after an hour of refactoring. But it can be limiting. Async and Checked Exceptions are probably the most colored functions and they both need escape hatches because of that.
Check out Koka lang for a very interesting implementation: https://koka-lang.github.io/koka/doc/book.html#why-effects
Sadly I don't see that happening anytime soon.
You see the same thing happen in async circles. All of a sudden every single function has to be labeled async, whether it needs to or not.
And later on, checked exceptions were incompatible with new stuff like lambdas and streams
Zig actually does pretty well on this front. you can write a library that accepts functions that return errors, and your functions will be specialized to return those errors in addition to their own if you don't handle them. Callers who want to handle errors exhaustively with a switch will get compile errors if the error set changes and they fail to handle new errors.
Haskell’s use of monads is perhaps the ultimate in colouring (for values not functions, but that doesn’t make that much of a difference); you routinely colour everything with which pieces of state it needs to read and/or write to do its thing. The result is you either have everything painted in “global state”, which just feels like unnecessary boilerplate, or you spend most of your time defining and converting between elaborately carved pieces of said state.
People could and did invent various intricate constructions which would do the conversions and possibly even the carving for you, but overall I think most agreed it was a gigantic pain.
It’s possible the situation is fixable with a correct combination of features (records, polymorphic variants, implicit parameters, possibly even algebraic effects), as a good portion of the pain comes from not having a standard solution every library would use (they all have drawbacks). But then the checked exceptions story in Java, IIRC, is from pre-Java 5 times, and not being able to be polymorphic in the exception signature of a callback you take is really stupid however you look at it, as well as also a case of inadequate language features.
In a similar vein, as another commenter above said, really the only problem with function coloring is that we're not treating it polymorphically.
public class SomeClass<E extends Exception> {
public someMethod() throws E {}
}
The other problem with this approach is that it requires the caller to explicitly specify all the exceptions thrown by the function/object being wrapped. So every time you do map() or fold(), you have to spell out the full exception spec for the closure you're passing in. And god help you if there's nested maps...
For example, one's expectations of a pure function would mean that you could do some arbitrary numeric computation on inputs and return a value. Unfortunately, the compiler is not so naive. If you should do some transcendental math and call a standard library function to do so, you have manipulated hardware state. You are now impure, and so are all your callees.
And so you have to decide at that point: is the code you are writing wrong? Or is it just using too coarse an assumption for its invariants?
A huge amount of what determines our code architectures are the way in which we've organized our own thoughts. In this respect we always go from looser to tighter specification over time as the code matures, until tightness has produced brittleness and cracks. I believe coloration forces some tightening decisions to be scheduled earlier. In that respect it should be treated with some suspicion, as it may produce surprising couplings.
The problem of colour is rather that even though two functions should have the same structure, regardless of their effects, the language still forces you to write two functions each with its own syntactical structure. In type theory terms, this is a problem stemming from a lack of effect polymorphism.
Ideally you'd write one function, polymorphic in its effects, and then just be able to use it in both sync and async contexts. In the sync context, the function's type would specialize in a way that it lacks the `Async` effect, so it wouldn't be able to do async stuff there. In the async context it would specialize in a way that it does have the `Async` effect. Therein you would be unable to apply it in contexts requiring the lack of an `Async` effect.
You denote that the color is optional in the function's type signature, and inside the body of the function, you have a construct that lets you execute different code depending on the state of the function's colors. The call site could either automatically infer the colors used or the colors used could be manually specified.
Basically a type parameter with compile-time if statements.
It makes sense to avoid this in a javascript runtime because you actually almost always want (3) there. It's too late for node to change it unfortunately. But I'm also pretty confused as to why zig, a (relatively) low level language, considers it important to remove this distinction.
Let’s take the example of JS. Does JS have function colors?
In one sense, yes, there are async functions and sync functions and you can only call an async function from an async function. So under a certain definition, JS functions are colorful.
But wait... a JS async function is just a normal function that returns a Promise object. So under another definition, JS functions are colorless. Perhaps someone might not like how Promises work in JS, but this has nothing to do with what color the functions are.
There are some languages that do break this and implement async functions as a completely different entity to normal functions. I believe this kind of function colors is lazy and bad language design and these async implementations tend to feel like they don’t belong with the rest of the language and were duct taped on.
The two scenarios are the same, yet in a colored functions language f' will need to be annotated if r does, for example io.
The way I see it, a language should either allow 'closing' over the color (or effect or capability, or whatever) hiding it, so that f' doesn't beed to care about the details of r; or it should provide enough of polymorphic behaviour so that f' the signature of f' can be inferred from that of r.
Most languages with async do not provide this capability.
> Most languages with async do not provide this capability.
Those where async is syntax sugar for something else (e.g., JavaScript, Python, I forget if C# qualifies this way) allow hiding; you don't need annotation to call an async function, you only need it if you choose to use async syntax in the calling function.
> you don't need annotation to call an async function, you only need it if you choose to use async syntax in the calling function.
The proposed thing is "I want to get a function, and call it, and use the result, and if it is effectful in a way then I am effectful in that way, and if it is not then I am not" where "the result" is defined (among other things) as the value of the expression that appears in a return statement in the body of the function. Redefining "the result" to be some object representing a continuation or whatever is missing the point.
The lack of async-hiding in JavaScript is why the original "What Color is Your Function?" blog post exists.
Despite not liking the idea of function colors, I do think I like your idea of having functions express what capabilities you have to have permission for to call them. A long time ago, I saw someone talk about fine-grained permissions within a programming language, and I think they are the same idea.
I would love if that could be done without function colors, though.
In the "real code" I linked to in the post, I have a concept I call "context stacks". The idea comes from Jai, and I use to do implement things such as error handling and allocators in such a way that functions don't need to take an allocator argument. To make callees use your allocator, you push your allocator onto the allocator context stack and then call the functions you want to use that allocator.
I made it useful for many more things, and users could add their own.
In particular, I realized that it could be extended to where functions could get permissions (such as popping up an Admin dialog box on Windows or asking for sudo on Linux), push a capability [1] onto the capability context stack, and then call the functions that actually do the work. These functions could then grab that capability and do what they need to do.
One good thing about this is that permissions don't leak up to callers that should not have them. In other words, this system keeps permissions from infecting code that really shouldn't have them, but might need them because they might call functions that do.
I don't know if all of this made sense, sorry if not. I wrote it really fast.
[1]: https://en.wikipedia.org/wiki/Capability-based_security
for great justice