Show HN: A new stdlib for Golang focusing on platform native support | Hacker News Reader

Show HN: A new stdlib for Golang focusing on platform native support

github.com

154 pointsLeotard696382 comments

No gc, No goroutines, Produces small binaries while using the unmodified official go toolchain, and comes with complete Web SDK (generated from w3c/webref).

We are building `pcz` to provide a reimagination of Go the language, in an effort to make it suitable for all kinds of programming tasks, and currently you can use it to build efficient web applications in Go using the generated Web SDK (as shown with the live web demo[1]).

The journey is just starting, any suggestions? or any critics?

[1]: https://primecitizens.github.io/livedemos/10-plat-web/

Open articleView on HN

I love how they clearly state their goals at the beginning. This makes it immediately clear what it is about. Especially an importantly:

"Non-Goals"

"Change the way the Go team works or make them change their decisions."

Not every project (and very much not every fork) is a political statement against the original. Sometimes people are glad what others do and want them to continue. Still, wanting their own different version to coexist is completely OK.

I don't think the goals are explained very clearly. Or, to put it another way, after the goals were explained I still had no idea what it actually does or replaces.

Sorry to cause the confusion, we are building a stdlib for Go, and all these goals are related to the development experience when using our stdlib, and we are not aiming at anything like replacing the official go stdlib (which we also enjoy work with).

Were the confusion caused by the choice of project name `pcz` instead of `xxxstd`? or any suggestion for us to improve?

I think the confusion is caused by the notion of a separate library that is also “stdlib” - which is the version that comes with the official distribution.

This comment is the first time I realised it was not intended to be a drop-in replacement for the Go standard library (in a musl - glibc kind of relationship).

You are right, the pcz std won't be a drop-in replacement to the official std as they have different modulepath.

We decided to make it this way so:

1) you can import packages from the pcz std module like you do with any other Go package without setting `GOROOT` to an unofficial std.

2) and because of 1), you can import packages from pcz std in your go application but still using the `go` command rather the `pcz` command.

In the future, we may provide drop-in replacement for applications using official std by introducing a compatibility module and a cli flag to redirect imports using packages from the official std to that module.

There are a number of languages where there's been an alternative standard library developed, the problem is 'by stdlib do you mean standard-for-the-language or standard-for-you-to-code-against.'

But there was e.g. https://github.com/eloraiby/alt-std for rust, and for D there's a couple listed here - https://wiki.dlang.org/Alternative_Standard_Libraries

So if you're coming at it thinking in those terms, it's 'obviously' the right term to use to describe it.

Note that (a) I wasn't confused at all (b) it makes complete sense people -are- confused by it, and you're not at all wrong.

Actual stdlibs tend to be magic. Take Rust's NonZeroU8. This type is an unsigned byte which isn't zero, but as a result the bit pattern where zero would fit is unoccupied and thus Option<NonZeroU8> is also a single byte. This allows code which has the safety and convenience of modern type safety where we can't screw up and forget that None isn't just zero - yet also the performance of a C-style sentinel trick where you just treat zero as indicating None, maybe using macros.

But, you can't make such types in your own Rust. If you wish you had NonSixU8, a type which can't be six, the only way to make it would be to base it on NonZeroU8 and use some sort of XOR scheme. The standard library is blessed with the ability to do this even though it's not available to mortals and that blessing is only open in practice to the actual standard library for the entirely practical reason that only they can co-ordinate with the compiler team to ensure it happens - if the Rust compiler re-defines how niches work the library team have to change NonZeroU8 at the same time.

So while the actual stdlibs get to do magic, which makes them stand out from a popular third party library like SDL or zlib, these "alternatives" don't.

If they provide all the same stuff as the "real" standard library that makes sense, but when they instead provide different stuff and they can't do magic I don't see any difference from any other library. Is SDL an "alternate stdlib" if we and some friends all use it all the time?

One of my favorite parts of Rust is just how little magic there is. Go to the docs for NonZeroU8 and click source. The code is macro heavy to template out all the different nonzeros so it's not very readable to beginners. But you can see it uses the attributes rustc_layout_scalar_valid_range_start and rustc_nonnull_optimization_guaranteed.

Those are unstable, so you have to use another attribute opt in to using them. But you can absolutely put them on your own type.

There's nearly nothing in the Rust stdlib you can't write yourself. OIf I recall correctly one example is the function Box::new which constructs a heap allocated smart pointer to it's argument. It "magically" skips allocating the argument on the stack of the caller in some situations.

> Those are unstable, so you have to use another attribute opt in to using them. But you can absolutely put them on your own type.

They are, as their name suggests, deliberately perma-unstable rust compiler (rustc) only. You technically can use them in types anyway (and if you go look at the nook crate, I have, that's why I know so much about them) but because they're perma-unstable that's a terrible idea and you clearly shouldn't use this.

Even if they were just unstable, (like say const traits) you can only do this in nightly Rust, it's not available in stable Rust and as the people who'd decided "Oh, const traits, those sound great, I'll just use them and eventually they'll be stable" discovered, nope, that's not what we meant when we told you they're not stable. We meant what we said, and those traits just plain went away (probably they'll be back eventually, but maybe not).

But I agree the stuff that's "just" unstable isn't compiler magic, after all the intent is, even if it doesn't always work out, that those features eventually get stabilized. These are perma-unstable because they're compiler internals. They are magic.

My bad

I think it does what it says on the label but maybe I'm missing something.

The major question that the README should answer are:

As a Go developer, can I expect to be able to build modules that will work both with both Go's stdlib and pcz std (maybe using build tag guards) ?

Are you building a completely separate ecosystem or can we expect some overlap?

Thank you for your suggestions, but I'm afraid the best we can do is to list packages compatible with both std.

The problem is, in order to build a std, we have to use a lot of <ABIInternal> stuff in assembly, and use internal/abi.FuncPCABIInternal in Go, so we can not make any promise regarding to the inter-std-compatibility for the time being, unless the Go project decided to make these features stable.

After looking at the page it seems pretty clear overall but the biggest source of confusion to me is emphasis of "a new stdlib". To me it seems like a good chunk of their goals start with "a new compiler/runtime". They are replacing the go command.

The fact that requires a different standard library seems like a consequence of the first decision but it's unclear.

Interesting, I don't fully understand the goals.

I have certainly encountered issues where Go is hiding the platform from me and this is very annoying. This was only ever a real problem on Windows though, the "virtual platform" that Go provides is generally fine for Linux IMO. Is this mainly about non-Linux platforms? (Which would be perfectly reasonable, BTW. Linux is not the only platform )

Meanwhile, I can see why someone would be interested in no-GC. But this seems orthogonal to the goal of "adapting to the platform" - am I missing something there?

And on the no-go routines thing, what's the motivation for wanting that? Does their presence in the language have some related impact on it? Or is it just that you don't think they're useful and dropping them lets you focus on the other stuff?

"adapting to the platform" is mostly related to the goroutine topic, as we are working on the js/wasm support and these applications prefer async/await operations.

re: goroutines

the official go runtime spawns goroutines during program startup, which makes the usage of M:N goruntine mandatory, and they have to call lockOSThread()[1] before initializing packages for certain applications (mostly GUI application).

We like the design of goroutines (especially the idea of a `g` register), but `pcz` currently doesn't have goroutine support, and we are making it possible to spawn goroutines with custom allocator (with or without gc) and scheduler attached, so you can chose 1:1, M:N model for goroutines on your own (as decided by the scheduler attached).

you can find more details in the project ROADMAP.md[2]

[1]: https://github.com/golang/go/blob/352c8835e7609ad72872b5a63b... [2]: https://github.com/primecitizens/pcz/blob/master/ROADMAP.md

Reminds me of https://tinygo.org/ - a project that brings Golang to embedded devices, browser (wasm) contexts. Do you converge or diverge from that project?

> converge or diverge

Neither, though the final result can be similar, but at project level, tinygo and pcz has very different approaches to achieve the final result:

- tinygo chose the hard path to write a llvm backed toolchain with some custom behavior that compiles Go code (using official std) to llvm targets.

- pcz chose the easy path to write a stdlib from scratch that compiles to desired targets using the official Go toolchain.

One major benefit of pcz's approach is you can import packages from the pcz std module like you do with any other Go package, the development experience can remain unchanged.

I'm glad seeing this, but without GC and goroutines, then I'm wondering why not use something like Zig instead?

Go is such a simple language, you take out goroutines and you only have to reason about a single threaded application with no concurrency, I would guess this lowers the bar for doing platform native coding.

Zig is much more complex than go, I’m sure there are a lot of people who don’t want to put the time in to learn it.

Why do you think Zig is much more complex than Go? I'm not sure that I agree, except for the low-level stuff on things like alignment, packing, etc.

I'm still a little confused after reading through the goals. It seems like there are a bunch of different unrelated goals:

  - better js/wasm support
  - "platform native sdk" (I don't know what this is)
  - cross platform GUI
  - Better FFI support
  - Kernel support

While all these goals seem cool on their own, they also seem completely unrelated. It makes it a bit hard to understand what this project actually is.

"platform native sdk" is the collection of apis exposed on that platform exclusively (think win32 apis, android ndk).

The project is to build a stdlib, and by its nature is to be used by developers to build applications just like the official Go std.

Can you expand more on the ffi piece? Using CGO has never been fun. Will this expose threads and allow for C interop that is easier to reason about? Can I dream of using this version of Go for all of my "wrap this C lib in a nicer Go experience" use-cases?

Yes, that's one of our motivations to build this std!

The feature of generating go packages from C in `ffigen` still needs significant work to be done (mostly dealing with C calling conventions on different platforms), and the generated go package only works with dynamic libraries (as it uses the pragma `go:cgo_import_dynamic`).

One of Go's big selling points is the avoidance of the "red-blue problem" (https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...). The standard Go implementation does that by using lightweight "goroutines" that automatically suspend themselves whenever they do some operation that might block - there's no need to use an OS-level thread for each thread of control.

I wonder how the authors plan to accomplish a similar thing while remaining fully FFI-compatible.

Note that this project is not an attempt to replace the Go runtime, but here is what we will do:

Precondition: in Go, each goroutine has its own `g` structure (like TCB, but for controlling goroutines) and its address is stored in the `g` register, which is reserved by the go compiler, active goroutines can call a function runtime.getg() to obtain the value of the `g` register.

In the official runtime, the `g` structure is a concrete type and the scheduler is implemented globally in the runtime package.

But we have made the `g` structure customizable (still requires a fixed header expected by the function prologue & epilogue and linker), so custom g implementations may contain a link to a custom scheduler, and the scheduler can decide how to handle an active goroutine.

> "goroutines" that automatically suspend themselves

The automatic suspend and resume happens by

- calling `runtime.entersyscall` from a running goroutine - and calling `runtime.exitsyscall` from an finished systemstack

all these operations can interact with the custom `g`, thus can interact with the custom scheduler.

Is the new std lib a re-implementation of the existing std lib or are you defining brand new interfaces as well? For example, will you still have an `os.WriteFile` and will its still take the same parameters and return the same results?

Yes, we are re-implementing (some of) the official std, but with careful redesign, and probably won't provide a lot of familiar apis, this is largely because packages from the official std often do implicit allocations with the assumption of GC running.

And Yes, we are defining brand new interfaces as well, in order to improve usability and fix inconsistent user experience.

For your specific example `os.WriteFile` (names mentioned below may subject to change):

The api won't make it inside the std, because the local filesystem will be implemented as a `fs.FS` (concrete type `*sysfs.FS`), so you can have multiple `sysfs.FS` instances each at its own working directory, and the `os.WriteFile` will be replaced with `fs.WriteFile(fsys fs.FS, path *fs.PathBuf, wctx *io.WriteContext) io.Status`, where `fs.PathBuf` is our solution to handle all kinds of path styles (windows, unix, cygwin, utf8, utf16, end-null) and operations (join, base, dir).

Beautiful! I will be watching this project closely.

> No gc, No goroutines

You're taking away what makes Go Go - you're now building something that turns Go into Pascal, perhaps unwittingly? Curly braces are what's left, basically.

I get where this is coming from, sometimes I hate it when it comes to Not writing network services or servers in Go, it’s just pure dislike I would love it if there a way we can have a plugin-able runtime since the runtime is viewed as a library

So this is the std made for you, and to make it compile, you have to import the runtime package explicitly.

One thing I do not understand is if there is no GC then how are stack allocated variables freed?in vanilla go there is no way to manually manage variable lifetime. Does this mean everything needs to be created with a a make?

just restart the server/app on a cron job

I have worked with a number of codebases where we did that.

In fact, I have worked with a number of codebases where I was the one who deployed that, because the app process could already do the relevant level of restart without dropping a request, and the leak was slow enough and hard enough to find that I couldn't justify keeping going until I -did- find it.

This continues to annoy me, but having made sure the leakiness wasn't able to -increase- over time without us spotting it immediately, I'm still convinced it was the business-correct decision.

(but, still, sulk)

I once worked on a project where a critical system started to leak memory during a critical weekend-long event. We basically took shifts every 4 hours to log into the boxes and manually rolling-restart the affected services.

We did find and fix the leak the following week once the event completed and traffic dropped to normal levels. But sometimes, you got to do what you got to do to keep the app running.

I was more thinking of some mad lads who turned off python garbage collector and just restarted the process when running out of memory. Apparently it gave huge performance and cost boost

Currently pcz is building applications with the `compiling-runtime` option, so variables implicitly escaped from the stack will cause compile error.

For variables you are sure living on the stack, you can use `mark.NoEscape(&v)` to obtain its pointer (a hack to cheat escape analysis found in the official runtime & reflect package), and the variable will be freed on function return.

For variables you want to control lifecycle, call `alloc.New` or `alloc.Make` with a explicit allocator, which can be an allocator whose storage is on the previous call stack (and there is a default goroutine local allocator thread.G().G().DefaultAlloc()) to allocate a non-stack-local variable, and call `alloc.Free` with the same allocator to free that variable.

The use of `make` is discouraged as go compiler does mid-stack inlining and escape analysis, so the return value of a `make` can be either on stack or on heap, thus to free it, the allocator needs to compare the address to thread.G().Stack.Lo and thread.G().Stack.Hi.

Goroutines was the selling point for me until they decided to introduce telemetry in their toolchain; that was what forced me to stop using Golang as a whole.

About GC, I would say: if you implement C++'s RAII mechanism to replace garbage collection, then I believe this project will have a bright future.

My final question is the following: how `pcz` compares to V language, from a syntax's perspective [1]?

[1] https://github.com/vlang/v

Sounds like go routines was not a selling point if telemetry tipped you over then.

Telemetry is opt-in [0], it shouldn't be a concern.

[0] https://research.swtch.com/telemetry-opt-in

its mere existence is a concern. people are so dumb when they ignore these things. whenever there already is something, it can only go further.

Debian, for example, has an opt-in telemetry. Would you apply that to Debian?

Yes, I 100% would. To any software. The only positive in this case is that Debian is not run by the freaking Google - the overlord of all information on the internet with backdoors for all three letter agencies in USA, despite what Go authors will want you to believe.

GOPROXY is enabled by default. “Since Go 1.13, the go command by default downloads and authenticates modules using the Go module mirror and Go checksum database.” https://proxy.golang.org/

Only if you’re not vendoring your dependencies, afaik.

So you might just need an anonymizing relay to the Google proxy.

The GOPROXY protocol is quite small to allow to implement such a relay: https://go.dev/ref/mod#goproxy-protocol

Also using a generic http_proxy that does the relay you need might just be enough as many solutions exist in that space.

https://stackoverflow.com/questions/10383299/how-do-i-config...

re: telemetry

We don't like it as well, just do not enable it :^

re: RAII

It is not possible to add RAII support without updating the compiler to call specific methods automatically, so currently it is not an option (as we are using unmodified official toolchain).

But the `g` register defined in Go is of great value, and we are making use of it to provide custom goroutine support, which means you can have custom allocator and scheduler for specific goroutines, so that you can have some of them with GC enabled and others not.

re: compare to V

pcz is a stdlib (plus a cli tool to build), not a new language, you still write Go code but in a slightly different style.

What is that `g` register that you are talking about. It seems to be a reference to some internals of the Go. Do you have some references to some existing documentation about this thing?

I'm afraid there is no such resource, but you may find cmd/compile/abi-internal.md[1] helpful

And in brief introduction, the `g` register is a non-scratch register, and is preserved by the go compiler, it stores the poitner to current goroutine (type `g` in the official runtime, a structure serves the similar purpose of TCB), and since all general purpose registers are thread local, the goroutine may enjoy some thread-local features without any thread-local requirements to the running environment.

[1]: https://github.com/golang/go/blob/master/src/cmd/compile/abi...

What bothers me: Since a while the documentation links to cs.opensource.google with the typical google flavor. Some years ago, you could simply click on some function in the documentation and view the source code. When I do this now, I find myself on a permission denied page. Just a random example:

Permission denied.

Please contact the administrator.

https://cs.opensource.google/go/go/+/go1.21.1:src/strings/cl...

I don't know what causes this, but is seems to be related to my Firefox, because in Chromium I can see that page. I just wonder, what on earth makes it so hard to serve a documentation in a way that is available to all browsers.

For a counterpoint: I can see that page in Firefox. Probably related to some setting on your computer.

Are logged in to some google account?

I have faced the same problem numerous times. Sometimes the cs.opensource.google links work, sometimes they don't. I haven't yet bothered to try and understand why that's the case.

Why can't they use the github.com/golang/go link instead? (no pun, genuinely wondering)

1. github.com/golang.go is just a mirror. The main Git repo is available via https://go.googlesource.com/go

2. GitHub.com is Microsoft infrastructure. cs.opensource.google is Google infrastructure.

I still wonder why there is both cs.opensource.google and go.googlesource.com.

Because GitHub is a mirror not the actual repo, I think I was around 2015 when Go was gaining massive attraction they had to use GitHub for issues also, still you can’t change the fact that more than 70% of commits is from google

Do you have any privacy/ad-blocking extensions installed? I used to have the same issue and realized it was due to the DuckDuckGo Privacy Essentials extension. When I turn that extension's protections off for cs.opensource.google then it works.

It's source code. If we can't read it without privacy extensions, it doesn't work.

Even more: as reading on a machine that has the Go toolchain installed, this is source code that is already available on the local filesystem.

> If we can't read it without privacy extensions, it doesn't work.

What doesn't work?

Reading source code.

The same source code is also part of the Go distribution ($GOROOT/src). The Google source browser isn't the only place it's available.

Yes, you can find the documentation in other places and that is what I usually do, but it sucks because this is the official documentation of the programming language.

Having those links to quickly take a look at the implementation is what made this documentation so good.

1. I'm not always on a computer that has the go distribution

2. In that case, why allow reading the code un the browser ?

Well, yes I do use uBlock Origin as well as DuckDuckGo Privacy Essentials, but for that page I turned them off months ago. It still doesn't work :-/

Hey, DDG engineer here. Sorry about the issue, that’s super annoying. I want to get this fixed, but I’m having trouble reproducing the issue - navigating to the link in the parent comment seems to work as expected for me. Any chance you’d be willing to provide a few repro steps so I can track down what’s going on? Namely: - which browser you’re using and version - extension version - when it’s happening. Does it only happen when you click a link from another site?

Thanks so much for your time, and for sharing the issue.

Well, turns out I'm no longer able to reproduce the issue either. I just turned DDG Privacy Essentials back on for https://cs.opensource.google and I was able to view the site just fine.

Back when it was happening (maybe ~1 year ago?), I was using the latest versions of Firefox and DDG Privacy Essentials and it ocurred even if I went directly to https://cs.opensource.google. I had confirmed back then that when I turned DDG Privacy Essentials on I got "Permission denied", but with it off I was able to view the page.

I'm on the latest verisons of Firefox and DDG Privacy Essentials now. Seems it is no longer an issue in the latest version(s).

Thanks for following up! Appreciate it. We’ve been investing quite a bit of effort over the past few months into tracking down and understanding issues like this that are caused by our various privacy protections, so it’s nice to hear that things are improving.

What is the point of Go without garbage collection and goroutines? You can use C.

Golang is pretty good, but you can't really count on gc and frankly goroutines are as much problem as solution. Nice to see a C with useful types and builtins.

Can you expand on why/what situation you can't count on GC and why goroutines are a problem ?

Go has 25 keywords, C has 32; it's an easier language. Even without the garbage collection, Go's memory management is easier and safer (memory is automatically zeroed, no pointer arithmetic). Go's toolchain is portable and straightforward.

TL;DR it keeps having advantages over C even if you takeaway GC and goroutines.

That is like saying what is the point of Pascal, you can use C. Well, one is a (dearly loved by me) footgun.

They appear to be aiming for Go without -compulsory- garbage collection and goroutines.

Also go has a -lot- fewer sharp edges than C.

It's an (extremely) ambitious goal, and skepticism is warranted, but I think their desire for such a thing to exist does actually make sense.

At risk of derailment isn’t this a much better fit for Rust? I am saying this as someone who has used both extensively professionally and prefer Go in most cases.

If you’re dropping GC, then RAII is extremely useful (which Go doesn’t have). If you’re dropping runtime concurrency, Go’s (subjectively) strong multitasking, cancelation, timeouts go away. And finally, if you’re dropping std, which is perhaps the strongest suit of Go, then what are the remaining strengths of Go vs Rust? Tooling? Compile time speed?

TLDR a simple subset of Rust (enums, pattern matching, RAII, etc) sounds appealing, but a simple subset of Go seems quite… limiting. That said, I’m not an embedded programmer.

> dropping GC, dropping runtime concurrency

Making GC/goroutine optional & customizable requires building a std works with no GC/goroutine.

> dropping std

relax, this is a new std, let's spare the baby some hope?

> Making GC/goroutine optional & customizable requires building a std works with no GC/goroutine.

Right.

You've picked a goal that require you to do some hard things to make that goal even possible.

OTOH even if you never reach the -final- goal, making any of those hard things work would be an achievement.

The only way to find out is to try. So: Good luck!

Thanks, but the actually hard things are done by the Go team, the design of the interface type and availability of runtime type information made it relatively easy to complete that final goal, but let's go find out!

> Making GC/goroutine optional & customizable requires building a std works with no GC/goroutine.

Isn’t there a big difference between customizable, ie changing the runtime backend, vs NO runtime where goroutines and GC don’t work at all? In the latter case, you’d have to write very different, restricted Go code, no? And you’d even have to add some new features for manual memory reclaim?

> relax, this is a new std, let's spare the baby some hope?

Yeah, depends on what hat you put on. I wrote it wearing the “when would someone use something like this?”-hat. I don’t claim to know one way or the other.

Reply on news.ycombinator.com