Tcl Ported to Go | Hacker News Reader

Tcl Ported to Go

gitlab.com

174 pointsblacksqr99 comments

cznic (aka modernc.org) again…

This person or this collective have been transpiling C projects into Pure Go projects, like sqlite[1], libx11[2], xcb[3], and much more[4]… They use their own homegrown transpiler from C99 to Go.[5]

I can only have respect and admiration for whoever is behind this effort :)

[1] https://modernc.org/sqlite

[2] https://gitlab.com/cznic/x11

[3] https://modernc.org/xcb

[4] https://gitlab.com/users/cznic/projects

[5] https://gitlab.com/cznic/ccgo

cz.nic is a Czech domain registrar. (CZ NIC...)

Even as a Czech, I have no idea if they are owned/governed by state, or not. I need to look into wiki for that.

edit: oops it might be entirely unrelated. Sorry!

edit2: my confusion is understandable, as cz.nic has some open-source projects, too!

https://www.nic.cz/page/2049/projekty-pro-odbornou-verejnost...

For example open source routers Turris

https://www.turris.com/en/

Based upon info in an AUTHORS file [0] for a different project by the user cznic, I think there may indeed be some kind of connection with nic.cz ...?

But I believe, based upon my membership on the GoNuts group / mailing list, that this is mostly the work of one individual, Jan Mercl (he is quite active in GoNuts) — as also stated in the previously mentioned AUTHORS file, and in the AUTHORS file for the port of Tcl that is the subject of the OP [1].

I have used some of his non-transpiled code/projects in my own Go projects in the past. He seems to be a very solid coder, often happy to share his views in GoNuts, and also frequently reviews others' code too.

[0] https://github.com/cznic/golex/blob/master/AUTHORS

[1] https://gitlab.com/cznic/tcl/-/blob/master/AUTHORS

Ah googling him, it seems to me he was employed by cz.nic in 2012 or such. All right.

cznic or modernc.org is just a name clash, AFAICT. They have nothing to do with cz.nic.

he actually was employed by cz.nic according to some public record.

Transpiled should be treated separately from ported.

This sort of code isn't readable, idiomatic nor maintainable.

https://gitlab.com/cznic/tcl/-/blob/master/lib/capi_windows_...

I agree - scrolling through the codebase, I see a lot of unsafe.Pointer usage, which is rarely used in idiomatic Go, and throws a good part of static analysis through the window. So all the bugs in C code would also be in Go code - or might even be worse because of different runtime.

ISWYM but

> This sort of code isn't readable, idiomatic nor maintainable

isn't really fair. The original code is hopefully readable, idiomatic and maintainable, and if the transpiler does a correct job, who cares about the output any more than I care about the ugliness of the ASM in my object code when I compile it from source.

It seems fair to me. They only said that it should be seen as something separate than a port, not that it was bad. Honestly I think you're both right.

Yes, it's true that it doesn't matter if you only want to get the code running in a Go environment for whatever reason. But anyone wanting to learn about TCL or idiomatic Go by reading the source code for this project will be disappointed, and that struck me as the main appeal of this when I saw the headline.

If a program was classically built for x86 and was ported to ARM, the compiled output wouldn’t be any more readable. I’m not sure transpiled, which is just compiled by another name, is significant.

The point of GP is to differentiate porting from compiling, whether you call that compiled or transpiled is immaterial, the goal is to differentiate between a codebase which is idiomatic, understandable, and maintainable, and one which is a compiler artefact.

A port would imply and independent maintenance.

Here you wouldn't maintain that codebase, instead you'd fix the upstream or the compiler, and recompile.

> A port would imply and independent maintenance.

Quite the opposite. Port refers to targeting a different environment using the same codebase. The aforementioned ARM port wouldn't require a new codebase to maintain, but rather a new/modified compiler to target the ARM environment. Similarly, a macOS port of Linux software wouldn't require maintaining another codebase, it would require changing the compiler to support a macOS target.

> Here you wouldn't maintain that codebase, instead you'd fix the upstream or the compiler, and recompile.

Exactly. In this case a different compiler that targets the Go environment has been introduced. That does not imply that the compiled output is what should be maintained going forward. You wouldn't maintain the product of an ARM port or macOS port either. It is a port, not a fork. Fork is the word we use when there is implication of independent maintenance.

These are rather unusual definitions. Where I come from, porting to a new platform would mean rewriting platform-specific parts of the app, not coming up with a compiler that magically supports both targets. The choice between forking and refactoring the app into shared and platform-specific modules depends on many factors. Though I'd really like to see a compiler, not wrapper library, that supports both Apple's Core Whatever and your choice of Linux libraries.

On extreme end, porting something from let's say Commodore 64 to Spectrum ZX would be more a complete rewrite.

I also don't really understand the point of this project. Since it's compiled from C, it doesn't get any of possible benefits of using a language that is not C.

> porting to a new platform would mean rewriting platform-specific parts of the app

That may be part of the process, sure. The formal definition isn't clear on exactly what process is necessary to target a new environment, just that some action is necessary. Introducing a new compiler is one possible action to achieve the result.

However, when you do have to modify the code like you suggest, typically you are adding the environment specific aspects in #ifdef-type directives, so you're still not introducing a new codebase to maintain. It's still the same codebase that will be maintained just as it always way.

And even if you want to split hairs, no matter how you slice it there is no implication that the product of the port is what you will maintain. I still have no idea where that is coming from.

> I also don't really understand the point of this project.

To avoid cgo when using the library, I'm sure. Isn't that the point of a lot of cznic/modernc.org's projects?

> The formal definition isn't clear on exactly what process is necessary to target a new environment

I'm interested in the source of this "formal definition", because it doesn't really match anything I have come across over the years.

> typically you are adding the environment specific aspects in #ifdef-type directives

Not really, more like writing platform-specific parts as separate modules. Of course you can write Java for Android and Objective C for iOS in the same file if you really want, but it doesn't make much sense. Or even write a compiler that takes a source file with target-specific paths and transpiles to Java and Objective C if you want to go all in. I've never seen anyone do that.

> I'm interested in the source of this "formal definition", because it doesn't really match anything I have come across over the years.

"In software engineering, porting is the process of adapting software for the purpose of achieving some form of execution in a computing environment that is different from the one that a given program (meant for such execution) was originally designed for (e.g., different CPU, operating system, or third party library)."

Here we have software (Tcl) that was adapted to work in a computing environment (Go) that is different than the given program was originally designed for. This doesn't seem very controversial.

> Of course you can write Java for Android and Objective C for iOS in the same file if you really want, but it doesn't make much sense.

If you are changing the language software is written in, it is a stretch to call that porting. That isn't even a fork. That is a new project.

It make it sound a lot like you think that "porting" means what the rest of the world calls compiling and have no idea what porting is.

We certainly aren't talking about compiling. Compilation is repeatable. A port is done once.

Compiling can also be done once and the object files can be published on Github if you for some reason want to do that. As far as I understand, this "port" is transpiled from C source and the Go source files are compiler artifacts just like object files.

Compiling is repeatable, however. Porting is done once.

Ergo, traspiling to a different language is not porting.

Nor is transpiling - or compiling as it is more commonly known - to another CPU architecture or operating system, but that’s not what we are talking about here. The topic of the day is porting. Porting and compiling are quite different.

Please see https://en.wikipedia.org/wiki/Source-to-source_compiler to see the usual definition of transpiling, and how it is different from compiling and cross compiling (mentioned under "not to be confused with").

What I am to see? This Tcl port is clearly not source-to-source compiled. Everyone in this thread, including yourself as far as I can tell, agree that the product of this port is not usable as the source. The original Tcl codebase written in C remains the source and must be compiled each time it is modified to keep this Go package in sync. Hell, this entire thread started because someone pointed out that the Go target output isn't usable as the source...

The Go project's gc compiler was source-to-source compiled from its original C codebase into a Go codebase, which became the new source as the original C code was abandoned. Is this the origin of your confusion? That is a strange place to arrive just because Go is in the title, though.

Or is it that you also don't understand what the word source means?

I don't understand the logic of the argument. Let's go through the claims. First,

> Compiling is repeatable, however. Porting is done once.

Second,

> The original Tcl codebase written in C remains the source and must be compiled each time it is modified to keep this Go package in sync.

which implies that it is done more than once. If you are talking about something else than the Go "port" of tcl, it is not really obvious to me what it is.

I'm not sure what the struggle is here. The terms are well defined and well understood. Porting is the process of adapting software to work in an environment it was not originally designed for. Compiling is the process of transforming code written in one language into another language.

In the case in question, Tcl was both ported and compiled to Go. There was a process required to allow the software to run in an environment it was not originally designed for, and there was a process required to transform the software into another language.

Said compiler is one of the things that was produced to allow porting to the new environment. The compiler is not the port. Porting is not compiling. We've been over that many times now.

Ok, the confusing parts are first the computing environment: So, Tcl was ported from Unix to Unix. To me they sound the some environment, but I guess you could just claim otherwise. Second: The "port" was achieved by source-to-source compilation and is unmaintainable by itself. Yet somehow "Porting is not compiling." Where exactly is the port?

It was ported to the Go environment. Go is not Unix. Go is closest to Plan 9, if anything, but stands as its own thing.

This is interesting. Everything I had read so far has pointed to the direction that Go is a programming language.

well it gives a lower bound on port performance.

Cool, this is excellent. AOL embedded TCL interpreters attached to a listener on the network for all the services, so you could telnet in and see in memory what was going on and change config and what not. A tcl command is just a CS 1 program with argv and argc, so even pretty junior people could safely add commands. The framework was event based, no threads, so while the tcl was running nothing else was being done, so you could rearrange memory and it was cool as long as you finished before the end of the command. If would be pretty simple to add in a control port thing to Go. The advantage of TCL with its simple syntax is most commands are just cmd arg1 arg2 arg3 but if you have some general logic, you can express that as.

This is horrifying. Thanks for the nightmare fuel.

> while the tcl was running nothing else was being done, so you could rearrange memory and it was cool as long as you finished before the end of the command

My mind immediately springs to all the ways this can go horrifically wrong.

Tcl uses a memory model internally where each piece of memory belongs strictly to the thread that allocated it, with this enforced in lots of places. There are a few loopholes past it (for process-wide concepts such as the current working directory cache, or for inter-thread messaging) but by and large you write what appears to be single-threaded code.

There's support for deep coroutines and non-blocking I/O so it isn't limiting. You only really need multiple threads when dealing with compute-heavy code or a particularly crufty API.

This is how most high performance networked software is written, it’s mostly fine, when you’re running on the event loop, nothing else is.

What server are you talking about? I distinctly remember naviserver/aolserver being multithreaded and being able to share information between threads using a specific set of nsv_* commands.

SAPI - the Server API C stack. AOL server was a web surfer purchased from some small innovative company. It did also have TCL but was threaded. Way harder to extend then the No Threads Kernel that back-ended all the classic AOL clients and the AIM clients. I wrote the ns_flap.so shared library for the threaded AOLserver and ai maintained the non-threaded flap library ( a simple framing layer for TCP connections https://handwiki.org/wiki/OSCAR_protocol#FLAP_header ).

It was desgin3d to be easily horizontally scalable, and to be easy to write code for that was stable. We would put one process per CPU, and leave one for the OS. Also would put in one Ethernet card per process, if it was one that talked to the internet, and then used a user-land TCP stack on top of DLPI (?).

The services there were all high volume but not super latency sensitive, so. A poll/select loop that spun a thousand times a second or so was fine. You could add or reload all kinds of config without restarting and hence without losing any traffic. It was designed for zero downtime maintenance.

Interesting, thanks for the info. I had no idea there even _was_ a second web server at AOL that ran Tcl.

They weren’t, mostly, web servers, but handled various proprietary network formats from the proprietary clients and amongst themselves. Point to point mesh connect for AIM, hub and spoke for AOL, smart clients that could route to the appropriate endpoint based on the message contents. AIM had a “reconnect here” host to client message so you could dynamically move load around for maintenance. It was a type of select or pool loop like libuv, but before we had more than a few hundred hosts to interconnect. He entire internet and all client traffic came thru one file descriptor for the dedicated Ethernet for that process. It was all straight C, timers, Various layers of network protocols (tcp, ssl, flap, etc. the AOL client servers had lots of international localizations, which could be reloaded via a TCL command. You could see histograms of al the network latencies, at various levels, in a TCL command. I think at some point we added times to the prefix of allocated memory and had histograms on memory lifetimes. (Mostly we only allocated large chunks of memory, then at steady state would shuffle stuff of of a free list.

There were a few HTTP servers written for things like AOLTV and as an RPC gateway from third parties into the “host complex” e.g. one called ewoks was used for user registration. It was headless, only gave out 302 replies or 4xx replies.

> even pretty junior people could safely add commands.

What about the unattractive ones, Harvey Weinstein?

Ok I’ll leave quietly

One of my first jobs was in Tcl and I still wake up with night sweats about it. Some languages were just meant to die. I have no idea why anyone would be interested in using it in 2022.

Tcl is perhaps an acquired taste, but some of us weirdos still like it - https://colin-macleod.blogspot.com/2020/10/why-im-tcl-ish.ht...

I had to use Tcl/Tk for some computational chemistry circa 2005. I'm sure it seemed amazing in the early 90s for quick GUI apps, but 10 years later it was just an absolute nightmare compared to powerful languages like Python.

Tcl was definitely powerful, and didn't fall short of Python on that front.

It's on most of the other fronts that it did.

Some of my graduate level work was in Tcl, and my first job used a lot of Tcl. I have fond memories of the time, and Tcl is one of my favorite languages to this day. It's power and flexibility are fantastic.

My first job too! So true. Everyone knows how easy it is to make serious quoting mistakes in a Bourne shell script, but Tcl takes the risk to a whole new danger level.

But any harm done by Tcl is more than redeemed by its creator being the author of one of my favorite of all programming books, A Philosophy of Software Design.

It is the de facto scripting language in the EDA world unfortunately. I agree though. Terrible language. I initially thought this was going the other way round so you could write your EDA scripts in Go. No idea why you would want to use TCL when you can use Go.

I briefly worked on a WASM to TCL transpiler so I could write EDA scripts in Rust. Got it kind of working but WASM is actually quite large and TCL integers in the version I was forced to use (thanks PowerArtist) are completely broken so it ended up being not worth it.

I wrote some small scripts in it for VHDL projects but I can't imagine writing any large project in it, that is very cringe.

Tcl (expect) was joyous compared to my time with IBM JCL.

many years ago, when my ex-girlfriend was in college, for whatever reason she managed to get herself into a course that involve programming and for some unknown reasons, they were asked to do it in tcl.

obviously I was the go to expert on everything related to computers and I still remember the shock when she told me that I need to spend my weekends on helping her to understand tcl.

My first long term job was in tcl and I have nothing but love for that language. Extremely simple syntax wise with the ability to extend the language whenever you wanted. Still makes other languages look bad imo. Only experience that came close was clojure. There is so much power in tcls expressiveness.

+1 :)

Is "ported" the right term here? It know the repo's README says "CGo-free port", but this is the C version of TCL transpiled from C to Go (see the ~13MB .go files per platform in the "lib" directory). Which is a very cool idea, and the author has done the same thing with SQLite, to avoid CGo (https://gitlab.com/cznic/sqlite).

Here's a link to his C to Go translator: https://gitlab.com/cznic/ccgo

whoa you weren't joking, what the hell:

https://gitlab.com/cznic/tcl/-/blob/master/lib/tcl_windows_a...

Nothing says "easily maintainable" like a 298,000 line go file. Gitlab is struggling to render it for me.

This is of course generated code (output from the C to Go converter), so to "easily maintain" it, you'd edit the Tcl C source files and re-generate these. The generated Go code is committed to the repo so the person using the library doesn't need the C-to-Go converter tools.

Can't lose context if you never leave the file.

Can't have context if the file never loads.

Sounds like you need a better editor :)

But seriously, I would expect most editors to handle large files relatively gracefully, even if they aren’t as fast as vim for example. Browser based code viewers on the other hand, I grant you, aren’t generally up to the task.

That is just a compilation artifact, though. He only commits it for online importing.

refreshing to see a port to a language other than rust for a change. Not saying that rust isn't a good language, just good to see some diversity

a little bit of variety here: https://wiki.tcl-lang.org/page/Tcl+implementations. new, old, maintained, etc.

some don't care about the "in $lang" branding.

wait til they transpile it to rust!

I know TCL has an interesting story about how the syntax is basically all embedded strings, or something. Does anyone know if it has a recursive structure, as in lisp? Maybe it's more of an "everything is an X" structure, as in forth where everything is a "word".

The entire grammar is spelled out in the tcl(n) man page[0]. It’s amazingly concise and easy to learn from just this. To answer your question, it’s “as if” everything was a string, but the implementation is reasonably efficient and doesn’t convert all the time.

[0] https://man.archlinux.org/man/Tcl.n.en

Yes, in Tcl everything is basically a string, but has Lisp-like recursive structures: An expression like `{foo {bar baz}}` evaluates to the string "foo {bar baz}" at first, but then that might be split into two strings "foo" and "bar baz" by the interpreter.

I recall Tcl being described as a Lisp without the parenthesis many years ago, which I still think is somewhat apt.

Antirez' article "Tcl the Misunderstood" [0] is well worth a read if you want to know more.

He also has an interpreter, Picol [1], in 500 lines of C code that is very easy to read and to understand.

[0] http://antirez.com/articoli/tclmisunderstood.html

[1] http://oldblog.antirez.com/post/picol.html

I find this is a good explainer:

https://wiki.tcl-lang.org/page/everything+is+a+string

The page on the lack of "null" kind of goes with that as well: https://wiki.tcl-lang.org/page/null

Lots of people say Tcl is homoiconic. It can feel pretty lispy if you want to code that way, since evals and uplevels work a little like macros.

In my mind, Tcl _is_ a lisp. Admittedly, it's pretty far out on the family tree branches, but the overall structure feels very lispy to me.

I was really disappointed when i opened the project. I believe its transpiled and also drastically missing documentation.

"hey this can do everything tcl does, but we are not going to show you, you just will have to find out yourself"

Would love to find some kind of write up by CZNIC about why they're doing this.

Tcl can still be found in a few places powering some pretty critical infrastructure, I know a few networks running with significant automation being driven with Tcl

OpenBet use it quite a lot as well, so it powers a large number of the world's most popular gambling sites.

He’s doing SQLite too, and SQLite uses TCL extensively in its tooling.

I am not sure what is the point of this? (of course except for "because you can", which is a good reason to do things, sure)

Why be cgo-less, when in the end, it is just transpiled, with unsafe.pointer everywhere?

Do you gain some portability, that was not in the original C? Is there a speed gain, or ease to build gain?

Maybe the end "consumers" have it simpler, because they don't need to worry about cgo, but then it's harder to compile in the first place.

Does this mean it can run an eggdrop? Otherwise, in my eyes, it's not real Tcl.

Why though? Glad I don't need to write iRules anymore tbh.

So we know this is transpiled, wondering what is the usual approach to determine the quality of the transpliation output? Any systematic work into this aspect?

I would be more interested if it was ported to D. Also I think it needs a change of surface syntax to something Lisp-like.

Please check this TCL inspired command language in D [1].

I also share the same sentiments [2].

[1] A command language on top of D:

https://til-lang.github.io/til/

[2] Tcl-inspired command language on top of D:

https://news.ycombinator.com/item?id=27167762

Thank you very much

Looks like they are avoiding Tk altogether? Anybody know if that's the plan going forward?

https://gitlab.com/cznic/tk

I was wondering how this was done entirely without CGO since hooking up the transpiled code to a native Go X client like xgb sounds really hard. Apparently it uses a transpiled version of libX11 and freetype. Which seems pretty crazy but I guess there is no reason it couldn't work once you have a robust translator.

Ah cool, ty. Now I kinda wish I'd been aware of these projects sooner.

You should examples to the repo.

Tcl examples? I assume it's just a standard Tcl interpreter, so should run standard Tcl code. So you should be able to find examples anywhere online... e.g. https://www.tcl.tk/about/language.html

Now will someone please implement Tk in Qt or maybe Electron...?

Haha, indeed - I also greatly prefer Tk to Qt or Electron.

Some things I love about Tk:

    - adds a cool 1990s retro aesthetic
    - usable in X11 tunneled over an SSH connection
    - doesn't consume all available CPU and memory resources
    - the Tk canvas is a beautiful thing

> or maybe Electron...?

I just physically shuddered. Tk with GDI was a work of art. Electron is... not that.

Just imagine something as light and expressive as tcl put into an electron frankenstein gave me a good chuckle :D

I always considered Electron and Qt/QML doing the inverse, just with more BCPL-ish languages.

They were so preoccupied with whether they -could- that they didn't stop to think about whether they -should-.

Pithy joke aside, TCL is an underloved language and I'm always happy to see it in projects. Nice work!

All they need is to port Go to Python for a garbage language trifecta. Good god. (I am somewhat joking. Somewhat.)

My first exposure to TCL was for writing IRC bots in the 90s.

Thankfully that was my last exposure to TCL.

Yup, eggdrop[1] and expect[2] for me

[1] https://www.eggheads.org/

[2] https://core.tcl-lang.org/expect/index

Me too. Or not bots but I did script some IRC client with it and it was a lot of fun.

TCL for many people begins and ends at Eggdrop.

Thanks Robey.

Eggdrop? Me too.

Reply on news.ycombinator.com