Why I'm Tcl-Ish
colin-macleod.blogspot.com
colin-macleod.blogspot.com
That being said, here are some interesting things that tripped me up:
- using expr for arithmetic. The author mentions it, and it is cumbersome, but it also exemplifies what makes TCL special
- upvar for pass by reference. TCL does pass by reference by the command upvar, which says "take the variable named x in the scope above and assign it to the variable y in this scope". Confusing at first but a nice way to have pass by reference in a dynamic language
- docs are hard to read sometimes. The manual and tutorial for TCL are fantastic; the wiki is a little hard to read sometimes. The pages kinda seem like conversations, with authors weighing in on an article and tagging their names to their comments.
- lack of structured data. Saw another commenter posted this, and its something I wish TCL had better out of the box support for. The addition if dict was huge. The good news though: it's extremely easy to build a new command set to build a data structure out of dicts.
Despite all of this I'm a big fan of TCL. I try to tell as many people I can about it, because it really feels like a breath of fresh air compared to my day job Python. It allows you to be creative (and reckless!) with its power.
Edit: going to post the tutorial because it's so essential to getting started: https://tcl.tk/man/tcl8.5/tutorial/Tcl0.html
% interp alias {} = {} expr
=
% set x [= 2 + 3]
5
% interp alias {} @ {} lindex
% set a {1 2 3}
1 2 3
% @ $a 1
2Example:
% string index abc 1
b
% string index abc 1+1
c
% string index abc 1+1+1
bad index "1+1+1": must be integer?[+-]integer? or end?[+-]integer?
% string index abc end-1
b
The shell-like "everything is a string" is seductive at first but in practice you spend way too much time messing up your quoting. It's similar to how sometimes people suggest dropping yaml/json for a "simple ini file": it quickly breaks down and you end up with a custom ill-defined mechanism for nesting values.
TCL can handle more than just string variables: you have arrays, dicts and a bazillion libraries implementing object orientation. The result is that manipulating structured data is still more awkward than in any other modern language, except perhaps shell scripting.
Unless you're interfacing with a legacy or third-party system please don't use it. If you think you need expect please just use pexpect.
To my way of thinking, Tcl is one of the languages that falls into the same category as Lisp, Smalltalk, and Forth. Like those three, Tcl takes a relatively smaller number of concepts compared to most other languages and tries to get more mileage out of combining them together than resorting to adding new conceptual layers. For Tcl, the core concepts strings and string manipulation. Most languages have these things, but very few languages build their list libraries, evaluators, etc. nearly as directly upon them as does Tcl.
The net result of this approach is a language that's almost guaranteed to feel strange for the vast majority of developers, who work in languages that tend to be composed of larger numbers of more specialized features. So immediately off-putting to large numbers of potential users.
The other property these languages share, once you get past the initial barriers to entry, is a certain sense of enablement and power. The first few times, you realize how control flow structures, etc. are easily implemented in terms of user-visible language constructs, it's at first gratifying and exciting to see and then potentially enabling. After all, it's easier to write a custom expression evaluator in Tcl than, say Java or Python.
So the community of opinion around these opinions tend to bifurcate along lines of whether or not the individual in question has spent enough time to truly 'grok' the nature of the language.
And note that this can be totally orthogonal to opinions on whether or not the language should actually be used. It's easily possible to not 'get' the nature of Tcl or Lisp and yet have it be the thing you have to write to program your EDA tool or AutoCad. Similarly, it's easily possible to see the power of a language like Tcl and lament either that too few people understand its full power or that it's full power is maybe too flexible to be used in a production setting on a large industrial project.
None of that means these languages are not worth pursuing, but I do think this line of thought helps explain how they succeed and fail in the marketplace of ideas.
> I never got the impression that Tcl was popular nor that its popularity was increasing.
Because it was developed as part of work on EDA tools, it developed a niche there. Also in early web via AOLserver.
Where it truly shown, however, was in GUI programming on Unix. Tcl and its GUI library Tk were around X11 before virtually anything other than the original Xt-derived widget libraries. Even more than that, you could open up an interactive 'wish' session and get an active and clickable button on the screen in literally three interactive commands. Compared in ease of use of almost anything other than Visual Basic, Tcl/Tk was light years ahead, and Tcl/TK far surpassed VB in expressivity. (For the reasons I mention above.) Between this and a few other libraries like 'expect', it's hard to overstate exactly how much power Tcl/Tk put in your hands, and relatively easy to use at that. To be honest, there are still large swaths of modern development where we've really regressed in expressive power. (Although HTML/JS does make it pretty easy to get a button on the screen in the simplest case.)
That's the thing that Lisp/Smalltalk/Forth/Tcl proponents, who lament the unpopularity of their favorite language [1], frequently get wrong. It's not the case that others don't see the light and don't grok the power and flexibility that those languages bring, but that those don't really matter that much in most practical use cases, and that they're even detrimental to building and maintaining a large code base with multiple developers.
Yes, it's great that you can write your own control flow constructs. So what? A sane language already includes those constructs, and they're known and understood by all users of the language without having to first read the documentation of some third-party lib (haha, just kidding, there is none – read the code, dummy!).
[1] I don't mean to imply that you're one of them, I'm talking in general.
This statement does a good job of summarizing where I've landed on these sorts of languages, myself. Viewed in terms of Amdahl's law, the most you can improve a system by adjusting a single component is the impact of that single component. From a systems development point of view, the most impact you can have by changing programming language is to eliminate or reduce the cost of expressing your system's processes in a form the machine can understand.
Notably, this still leaves open most or all the other problems that beset software projects. The requirements are still under specified, the team is still subject to staffing constraints, funding still has to be justified and acquired, legal and business and scheduling constraints are all still there, and is the business viable at all in the timeline that you have? [1] The list goes on and on... and because the language doesn't really help, the value prop for niche languages goes way down.
That negativity aside, I do see potential counterarguments to this in the fact that an expressive enough language can help you improve your process to the point where you get answers faster and more reliably. But even that argument requires both a leap of faith and the willingness to take on the risks that implies.
1] Looking at this list, it's written in a way that might seem to imply startup/VC-funded project work - but that's not intentional. Those issues are also almost all faced by internal corporate development projects - either IT or for externally facing projects.
> [1] I don't mean to imply that you're one of them, I'm talking in general.
I didn't think you were, but thanks! :-)
I do not believe that novel control statements are any different from novel functionality. A sane language already includes plenty of standard functions, but that doesn't relieve the need for more. New functions, like new control statements, may be poorly documented.
The fundamental power of computing is abstraction. Functions, macros and control structures are all different types of abstraction. The only way to build a successful, malleable large program is to abstract, and to avoid macros and new control structures is simply to require larger teams (because someone has to hand-write all the code which the computer could be writing for you) and more ad hoc approaches.
The difference is that I write new functions on a daily basis, while I almost never need new control flow constructs. And if it turns out that such a construct would be useful, then it's almost always better to also have syntax and compiler support for it (for error checking, more descriptive parser errors, optimizations).
I wonder if perhaps you only think you rarely need new control flow constructs because you are using languages without good support for macros.
Where a Lisp or Tcl programmer might just code up a true DSL (cf. CLOS or the condition system), perhaps you just write functions, never realising how much more readable and understandable your systems could be.
However, since I did use expect and spent some time learning Tcl, I actually did appreciate one useful thing about it. It's very easy for me to in-line Tcl code into, say, a configuration, parse it with the help of `[info complete]`, attach it to a list and then expand that list `{*}$some_list` as an argument to `expect` or `interact`. I can now have UART specific macros that are bound to keystrokes like Ctrl-t x. Sure I could do something like that in python, but it probably wouldn't be as clear-cut as:
# Fix terminal row/col height for less, vi, etc
# type Ctrl-t r
bind[r] = {
# depends on xterm size info, may not be universal
send "\{ "
send {old=$(stty -g);}
send {stty raw -echo min 0 time 5;}
send {printf '\033[18t' > /dev/tty;}
send {IFS=';t' read -r _ rows cols _ < /dev/tty;}
send {stty "$old";}
send {stty cols "$cols" rows "$rows";}
send {unset old cols rows;}
send {shopt -s checkwinsize;}
send {clear;}
send " \}\r"
}It might also help to export the local TERM value remotely because it's not otherwise passed through a serial port.
[1] the tool if anyone is interested: https://gist.github.com/adedomin/7e615405a2275b01dbae3552116...
eval 'resize'
Do the same thing?
It was really cool for a little while though- I have fond memories of using it for a couple years in the mid '90s, but I would never go back to it once I moved on. Also, at the time I remember thinking Tk (Tcl's GUI toolkit) looked mostly modern and nice, maybe even more modern and nice than other GUI toolkits common on Unix systems at the time (Athena widgets, Motif, OpenView, etc.). As ancient and bizarre as it might look nowadays, back then it looked halfway decent if not outright OK. I don't know what Tk looks like now but I have to think it's not that different from 20+ years ago, with the weird autolayout/resizing policies and everything. Tk was a huge draw then but probably not so much now.
The quoting issues can be a pain, but if you learn to treat it like a language rather than extended shell, you learn the tricks to avoid escaping/quoting issues.
Maybe if TCL had a single standard OO library in the late 90s it would have been more successful? But I still think that languages with native OO constructs are much better that a shell-like string-oriented interpreter.
Exactly. Tcl is what we should be using for "shell scripts" instead of bash.
spawn -open [open /dev/ttyUSB0 w+]
interact {
`u {send "uname\n"}
`v {send "show-version\n"}
`` exit
# lots more
}
https://yosefk.com/blog/my-history-with-forth-stack-machines...
I believe now as then that the great contribution of Tcl was Tk (the graphical toolkit), which is fantastic and has bindings in other languages. I really wish GIMP had used Tk and improved it rather than inventing Gtk which is inferior in most ways.
Maybe this is all because I'm doing it wrong, writing scripts, and I should stick to its original purpose as an extension language, but it's easy enough to add C extensions to Lua or Python, and at some point you may end up writing a significant sized extension and the language will matter.
[1] http://git.annexia.org/?p=libguestfs-talks.git;a=tree;f=2019...
This is the oft-overlooked embedding case which can be difficult in other languages (compared to the normal case where you start out in language X and need to call into a C function to do something low level). For example in Python the embedding case is tricky and lightly documented, and with only a semi-stable API which has changed under us a few times.
In nbdkit we do a lot of language embedding (https://github.com/libguestfs/nbdkit/tree/master/plugins) including Tcl, Lua, Python and many others.
It turns out that when you make a language like that, you end up something where your code looks and feels like a list of commands (much like a shell script) but is also capable of doing "real" programming stuff where needed without much pain.
The test suite for SQLite is (still!) written in Tcl, I think partly for this reason. You probably wouldn't want to write a test suite in bash, but you don't quite get that feeling of "it's just a list of commands."
I really disliked TCL at first because the syntax seemed so arbitrary compared to other languages. It's only when you realize that TCL is actually more pure than other languages that you begin to appreciate its beauty. When the man page starts with "The following [12] rules define the syntax and semantics of the Tcl language:", they mean it. Those are the 12 rules. They are followed exactly and in order every time. Beyond those 12 rules, the rest is really DSL to make it look more like a normal programming language. It's very easy to code an entire TCL interpreter in 1000 lines of code. By coding in such a pure style, it made me appreciate how much the quirks of other languages are fundamentally designed to help programmers and not create frustration, which is how I always experienced all the "gotchas" I had to debug.
Edit: Here it is: https://code.activestate.com/recipes/68396-try-catch-finally... not needed now since latest version has it built in.
The design of Tcl is indeed simple and elegant. For example, check out the Tcl man page linked in the article:
https://www.tcl.tk/man/tcl8.6/TclCmd/Tcl.htm
The 12 stated rules suffice to define the syntax and semantics of the language.
And indeed, the Tcl/Tk tandem is great for building applications with convenient graphical user interfaces!
Many thanks to the author!
The infinitude of footguns of Unix shell make us discount it as a serious programming language but Tcl shows that it doesn't have to be that bad :)
0: There is no reason why a shell couldn't fix bugs like:
$ echo "$X"
*
$ echo $X
foo.txt bar.mp4
1: especially the incessant stupidity about adding type information to pipes, as if grep were a function rather than the arbitrary binary file /bin/grep
A LONG time ago, I used to use it (tcl/tk) to do quick-and-dirty apps on X-windowed Unix boxes and port them to Windows without really knowing much about Windows. (That was handy!)
And Expect blew me away back in the day of distributed systems management (I'm more of a programmer than a sys-admin, so I guess it would even be more important to those folks?).
The whole "There is no math, only strings" thing, still irritates me to this day, but I suspect you are right - this might be tied to the simplicity and strengths.
IDK. Food for though. Interesting piece. I think that I'm the only person I know that still dabbles in TCL after about 15~20 years ago (maybe I need to get out more), but it certainly rocks where it rocks! It is always surprising and pleasant to see a modern piece on it.
The first "higer" language I ever learned was Forth, and I never quite got to terms with infix notation, so I might be biassed ;)
set a puts
$a "Hello World"
a = print
a("Hello World")
fn puts(s: &str) {
println!("{}", s);
}
let a = puts;
a("Hello World");
#include <cstdio>
auto a = puts;
a("Hello World");
Also probably all shell scripting languages, like Bash:
a=echo
$a Hello World set a pu
set b ts
$a$b "Hello!"
((eval (string->symbol (string-append "dis" "play"))) "Hello world")Each to their own of course, but..
Here [1] is an alternative do..while loop implementation provided by Tcllib, written in pure Tcl.
[1] https://core.tcl-lang.org/tcllib/artifact/4f7e97706ff67a40
\def\a{mess}\def\b{age}\csname \a\b \endcsname {Hello there}\bye
in (UCB)LOGO:
? run list word "pri "nt [Hello there]
set a "puts"
$a "Hello World".
https://wiki.tcl-lang.org/page/Why+can+I+not+place+unmatched...
If someone with 40+ years experience in the industry or gray/white/no hair says something about a technology trust him/her, after 8 month you can agree or not...but the chance is really high you will. And the big joke, old people are not IN because they never heard of "Ruby on Rails", but they already had experience with something like it, and in the long run it just don't work.
set a "pu"
set b "ts"
$a$b "Hello World"
---> Hello World
a = "pr"
b = "int"
a+b "Hello World"
---> Hello World
For a simpler, cleaner alternative, you might want to take a look at antirez's jim tcl
Handling binary data is pretty trivial, e.g. to read a full binary file into a binary string, you do:
set h [open "filename.bin" r]
fconfigure $h -encoding binary -translation binary
set data [read $h]
Tcl was one of the first languages I knew that properly and consistently supported multibyte text throughout the full language, including in Tk GUIs. I.e. writing a cross-platform GUI that could handle Japanese character input/output basically "just worked". You could even read ShiftJIS or ISO encodings of japanese text, which was at the time a prerequisite as Japanese Windows versions were non-unicode (everything done in ShiftJIS AFAIR).
[edit] here is a real-world example of a trivial script handling binary data [2]: reading binary RGBA 4 bytes/pixel image data and outputting RGB 3b/pixel into a pipe that finally outputs a .PNG image. It's difficult to find a programming language that can do that in less lines and with more clarity.
[1] https://wiki.tcl-lang.org/page/binary+scan
[2] https://stech.muecke.pw/opensvn/free/trunk/dkbin/nn-fbtopng
If fact in 1999 I used TCL (and Expect) on an ISP I worked for to run various tests checking for Year 2K issues and we used the same framework for patching and system building. Talked to somebody I worked with at that place recently, he had found a bit of the old application, and we noticed we have a pretty good Ansible base written in it.
I also use it to write fairly easy DSL languages for various things, package them as tclkits, and have binaries that can be run on various machines.
Instead it's just a case of using it. In the past I used TCL for "expect" and for the Tk graphical toolkit. I can't say I've written anything major in TCL, but I think that's a common-case. Most people use TCL because it is embedded in applications as a scripting language/DSL.
When people express interest in TCL I usually point them at antirez's simple writeup which gets a lot of the basics shown of in a simple manner:
A classic use case is a debugger like gdb. What else, though?
Its like knowing how to use lisp but not knowing how macros work. Funnily enough this is actually where I am now with lisp.
Coming from that background, it's interesting to hear of TCL (which I do not personally know how to program in) embracing strings.
It would be interesting to read an article that goes in to some depth about the sorts of useful things that TCL's focus on strings makes possible or easy that would be difficult or unnatural to do in other languages.
Tcl not only allows you to share string representations of data between different programs, but also internally between different modules of the same program, easing issues of interoperability, scalability and deployability in wildly different systems and environments.
So how does that differ from just passing around a JSON blob or something?
First of all, lisp sexps have nothing to do with strings (which is why noone says that), whereas in tcl, every data structure has something to do with strings.
In tcl, semantically, everything is a string. Yes, the native internal representations of compound structures like lists and dicts are actually very efficient, but you can refer to those structures as if they were a string at any point.
And that will often cause the internal representation to switch to a string, and dump the more efficient "native" rep aside.
And that can happen when you don't really intend for it to happen (the infamouse "shimmering" problem).
Thats the biggest flaw I've found in years of hacking with tcl. It makes it hard to write funcs which take polymorphic data types. Is this a list of lists (where each sublist contains strings) or a list of strings (where each string may have whitespace)
You end up creating your own dynamic typing system sometimes. I know I have, and so have others (see the wiki).
There is SO much to love about tcl. It really is a lisp smashed into shell. And its so dynamic that you can do really powerful things in it.
Its really a lisp,, where the standard data structure is the string, not a linked list. And any func can be an f-expr (basically, sort of a runtime macro, which older lisps used to have).
I had a lot of fun redoing things like quasiquote (from lisp) in tcl. Yes, you really can generate tcl code as a template rather than manipulating strings.
Combine some sort of diy tagging system with ensembles (so that your code will run specialised funcs all automatically chosen based on your initial tag name).
The OO system is basically a toolbox for building any kind of OO system you want on top of it (almost akin to CLOS-inspired systems, in that regard)
Stackable io channels so that you can build a virtual filesystem with a various features all composed together (FUSE-like done beautifully, from a scripting lang perspective)
Great stuff. It really is a lisp/shell kind of thing, but sometimes its so dynamic that it shoots itself in the foot (shimmering, etc).
I wish they had decided to dump backwards compatibility post 8.0, and tried addressing some of horrible warts in the language and runtime, eg:
- the syntax rules are simple, but result in non-trivial code looking F UGLY (emphasis on the F). A couple of extensions to that syntax would have made the world of difference. How much more complicated would, say, 15 syntax rules be instead of 12?
- shimmering. NO. Stop it.
- not everything makes sense as a string rep (filehandles?)
- the standard commandset (really each is a DSL in its own right) needed to be revised long ago so that they were more consistent with each other (contrast list cmds with others)
Basically, maintaining backwards compatibility has killed tcl, even though it has so many nice features and potential greatness hidden inside.
(eg: jimtcl, a hidden gem)
Ok, I'm ranting now, I'll stop :-)
TCL is still the fastest way for me to make scripts. And as someone else posted, EXPECT was a great tool for testing applications and doing some basic automation. I laugh when the new consultants go "Robot Process Automation" RPA is the wave of the future, I've been doing that for two decades.
But the one thing that I think is very cool, is that it's very very simple to get a minimal Tcl interpreter (partcl) that can be embedded into a C/C++ application. It also still comes bundled in to Python for the Tkinter framework. You can even use it in Jupyter Notebooks now
In any case a very powerful language that is easy to learn and use.
Wanted to add a testing framework to refactor code. Didn‘t find something good though.
I've never found something that easy to use. Tkinter (Tk through Python) is awesome.
Since this is HN, if somebody knows a scene graph canvas that's even better and as easy to use, please let me know. I'm primarily interested in EDA (so not games).
And is there any reason to reach for Tcl today when lua and guile exist?
It’s battle-hardened, which is important, but it’s also fun, which is important too.
[0] https://devcentral.f5.com/s/articles/irules-concepts-tcl-the...
[1] https://www.a10networks.com/blog/aflex-tutorial/
[2] http://www-01.ibm.com/software/info/tealeaf/
[3] https://en.wikipedia.org/wiki/DejaGnu
[4] https://www.fidessa.com/products/training/training-courses
[5] https://www.mcs.anl.gov/~wozniak/papers/Swift_Tcl_slides_201...
Speaking of lesser used languages still used for specific domains, is REXX still used commonly for voice response systems? It was last time I had to deal with one around 2006.
I don't know why they chose such a brittle, error prone language for writing something where making a mistake can have dire consequences.
(I see someone else mentioned Tcl is used for Nuclear Power Plant simulators? Ouch!)
Embedded medical device software ... [1]
i.e., when the consequences of failure are just too great, a lot of people don't dare pass up Tcl.
Also interesting that you think flaws in the tool you use to learn to run a nuclear power plant are no big deal.