The Forth Methodology of Charles Moore (2001)
ultratechnology.com
ultratechnology.com
1. Iterating your understanding until you have the deep core of the problem.
2. Experimenting with possible solutions until you find the best approach.
3. Not coding the final production code until the first two steps are met. (Maybe this is a variant of Fred Brook's "Plan to Throw One Away"? see https://course.ccs.neu.edu/cs5500f14/Notes/Prototyping1/planToThrowOneAway.html)
4. Writing code and documentation that is simple, direct, and easy-to-understand. - go crazy in understanding the problem
- break it in small words
- repeat above steps until it's easy to write in forthI have used a similar approach when I needed to get a PoC out in 6 weeks, we made it, but not without drastically modifying "the spec". In one instance the PM wanted continuous queries over geospatial data, millions of points and arbitrary locations along with a time dimension. I changed the spec to use 3 buckets, near, medium and far. That feature took 2 hours, the continuous query itself would have taken weeks or longer.
Successful projects have a malleability to them.
This philosophy is often described by 80s lispers. They're not top-down nor bottom-up.. they lie in the middle. Trying to make a comfy intermediate layer so they can experiment various upper layers to fit the problem and adjust if rapidly in case of changes. Similar to IR in compilers I guess.
The way we do things today is basically guesswork. Which suits programmers just fine, as it lets them write a lot of code. But that's not the best of all possible worlds, only the default world.
This is really only an issue if you appoint programmers to serve as systems analysts. Sadly, this is all too common among businesses today.
Remember, programmers are detailists who must speak the language of the computer, and do not see the big picture, only one small part of a much bigger puzzle. A good systems analyst is a generalist who understands the business and can empathize with the users. They have strong communication skills, and can take a big-picture perspective and articulate their perspective and solutions clearly to management. They most certainly would not dismiss user requests and complaints as "incoherent ramblings". Programming and systems analysis are separate professions that require vastly different personality types, with the consequence that programmers make very poor systems analysts.
There's a reason we don't have them anymore.
Again, gathering requirements is a time-consuming and confusing process, making the software late and often wrong, because most companies assign programmers to the systems analysis process, not people skilled in actual systems analysis. A good systems analyst has the empathy and high-level insight it takes to understand the existing business systems and the true needs of the users, and design a new system to better meet those needs -- traits sorely lacking in programmers. Note that a business system is not the same as software. It incorporates all of the procedures, whether performed by humans or machine, necessary for business operations.
As a consequence, we tend to declare that nothing can be done about gathering requirements and implement "Agile" methodologies, which are nothing more than institutionalized guesswork: in the words of Milt Bryce, "do a superficial feasibility study, do some quick and dirty systems design, spend a lot of time in programming, install prematurely so you can irritate the users sooner, and then keep working on it till you get something accomplished." And the software is still late and wrong!
The world I live in is mostly crappy software products involving an infinite mess of APIs glued together and poor understanding of the core problem which leads to a huge mess. It's worse for the organization too, but good luck arguing that.
And now we get to add frequent supply chain attacks to the list.
I recently tried NextJS even though I consider myself an ardent framework minimalist. Create app (or whatever) proudly proclaimed it had installed 666 packages. I took that as a hint and bailed…
1) Out of order execution 2) Multiple levels of caches 3) Branch predictors
Forth has several neat advantages. I think it might be a better way to reason about a complex problem. It can be bootstrapped on almost any piece of hardware. It can be made to be very memory efficient in its runtime.
Those things were true in the past and they are still true now. But I do wonder whether the performance part of it is still true. Not because Forth has changed, but simply due to the fact that modern hardware (+ compilers) can take mediocre code and make it into pretty damn performant code.
I think he might be on to something here ...
So why don't we use Forth for everything? Does it have some ceiling that makes it not great outside things like making firmware?
Also, it's incredibly efficient, which is necessary when you have kilobytes of RAM. When you have gigabytes of RAM you can cobble together a Rube Goldberg machine without having to think too much, and time-to-market wins out over system coherence and quality.
It sounds very productive to always be able to express solutions with the perfect language for that type of problem. But you end up with a myriad bespoke languages that are very impractical to document and learn, and are a constant friction for collaboration (even with your future self).
That being said, not every piece of code needs to be accessible to everyone. Dsls allow you to create a hermetic world inside of a programming environment in which certain properties always hold, even if the environment does not guarantee these things for you. Programs written in the right dsl can be more productive to write and have much lower overall operational costs. But the price is that the rules of the dsl are not necessarily the same as the rules of the host language and this leads to confusion. For the truly ambitious programmer, it can be worth sacrificing accessibility to the median coder to build systems that the median coder can't even conceive of. Of course your reward is to be derided for not expressing the solution in a language they can already understand.
Wrt to Forth itself, Chuck Moore has pointed out that Forth is a multiplier: it makes good programmers better and bad programmers worse. I personally care more about the former property and frankly don't want to work with bad programmers anyway.
I do agree there is room for such approaches and they can have 10x effects in some situations. But more often than not, smart programmers can get wrapped up in the logical beauty of it all, distracted away from the core problem, feeling extremely productive but barely getting anything real done (like Vim can make feel so much faster but you really aren’t when you measure and compare properly).
Creating new languages is useful: config, markup, query… But it is hard and laborious to do well, and usually it is only successful if creating the language is your core goal, rather as a one-off auxiliary part of solving a problem, as languages like Forth or Lisp constantly encourage by their design. And, again, a non-trivial language without a manual is unusable (or it is trivial).
Limits to flexibility can be very useful: they make you focus on solving the problem with the tools you have, rather than making the perfect tools to solve the problem, tools that you will probably never use again because they are too tailored to that one problem.
I do it all the time. It is a huge productivity boost.
An example is a biz application I recently delivered to a large international customer. More than 90% of the C++ and Typescript code was auto generated from a simple declarative spec. The only custom code was the customer specific biz logic and custom interface logic to external tools.
Yes it isn’t generated with macros Lisp style but it works equally well.
https://yosefk.com/blog/my-history-with-forth-stack-machines...
...might recalibrate your expectations.
The fact was that he could run rings around whole teams of C programmers. His implementation would be more efficient, more reliable, and cheaper to mass produce (lower ram requirements, lower power requirements, etc). But he could never stay anywhere very long because all the Real Engineers knew that C was the Right Language to implement things in. His better solutions were seen as aberrations rather than a condemnation of their own mediocrity.
That’s the power of good marketing. Forth never had an industry giant spending billions a year on marketing to convince all of the engineers that it was the best language ever. C, C++, and Java all did. All the VPs have read articles and attended conferences that touted C++ or Java as the best thing ever, but they’ve never even heard of Forth.
People with deep non mainstream knowledge should avoid mainstream by wide margins and try to assemble in small groups of like-minded hackers.
And meanwhile C and C++ code runs the world. Without any commercial marketing behind it.
How about conferences? I’m sure none of the corporate sponsors have any marketing reason to contribute. <https://isocpp.org/wiki/faq/conferences-worldwide>, <https://javaconferences.org/>
Or magazines? Surely they were entirely supported by their subscribers, not their advertisers. <https://en.wikipedia.org/wiki/Category:Computer_magazines_pu...>, <https://en.wikipedia.org/wiki/Category:Defunct_computer_maga...>
There is a reason why it hasn’t happened (obviously).
It might not be for reasons X people agree with or value. However those reasons are important and valuable enough for non-X enthusiasts to bounce off X.
And to make matters worse, the typical reaction from X enthusiasts is to then claim that non-X developers are not smart enough to “get it” or that somehow people using X are smarter than non-X developers. Claims that are so obviously ridiculous that it makes non-X people bounce even harder off the X community.
I personally love learning new languages. And have written compilers for a few of my own. However that’s all fun and games. Not something I would use in anger for my day job where I have to work in a team and deliver highly complex software for large corporation that relies on my code to run their business.
However having said all that, Forth is really cool and fun to play with. I love minimum languages! Lisp is another fun one. Or the lambda calculus. All great fun languages!
Good news is that every problem you'll ever face can be broken down into sub-problems which are easier to solve.
Bad news is planning a software project in that much detail, down to the granularity needed for me to get accurate time estimates, takes as long as actual doing the programming would.
And even at that, there will still be many problems with the plan, because all that time we were not, say, making unit tests, or proto-types large enough to expose problems.
But yet, here he is, and his methodology certainly worked for him. I wonder how much Forth itself contributed to that success, or could he have used any interpreted language.
Any pointers on how to better estimate programming effort would be greatly appreciated.
(Previously discussed many times on HN; the most popular post: https://news.ycombinator.com/item?id=23537530)
To me, the Forth process is more rigorous and less error-prone. It may not be the 'industry standard', but for certain very specialized and intelligent teams (such as the Shuttle Software Group), I'm surprised it hasn't taken off (excuse the pun)...
This is excellent advice for humanity in general.
The strategic question is why going long (or, a better strategic/tactical balance) seems so impossible.
finally, aside from unpredictability, rapid innovation and capital growth implies a high discount rate: any new innovation must compete with the risk-free return on capital. if we assume 5% per year, we must discount a return 20 years in the future by 64% (≈1-1/e)
under these circumstances, it's sensible to focus on short-term gains with a rapid feedback loop, rather than long-term gains which might turn out to be chimerical
The 20% long(er)-term plays are going to require wisdom to sniff out.
do {
x();
y();
} while (!z());
begin x y z until
the thesis of this document is that you get faster results by taking the time up front to think your problem through until you understand it, and throwing away your code and hardware designs is not something to be afraid of
i'm not sure jeff and chuck's results at itv bore this out; hardware kept evolving out from under them too fast, other people's code became more useful (because of shitty hardware, because of the free software movement, because efficiency became less critical, and because of higher-level languages with better reusability), and they ran out of money
but simplifying a problem is still immensely valuable when you can do it. the problem with programming continues to be overcomplicating things
But that's actually not something specific to Forth. Every programmer does that to some degree. Sometimes it is included in big and small so-called "refactoring" steps. Moreover, one could say at first glance that the core of this methodology is the well known "if you spend more time on design you will spend less time on code".
The other thing is that, indeed, in this context it is less of a problem to throw away what you've done because of step 1-2. The phrasing is a bit too abstract; Perhaps clearer is what he said in an earlier interview: "Don't anticipate, solve the problem you've got." [2].
This piece of advice is so beneficial that it is almost a motto for me. It is also harder to do that one may think, because it is so easy to FUD oneself with "what if" and "what about" and sometimes you have to fight against your own hubris too (e.g. "I'll build something powerful"). So it requires some training; after all these years programming in Forth I still catch myself anticipating (e.g. "I think it could be useful to have that").
[1] https://www.forth.com/resources/forth-programming-language
[2] https://www.ultratechnology.com/1xforth.htm : Don't leave openings in which you are going to insert code at some future date when the problem changes because inevitably the problem will change in a way that you didn't anticipate. Whatever the cost it's wasted. Don't anticipate, solve the problem you've got.
One might not be able to anticipate future requirements, but there are situations, in which the engineer, provided they even get the right idea, can make their code reusable with little to no additional time needed, simply, because they know they they are doing and have experience with the matter. The worst that can happen to a product is, when less experienced engineers think to do well in telling the engineer with the experience, that this can be done in the future, when actually that future never materializes. Whole new directions of products die, because of this. Code might become even more, because of not anticipating things, and having to build workarounds later, because one does not get time to refactor and has feature pressure from management.
It can all be pretty shortsighted and can limit the output that skilled engineers can provide. Bad management is gonna manage badly.
That's not exactly what TFA describes, though.
> there are situations, in which the engineer, provided they even get the right idea, can make their code reusable with little to no additional time needed, simply, because they know they they are doing and have experience with the matter
Maybe. Problem is, how do you know when you or someone else has enough experience to make those calls? I guess the only way is to try and try again. But if you go that route, you should really keep the score, that is, note down when you made a call and check later if you were right or wrong. It's easy to trick oneself though because the future is virtually infinite, so a decision to make e.g. something more reusable at "this little extra cost" can be indefinitely neither right nor wrong.
In my experience, the best way to make code flexible and extensible is to make it simple. No hooks, no architecture, no pattern, just the smallest thing that does the job. And failing that, the most modular.
That way, when unforeseen requirements do come your way you can just modify your code. The simpler it is, the easier it will be. And it works for any future requirement, not just the ones you had the experience or luck to foresee.
The flip side though is that simplicity is not straightforward. One does not simply "spit out some [simple] code". Simple code requires time and effort. I would like to be able to just say "forget about extensibility, just make it simple", but it wouldn’t convey the difficulty of making things simple.