The Grug Brained Developer (2022)
grugbrain.dev
grugbrain.dev
I've seen your work in Hotwire in my role as a Staff Ruby on Rails Engineer. It's the coolest thing to see you pop up in Hacker News every now and then and also see you talking with the Hotwire devs in GitHub.
Thanks for being a light in the programming community. You're greatly respected and appreciated.
I’ve spent time at small startups and on “elite” big tech teams, and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.
I generally agree that it’s the best way to start understanding a system. Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.
Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.
It’s impossible? Or would take engineering work to enable
I guess you can attach a debugger for unit tests, but that's not very useful.
That is in fact incredibly useful
It’s a similar muscle to exercise as using a profiler.
But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case?
Or are you just struggling to say that your service utilizes multiple applications?
It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
(Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)
I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)
But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
My workflow is usually:
1. insert a breakpoint on some code that I’m trying to understand
2. attach the debugger and run any tests that I expect to exercise that code
3. walk up and down the call stack, stepping occasionally, reading the code and inspecting the local variables at each level to understand how the hell this thing works and why it’s gone horribly wrong this time.
4. use my new understanding to set new, more relevant breakpoints; repeat 2-4.
Sometimes I fiddle with local variables to force different states and see what happens, but I consider this advanced usage, and anyway it often doesn’t work too well on my current codebase.
I've often felt that I should, but never enough to actually learn how
> As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.
I tend to agree with them on this. For almost all of the work that I do, this hypothesis-logs-exec loop gets me to the answer substantially faster. I'm not "trying to run the code forwards in my head". I already have a working model for the way that the code runs, I know what output I expect to see if the program is behaving according to that model, and I can usually quickly intuit what is actually happening based on the incorrect output from the prints.
[0] The unreasonable effectiveness of print debugging (349 points, 354 comments) April 2021 https://news.ycombinator.com/item?id=26925570
Hard to get wrong, tedious to type and a huge speed increase to visually scan the output.
Most of us aren't Brian Kernighan or Rob Pike.
I am very happy for people who are, but I am firmly at a grug level.
I insist that my students learn a visual debugger in my classes for this reason: what the "stack" really is, how a loop really executes, etc.
It doesn't replace thinking & print debugging, but it complements them both when done properly.
It's useful for a beginner e.g in a for loop to see how `i` changes at the end of the loop. And similarly with return values of functions and so on.
Perhaps been underappreciating the gap in mental intuition of the runtime between then and now and how much the debugger helped to bridge.
He said that him and Brian K would pair while debugging. As Rob Pike told it, he would often drive the computer, putting in print statements, rerunning the program and so on. Brian Kernighan would stand behind him and quietly just think about the bug and the output the program was generating. Apparently Brian K would often just - after being silent for awhile - say "oh, I think the bug is in this function, on this line" and sure enough, there it was. Apparently it happened so often enough that he thought Brian might have figured out more bugs than Rob did, even without his hands touching the keyboard.
Personally I love a good debugger. But I still think about that from time to time. There's a good chance I should step away from the computer more often and just contemplate it.
My guess is that:
- Debuggers are most useful when you have a very poor understanding of the problem domain. Maybe you just joined a new company or are exploring an area of the code for the first time. In that case you can pick up a lot of information quickly with a debugger.
- Print debugging is most useful when you understand the code quite well, and are pretty sure you've got an idea of where the problem lies. In that case, a few judicious print statements can quickly illuminate things and get you back to what you were doing.
Most people who work on crud services do not see any benefit from it, as there is practically nothing going on. Observing input, outputs and databases is usually enough, and when it's not a well placed log will suffice. Debuggers will also not help you in distributed environments, which are quite common with microservices.
The only real issue in debugging a distributed/multithreaded environment is that frequently there is a timeout somewhere that is going to kill one of the threads that you may have wanted to continue stepping through after debugging a different thread.
1. When you get a stack trace from a failure, without knowing anything else find the right level and make sure it has sensible error handling and reporting.
1a. If the bug is reproduced but the program experiences no failure & associated stack trace, change the logic such that if this bug occurs then there is a failure and a stack trace.
1b. If the failure is already appropriately handled but too high up or relevant details are missing, fix that by adding handling at a lower level, etc.
That is the first step, you make the program fail politely to the user and allow through some debug option recover/report enough state to explain what happened (likely with a combination of logging, stack trace, possibly app-specific state).
Often it can also be the last step, because you can now dogfood that very error handling to solve this issue along with any other future issue that may bubble up to that level.
If it is not enough you may have to resort to debugging anyway, but the goal is to make changes that long-term make the use of either debuggers or print statements unnecessary in the first place, ideally even before the actual fix.
In my case (a digital audio workstation), a debugger can be handy for figuring out stuff in the GUI code, but the "backend" code is essentially a single calltree that is executed up to a thousands times a second. The debugger just isn't much use there; debug print statements tend to be more valuable, especially if a problem would require two breakpoints to understand. For audio stuff, the first breakpoint will often break your connection to the hardware because of the time delay.
Being able to print stacktraces from inside the code is also immensely valuable, and when I am debugging, I use this a lot.
But the debugger is essential for all the hard stuff. I’ll take heap snapshots and live inside lldb for days tracking down memory alignment issues. And print statements can be either counterproductive at best, or completely nonexistent in embedded or GPU bound compute.
- John Carmack loves debuggers
- Debuggers are most useful when you have a very poor understanding of the problem domain
John Carmack knows his domain very well. He knows what he expects to see. The debugger gives him insight into what “other” developers are doing without having to modify their code.
For Carmack, managing the code of others the debug environment is their safe space. For Kernighan et al in the role of progenitorous developer it is the code itself that is the safe space.
When I'm unfamiliar with a codebase, or unfamiliar with a particular corner of the code, I find myself reaching for console debugging. Its a bit of a scattershot approach, I don't know what I'm looking for so I console log variables in the vicinity.
Once I know a codebase I want to debug line by line, walking through to see where the execution deviates from what I expected. I very frequently lean on conditional breakpoints - I know I can skip breaks until a certain condition is met, at which point I need to see exactly what goes wrong.
In scripting languages print debugging makes sense especially when debugging a distributed system.
Also logging works better than breaking when debugging multithreading issues imo.
I use both methods.
If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes. And then I'll have to hope that I can reproduce the issue without optimizations.
I've worked on projects where incremental linking after touching the wrong .cpp file will take several minutes... and this is after I've optimized link times from e.g. switching from BFD to Gold, whereas before it might take 30 minutes.
A full build (from e.g. touching the wrong header) is measured in hours.
And that's for a single configuration x platform combination, and I'm often the sort to work on the multi-platform abstractions, meaning I have it even worse than that.
> If I wanna use a debugger though, now that means a full recompile to build the project without optimizations
You can use debuggers on optimized builds, and in fact I use debuggers on optimized builds more frequently than I do unoptimized builds. Granted, to make sense of some of the undefined behavior heisenbugs you'll have to understand disassembly, and dig into the lower level details when the high level ones are unavailable or confused by optimization, but those are learnable skills. It's also possible to turn off optimizations on a per-module basis with pragmas, although then you're back into incremental build territory.
Sometimes, you just really do need hundreds of thousands of lines of C++ (or equivalent) ...
I use the debugger in Intellij, using breakpoints which only log some context and do not stop the current thread. I can add / remove them without recompiling. There's no interruption in my flow (thus no excuse to check HN because "compiling...")
When I show this to colleagues they think it's really cool. Then they go back to using print statements anyway /shrug
10 minutes.
>If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes.
Typically always compile with debug support. You can debug an optimized build as well. Full recompile takes up to 45 minutes.
The largest reason to use a debugger is the time to recompile. Kinda, I actually like rr a lot and would prefer that to print debugging.
Have you tried out the Mold linker? It might speed it up significantly.
> You can debug an optimized build as well.
Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.
> Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
Yeah, but sometimes that's life. Reading the assembly and what not to figure things out.
>That must absolutely suck to work with.
Well, you know, I also get to work on something approximately as exciting as Chromium.
Yeah, which is why I was talking about source files. I was surprised that changing 1 source file (meaning re-compiling that one source file and then re-linking the project) takes 10 minutes. If you're changing header files then yeah it's gonna take longer.
Debugging kernel module issues on AL2 on bare metal ec2 with kprint. Issue does not reproduce in qemu. It happens
This is the "it's one banana Michael, how much could it cost, ten dollars?" of tech. I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds. I've worked in plenty of embedded environments where simply the download-and-reboot cycle took over a minute. Those are the places where an interactive debugger is most useful .. and also sometimes most difficult.
(at my 2000s era job a full build took over an hour, so we had a machine room of ccache+distcc to bring it down to a single digit number of minutes. Then if you needed to do a full place and route and timing analysis run that took anything up to twelve hours. We're deep into "uphill both ways" territory now though)
No C++ project compiles in 3 seconds, but your "change a single source file and compile+link" time is often on the order of a couple of seconds. As an example, I'm working on a project right now where a clean build takes roughly 30 seconds (thanks to recent efforts to improve header include hygiene and move stuff from headers into source files using PIMPL; it was over twice that before). However, when I changed a single source file and ran 'time ninja -C build' just now, the time to compile that one file and re-link the project took just 1.5 seconds.
I know that there are some projects which are much slower to link, I've had to deal with Chromium now and then and that takes minutes just to link. But most projects I've worked with aren't that bad.
Concurrency bugs just suck to debug sometimes.
First that visual debugging was still small and niche, probably not suited to the environment at Bell Labs at the time, given they were working with simpler hardware that might not provide an acceptable graphical environment (which can be seen as a lot of the UNIX system is oriented around the manipulation of lines of text). This is different from the workplace where most game developers, including J. Carmack were, with access to powerful graphical workstations and development tools.
Secondly there’s also a difference on the kind of work achieved: the work on UNIX systems mostly was about writing tools than big systems, favoring composition of these utilities. And indeed, I often find people working on batch tools not using visual debuggers since the integration of tools pretty much is a problem of data structure visualization (the flow being pretty linear), which is still cumbersome to do in graphical debuggers. The trend often is inverted when working on interactive systems where the main problem actually is understanding the control flow than visualizing data structures: I see a lot more of debuggers used.
Also to keep in mind that a lot of engineers today work on Linux boxes, which has yet to have acceptable graphical debuggers compared to what is offered in Visual Studio or XCode.
In any case, my suggestion is to understand both approaches and boldly use them in the right circumstance. If the need arises, be a rebel and break out the debugger or be a rebel and add some printfs - just don't weakly follow some tribal ritual.
By keep just one party to the loan, and most important, by me offering collateral to the loan in the event I do not deliver, then it keeps enforcement more honest and possible.
Furthermore, the loan contract should be written in such a way that the game is judged by the ONE TIME sales performance of the game (no microtransactions) and not qualitative milestones like features or reviews. Lastly, I would add a piece of the contract that says two years after the game is released, it becomes fully open source, similar to the terms of the BSL.
This is the fairest thing to the players, the bank, and the developer, and it lets the focus be absolutely rock solid on shipping something fun ASAP.
1) We only support Linux. 2) We only support Vulkan (or OpenGL, but I prefer Vulkan) 3) Singleplayer only. 4) Engine features: RGBA Diffuse, specular, shadow, normal/bump, emission maps for materials only. A simple triangle rasterizer with forward rendering. Support for audio, dae animations/rigs/geometry. Physics collisions through an octree and capsule and basic linear algebra. Basic first person/third person cameras. 4) As much content as possible from copyleft/free sources. Mixamo, Cgtrader, poliigon, textures.com etc. A programmer's job is to use whats out there. But, given an extra 6 months, I could also model and texture everything myself as well. I'll do all level design. 5) Dialogue written by me, voice acting through AI or vocaloid. 6) I would estimate the final code to be around 50,000 lines for everything. I can give you a breakdown if you want. All code is in C99, minspec is a ryzen 5600g, which is like $150 max these days.
Something I've missed? I've described a classic first person adventure shooter like Halo for you there. Simply let me know if I should do something else.
Back in college, we had to write a constructive geometry object renderer that used GLUT simply as a 2D canvas for a scanline-oriented triangle->trapezoid engine that roughly mirrored the capabilities of OpenGL with Phong and Gouraud shading, texture mapping, and bump mapping. It wasn't hard when each piece was broken down and spoon fed (quaternions and transformation matrices). The hardest part was creating a scene of complex object parts to model the real world programmatically.
The problem is, sometimes profilers don't identity anything in particular or unwinding things to a point where they can be fixed is actually very har. A more realistic ROI argument should include this as it is a real problem.
I think code should be reasonably fast out of the box. I'm not suggesting vectorizing everything or even taking extreme steps to avoid allocations, etc. Rather, if an algorithm can easily be O(N), don't dumbly implement an O(N^2) or O(N^24) version. If O(1) is possible, do that unless you know the numbers are going to be very small. Don't make 500 DB calls when one will do and finally, don't program by unexamined phrases - particularly those than include "evil" or "harmful" or other such adjectives.
"A debugger is how you get a view into a system that's too complicated to understand. I mean, anybody that thinks just read the code and think about it, that's an insane statement, you can't even read all the code on a big system. You have to do experiments on the system. And doing that by adding log statements, recompiling and rerunning it, is an incredibly inefficient way of doing it. I mean, yes, you can always get things done, even if you're working with stone knives and bare skins."
1. If I actually pause execution, the thing I'm trying to debug will time out some network service, at which point trying to step forward is only going to hit sad paths
2. The device I'm debugging doesn't *have* a real debugger. (Common on embedded, really common for video games. Ever triggered a breakpoint in a graphics shader?) Here I might substitute "print" for "display anything at all" but it's the same idea really.
Every debugger I've ever worked with has logpoints along with breakpoints that allow you to _dynamically_ insert "print statements" into running code without having to recompile or pollute the code with a line of code you're going to have to remember to remove. So I still think debuggers win.
But... that's where you put breakpoints and then you don't need to "single-step" through code. Takes less time to put a breakpoint then to add (and later remove) temporary print statements.
(Now if you're putting in permanent logging that makes sense, do that anyway. But that probably won't coincide with debugging print statements...)
Printf debugging gives you the full picture of an entire execution at a glance, allowing you to see time as it happened. The debugger restricts you to step through time and hold the evolution of state in your memory in exchange for giving you free access to explore the state at each point.
Occasionally that arbitrary access is useful, but more often than not it's the evolution of state that you're interested in, and printf gives you that for free.
And of course, should the problem you're debugging be throwing exceptions or crashing the app, the debugger can pause the world at that moment for you, and you get the benefit of debugging and having a "printf log" of the same execution already available.
I don't think you can achieve the same result using debuggers, they just stop you there and you have no context how you got there or what happens after.
Maybe some people just aren't good at print debugging, but usually it finds the issue faster for me, since it helps pinpointing where the issue started by giving you a timeline of events.
Edit: And you can see the result of debugger use in this article, under "Expression Complexity" he rewrote the code to be easier to see in a debugger because he wanted to see the past values. That makes the code worse just to fit a debugger, so it also has such problems. When I use a debugger I do the same, it makes the code harder to read but easier to see in a debugger.
One is finding a needle in a haystack - you have no idea when or where the bug occurred. Presumably your logging / error report didn't spit out anything useful, so you're starting from scratch. That and race conditions. Then print statements can be lovely and get you started.
Most of my debugging is a different case where I know about where in code it happened, but not why it happened, and need to know values / state. A few breakpoints before/during/after my suspected code block, add a few watches, and I get all the information I need quite quickly.
Temporary? Nah, you leave them in as debug or trace log statements. It's an investment. Shipping breakpoints to a teammate for a problem you solved three months ago is a tedious and time-consuming task.
Anyway, breakpoints are themselves quite tedious to interact with iterative problem solving. At least if you're familiar with the codebase.
That's not usually the case.
Setting conditional breakpoints, especially for things like break on all exceptions, or when the video buffer has a certain pattern, etc, is usually where the value starts to come in.
I can debug .NET application and change code live, change variable states if needed. Watch variables and all kinds of helpers like stack navigation was immensely improved since I started 15 years ago.
I can say debugging Java/.NET applications is totally different experience than using debugger from 1999.
Multi threaded apps debugging and all kinds of helpers i visual debuggers.
I just fail to see why someone would waste time putting in debug statements when they can configure debug session with conditional break points.
I tend to do both, print statements when I don't feel I want to be stepping through some cumbersome interplay of calls but diving into the debugger to step through the nitty gritty, even better when I can evaluate code in the debugger to understand the state of the data at precise points.
I don't think there's a better or worse version of doing it, you use the tool that is best for what introspection you need.
If you can detect a problematic condition and you want to know what will happen next, a debugger is a great tool.
If you can detect a problematic condition and you need to find out what caused it, it’s printf all the way.
My theory is that different types of programming encounter these two types of problems at different relative rates, and that this explains why many people strongly prefer one over the other but don’t agree on which.
> It takes less time to decide where to put print statements than to single-step to the critical section of code
Why would you ever be single-stepping? Put a break point (conditional if necessary) where you would put the print statement. The difference between a single break point and a print statement is that the break point will allow you to inspect the local variables associated with all calls in the stack trace and evaluate further expressions.
So when do you debug instead of using print statements? When you know that no matter what the outcome of your hypothesis is, that you will need to iteratively inspect details from other points up the stack. That is, when you know, from experience, that you are going to need further print statements but you don't know where they will be.
For example, one thing you wrote that jumps out at me:
> I already have a working model for the way that the code runs [...]
This is not always true. It's only true for code that I wrote or know very well. E.g. as a consultant, I often work on codebases that are new to me, and I do tend to use debuggers there more often than I use print debugging.
Although lots of other variables affect this - how much complicated state there is to get things running, how fast the "system" starts up, what language it's written in and if there are alternatives (in some situations I'll use a Jupyter Notebook for exploring the code, Clojure has its own repl-based way of doing things, etc).
1. Printing vars in unit tests may be the fastest first approach. If i know where the bug may be.
2. When that fails i usually bring in debuggers to unit tests.
3. When these aren't helping, you need debuggers on the entire binary.
4. Still stuck? Use a debugger in production.
3. What if the binary interacts with other networked computers, you gonna debug all of them? Do you end up instrumeting the whole internet? You scope out, you spiral out of control until someone puts a limit.
However, I use prints a lot more because you can, as you say, usually get to the answer faster.
This in no way disagrees with the quote. Both can be true. The quote isn’t saying debuggers can’t provide unique insights, just that for the majority of debugging the print statement is a faster way to get what you need.
I use a debugger when I've constructed a complex process that has a large cardinally of states it could end up in. There is no possibility that I can write logic checks (tests) for all source inputs to that state.
I don't use one when I could simply increase test situations to find my logical error.
Consider the difference between a game engine and a simple state machine. The former can be complex enough to replicate many features of the real world while a simple state machine/lexer probably just needs more tests of each individual state to spot the issue.
Professional debuggers such as the one in IntelliJ IDEA are invaluable regardless of one's familiarity with a given codebase, to say otherwise is utter ignorance. Outside of logging, unless attaching a debugger is impractical, using print statements is at best wasting time.
Than what? In languages with good debugger support (see JVM/Java) it can be far quicker to click a single line to set a breakpoint, hit Debug, the inspect the values or evaluate expressions to get the runtime context you cant get from purely reading code. Print statements require rebuilding code and backing them out, so its hard to imagine that technique being faster.
I do use print debugging for languages with poor IDE/debugger support, but it is one big thing I miss when outside of Java.
I didn't really get into debuggers until (1) I was firmly in Windows, where you expect the GUI to work and the LI to be busted, and (2) I'd been burned too many times by adding debugging printfs() that got checked into version control and caused trouble.
Since then I've had some adventures with CLI debuggers, such as using gdb to debug another gdb, using both jdb and gdb on the same process at the same time to debug a Java/C++ system, automating gdb, etc. But there is the thing, as you say, is that there is usually some investment required to get the debugger working for a particular system.
With a good IDE I think Junit + debugging gives an experience in Java similar to using the REPL in a language like Python in that you can write some code that is experimental and experiment it, but in this case the code doesn't just scroll out of the terminal but ultimately gets checked in as a unit test.
One of the first things I do in a codebase is get some working IDE/editor up where I can quickly run the program under a debugger, even if I'm not immediately troubleshooting something. It's never long before I need to use it.
I was baffled when I too encountered this. Even working collaboratively with people they'd have no concept of how to use a debugger.
"No, set a breakpoint there"
"yeah now step into the function and inspect the state of those variables"
"step over that"
: blank stares at each instance :
I don't use debuggers as often in Python, probably because it's eaiser to throw code in a notebook and run line by line to inspect variables, change/inject state and re-run. That's possible but a lot harder to do in C++.
Also for embedded work, using a debugger and memory viewer is pretty powerful. It's not something people think about for Arduino but almost every commodity micro supports some sort of debugwire-like interface (which is usually simpler than JTAG).
I only used debugger recently in C# and C, when I was learning and practicing them.
A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values. I add more detailed logging as I start to use the system and find out what needs extra scrutiny. This approach is very easy to get started with and maintain, and provides powerful insight into problems as they arise.
I also put a lot of work into formatting log statements. I once worked on a distributed system, and getting the prefix of each log statement exactly right was very useful -- node id, pid, timestamp, all of it fixed width. I could download logs from across the cluster, sort, and have a single file that interleaved actions from across the cluster.
I can always drop an entire state object into the log if I need it, but the only way for a debugger to approximate what a log can give me is for me to step through a bunch of break points and hold the time stream in my head.
The one place where a debugger is straight up better is if I know exactly which unit of code is failing and that unit has complicated logic that is worth stepping through line by line. That's what they were designed for, and they're very useful for that, but it's also not the most common kind of troubleshooting I run into.
- Log4j + MDC's so we could get a timestamp and thread ID easily
- some python scripting
I can't remember if it was PDF or SVG, but I do remember having to choose something that would allow for huge graphs and lots of scrolling!
These days there's better options on github, but nice to see I was on a decent path.
You can script this so all function entry/exists, or whatever, are logged without touching the code or needing to recompile. You can then bulk toggle these breakpoints, at runtime, so you only see a particular subset when things get interesting.
Modifying the code to print stuff will feel barbaric after driving around a fine tuned debugging experience.
Print debugging, while not clever or powerful, has never once failed me.
You can attach lldb without Xcode. Or you can open the lldb terminal in Xcode, pause execution, and inspect the breakpoints manually
Some IDEs promise to solve that, but I’ve not been impressed thus far.
YMMV based on language/runtime/toolkit of course. This might get added to my wishlist for my next language of choice.
This is an anti-pattern which results in voluminous log "noise" when the system operates as expected. To the degree that I have personally seen gigabytes per day produced by employing it. It also can litter the solution with transient concerns once thought important and are no longer relevant.
If detailed method invocation history is a requirement, consider using the Writer Monad[0] and only emitting log entries when either an error is detected or in an "unconditionally emit trace logs" environment (such as local unit/integration tests).
0 - https://williamyaoh.com/posts/2020-07-26-deriving-writer-mon...
I usually roll my own wrapper around an existing logging package, but https://www.npmjs.com/package/debug is a good example of what life can be like if you're using JS. Want to debug your rate limiter? Write `DEBUG=app:middleware:rate-limiter npm start` and off you go.
It is an anti-pattern due to what was originally espoused:
I add logging code, that stays forever. For a major
interface, I'll usually start with INFO level debugging, to
document function entry/exit, with param values.
Also, you did not address the non-trivial cost implications of voluminous log output.
> You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".
Regarding the above, production-ready logging libraries such as Logback[0], log4net[1], log4cpp[2], et al, allow for run-time configuration to determine what "areas" will have their entries emitted. So "all logs all the time" is a non sequitur in this context.
What is relevant is the technique identified of emitting execution context when it matters and not when it doesn't. As to your `npm` example, I believe this falls under the scenario I explicitly identified thusly:
... an "unconditionally emit trace logs" environment
(such as local unit/integration tests).
> the non-trivial cost implications of voluminous log output
If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
I assumed nothing of the sort. What I did was identify an anti-pattern and describe an alternative which experience has shown to be a better approach.
"Incompetence" is your word, not mine.
> ... and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.
I fail to see the nuance you apparently can see.
>> the non-trivial cost implications of voluminous log output
> If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
Cloud deployment requires transmission of log entries to one or more log aggregators in order to be known.
By definition, this involves network I/O.
Network communication is orders of magnitude slower than local I/O.
Useless logging of "function entry/exit, with param values" increases pressure on network I/O.
Unless logging is allowed to be lossy, which it never is, transmission must be completed when log buffers are near full capacity.
Provisioning production systems having excessive logging can often require more resources than those which do not excessively log.
Thus disproving:
> ... even at runtime the costs are often trivial.
When considering the implication of voluminous log output in a production environment.
The code that computes data for and assembles your log message may end up executing logic that affects the system elsewhere. If you put that code under conditional, your program will behave differently depending on the logging configuration; if you put it outside, you end up wasting potentially substantial amount of work building log messages that never get used.
As I said, responding to another comment of yours, a distributed system I worked on produced a few GB a day. The logs were rotated daily. They were never transmitted anywhere, during normal operation. When things go wrong, sure, we look at them, and generate even more logging. But that was rare. I cannot stress enough how much of a non-issue log volume was in practice.
So I ask you to quantify: What counts (to you) as voluminous, as in daily log file sizes, and how many times they are sent over the network?
I mean "a lot" or more specifically; "a whole lot."
Here is an exercise which illustrates this. For the purposes here, assume ASCII characters are used for log entries to make the math a bit easier.
Suppose the following:
Each log statement is 100 characters.
Each service invocation emits 50 log statements.
Average transactions per second during high usage is 200tps.
High usage is on average 2 hours per day.
100 x 50 x 200 x 60 x 60 x 2 = 7_200_000_000 = 7.2GB / day
The quantification is above and regarding log entries being sent over a network - in many production systems, log entries are unconditionally sent to a log aggregator and never stored in a local file system.
But in normal usage, even in the scenario you describe your argument about log volume is not convincing.
Resorting to an ad hominem and non sequiturs are we? Really?
C'mon, you are better than that.
Well, I agree completely, but those conditions are a tall order. The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended. If I have a failure, and suspect a major subsystem, I sure do want to see all calls and param values leading up to a failure.
In addition to this point, you have constructed a strawman in which logging is on all the time. Have you ever looked at syslog? On my desktop Linux system, output there counts as voluminous. It isn't so much space, or so CPU-intensive that I would consider disabling syslog output (even if I could).
The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue. And for the rare times that something did fail, we could turn up logging with precision and get useful information.
> Well, I agree completely, but those conditions are a tall order.
Every successful service invocation satisfies "all collaborations succeed and the system operates as intended." Another way to state this is every HTTP `1xx`, `2xx`, and `3xx` response code produced by an HTTP service qualifies as such.
> The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended.
Providing sufficient context in the presence of errors, or "situations in which things don't succeed or operate as intended", was addressed thusly:
If detailed method invocation history is a requirement,
consider using the Writer Monad and only emitting log
entries when either an error is detected or in an
"unconditionally emit trace logs" environment (such as
local unit/integration tests).
See above.
> In addition to this point, you have constructed a strawman in which logging is on all the time.
No, I addressed your original premise in the context of a production web application, where logging is configured during deployment.
See also your own contradiction by previously asserting, "I sure do want to see all calls and param values leading up to a failure."
So which is it?
Did I construct a strawman "in which logging is on all the time"?
Or do you "want to see all calls and param values leading up to a failure", which requires "logging is on all the time"?
> Have you ever looked at syslog?
This is a strawman. Syslog is a component for logging and has nothing to do with the programs which use it.
> The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue.
If this is the same system you described in a different comment also in this thread, I identified a standard industry practice of log entries produced by application nodes in a production environment being unconditionally sent to a log aggregator and not stored in a local file system. The reasons for this are well documented.
I don't know from Writer Monads. But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory? What about log volume? Does this writer monad implement log rotation? It sounds like you are sweeping a lot of the things you object to under this writer monad rug.
Let me be real clear about all calls and param values leading up to a failure.
- In normal operation, turn logging off completely, or turn on some level that produces tolerable log volume, (it seems like your threshold is much lower than mine).
- When a failure occurs: Restart the service with more logging enabled, (hence the all calls an param values), so that you have logging when the failure occurs again.
About local logs vs a log aggregator: The system I worked on was a shared nothing archive. To add storage and metadata capacity, you add nodes. Each node also stored its own log files. I get that this may not be the answer all the time, and that a log aggregator is useful in some scenarios. However, even in that case, your concerns about log volume seem overblown to me.
My point was to identify how common the "happy path" scenario is and was in direct response to:
>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.
> Well, I agree completely, but those conditions are a tall order.
Regarding your question:
> What if the HTTP server crashes, and doesn't send a response at all?
Again, the HTTP status codes were used to illustrate the frequency of successful invocations. But to your point, if an HTTP server crashes then log entries for in-flight workflows would likely not be emitted. A similar possibility also holds for local file system logging as buffering exists (sometimes on multiple levels).
> I don't know from Writer Monads.
No worries. All it is is a formal definition of a type having specific capabilities called "laws" in the functional programming world.
> But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory?
What is meant by "future failure" is a failure potentially encountered during the evaluation of a single invocation. In the context of a HTTP server, this would be the handling of each submission to a specific HTTP endpoint and verb. This is often defined within an IO Monad[0], but does not have to be, and is out of scope for this discussion.
The answer to the rest of your questions is the deferred log entry definitions are held in memory for the duration of the single service invocation, with any log emissions produced transmitted to a network-accesible log aggregator via the logging component used.
> Let me be real clear about all calls and param values leading up to a failure.
The scenario you kindly shared is understandable, yet is one which has been unacceptable in teams I have worked with. Losing the "first error" is not an option in those environments.
0 - https://en.wikibooks.org/wiki/Haskell/Understanding_monads/I...
What you describe is leaving around hand-rolled instrumentation code that conditionally executes expensive reporting actions, which you can toggle on demand between executions. Thing is, this is already all done automatically for you[1] - all you need is the right build flag to prevent optimizing away information about function boundaries, and then you can easily add and remove such instrumentation code on the fly with a debugger.
I mean, tracing function entry and exit with params is pretty much the main task of a debugger. In some way, it's silly that we end up duplicating this by hand in our own projects. But it goes beyond that; a lot of logging and tracing I see is basically hand-rolling an ad hoc, informally-specified, bug-ridden, slow implementation of 5% of GDB.
Why not accept you need instrumentation in production too, and run everything in a lightweight, non-interactive debugging session? It's literally the same thing, just done better, and couple layers of abstraction below your own code, so it's more efficient too.
--
[0] - https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
[1] - Well, at least in most languages used on the backend, it is. I'm not sure how debugging works in Node at the JS level, if it exists at all.
I think there's still value in adding toggleable debug output to major interfaces. It tells you exactly what and where the important events are happening, so that you don't need to work out where to stick your breakpoints.
I don't want logging done automatically for me, what I want is too idiosyncratic. While I will log every call on major interfaces, I do want to control exactly what is printed. Maybe some parameter values are not of interest. Maybe I want special formatting. Maybe I want the same log line to include something computed inside the function. Also, most of my logging is not on entry/exit. It's deeper down, to look at very specific things.
Look, I do not want a debugger, except for tiny programs, or debugging unit tests. In a system with lots of processes, running on lots of nodes, if a debugger is even possible to use, it is just too much of a PITA, and provides far too miniscule a view of things. I don't want to deal with running to just before the failure, repeatedly, resetting the environment on each attempt, blah, blah, blah. It's a ridiculous way to debug a large and complex system.
What a debugger can do, that is harder with logging, is to explore arbitrary code. If I chase a problem into a part of my system that doesn't have logging, okay, I add some logging, and keep it there. That's a good investment in the future. (This new logging is probably at a detailed level like DEBUG, and therefore only used on demand. Obvious, but it seems like a necessary thing to point out in this conversation.)
The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.
This is useful when you encounter unknown problem and need to find unusual stuff in the logs. This unusual stuff is not always an error state, it might be some aggregate problem (something is called too many times, something is happening in problematic order, etc.)
A case can be made for only logging the steps performed up to and including an error. This even excludes logging "important things" other than those satisfying system/functional requirements (such as request/response audit logs).
It is reminiscent of "the Unix philosophy", but different in important ways, and is essentially:
Capture what would be log entries if a there is a
future unrecoverable error.
If an error is encountered, emit *all* log entries starting
from the earliest point (such as receiving an external
event or a REST endpoint request) up to and including the
information detailing the unrecoverable error.
If the workflow succeeds, including producing an expected
failed workflow response (such as a validation error),
discard the deferred log entries.
While using a functional programming language and employing referentially transparent[0] abstractions (such as the Writer Monad) usually makes implementing this pattern much simpler than when using an imperative language, it can be successfully done with the latter given sufficient discipline and the workflow implementation being referentially transparent[0].
An important complement to the above is to employ other industry standard verification activities, such as unit/feature/integration tests.
It's one point in favor of async, and language that make async debugging easy.
And at the very least, you can encapsulate things so they can be debugged separately from the multi threading stuff.
If you're always using the debugger, you learn to build around it as one of your primary tools for interacting with code, and you notice right away if the debugger can't help with something.
Wtaf?
If there isn't one, I'd rather use a language with a debugger and write a thousand lines of code than 100 lines of code in a language I'm going to have to black box.
What sold me on debugger is the things you can do with it
* See values and eval expressions in calling frames.
* Modify the course of execution by eval'ing a mutating expression.
* Set exception breakpoints which stop deep where the exception is raised.
Not any excuse, but another factor to be considered when adding infra layers between the developer and the application.
Debuggers actually can hide entire categories of bugs caused by race conditions when breakpoints cause async functions to resolve in a different order than they would when running in realtime
I've never worked somewhere where logging is taken seriously. Like, our AWS systems produce logs and they get collected somewhere, but none of our code ever does any serious logging.
If people like print-statement debugging so much, then double down on it and do it right, with a proper logging framework and putting quality debug statements into all code.
That's all event-sourcing is.
By all means, learn to use a debugger well but don't become overly dependent on them.
If you usually aren't able/allowed to use a debugger in production and must rely on observability tools, it's helpful to know how to utilize those tools locally as effectively as possible when debugging.
I really don't get this at all. To me it is infinitely easier to iterate and narrow the problem rather than trying to identify sight-unseen where the problem is—it's incredibly rare that the bug immediately crashes the program. And you can fit a far higher density of relevant information through print statements over execution of a reproduced bug than you can reproduce at any single point in the call stack. And 99% of the information you can access at any single point in the call stack will be irrelevant.
To be sure, a debugger is an incredibly useful and irreplaceable tool.... but it's far too slow and buggy to rely on for daily debugging (unless, as you indicate, you don't know the codebase well enough to reason about it by reading the code).
Things that complicate this:
* highly mutable state
* extremely complex control or data flow
* not being able to access logs
* the compiler lying to you or being incorrect
* subtle instruction ordering issues
i'd rather never use debugger, so that my coding style is enforced to be clean code, strong type system, immutable variables, explicit error control, explicit control flow etc
In short, which is better? It depends, and varies wildly by domain.
The only thing that's really missing is history of the values of the watches you added, all in one log. With filters because why not.
For some reason I never seen this option in any debugger I tried.
Heh, who knew, debuggers already have this feature under the name tracepoints (VS) or logpoints (Viscose).
https://code.visualstudio.com/blogs/2018/07/12/introducing-l...
It should be way more well known among print debug crowd.
TIL Linus Torvald is a young grug.
Now, some 35 years later, I greatly prefer logs. This way I can compare execution paths of different use cases, I can compare outcomes of my changes, etc. I am not confined to a single point of time with tricky manual control as with debugger, I can see them all.
To old grugs: learning to execute and rewind code in your head is a major superpower. And it works on any codebase.
However debugging is an essential tool in the arsenal. If something is behaving oddly even the best REPL can't match debugging as a dev loop (maybe lisp excepted).
I even miss the ability to move the current execution point back in .NET now I use Go and JS. That is a killer feature. Edit and continue even more so!
Then next level is debugging unit tests. Saved me hours.
Which is a good thing! Easily comprehended code that you can reason about without stepping through it is good for grug brain.
But in 2000 I started working in a role which required understanding the misbehavior of embedded systems that were forwarding the live traffic, there was a technical possibility to do “target remote …” but almost never an option to stop a box that is forwarding the traffic.
So you end up being dependent on debugs - and very occasional debug images with enhanced artisanal diagnostics code (the most fun was using gcc’s -finstrument-function to catch a memory corruption of an IPSec field by an unrelated IKE code in a use-after free scenario)
Where the GDB shined though is the analysis of the crash dumps.
Implementing a “fake” gdb stub in Perl, which was sucking in the crash dump data and allow to leisurely explore it with debugger rather than decoding hex by hand, was a huge productivity boon.
So I would say - it’s better to have more than one tool in the toolbox and use the most appropriate one.
Your experience sounds more sophisticated than mine, but the one time I was able to get even basic debugger support into a production Ruby app, it made fixing certain classes of bug absolutely trivial compared to what it would have been.
The main challenge was getting this considered as a requirement up front rather than after the fact.
Sigh. Same. To a large extent, this is caused by debuggers just sucking for async/await code. And just sucking in general for webdev.
On the other hand, the humble console.log() just works without requiring insanely tedious and frustrating debugger steps.
Line debuggers usually have some nice conveniences, but the major bottlenecks are between the ears, not in the tool.
So long as no one sees your workflow as valuable, they will happily destroy it if it means getting the ticket done.
I used to debug occasionally but haven't touched a debugger in years. I'm not sure exactly why this is, but I'm generally not concerned with exactly what's in the stack on a particular run, but more with some specific state and where something changes time after time, and it's easier to collect that data programatically in the program than to figure out how to do it in the debugger that's integrated with whatever IDE I'm using that day.
And the codebase I deal with at work is Spring Boot. 90% of the stack is meaningless to me anyway. The debugger has been handy for finding out "what the f*$% is the caller doing to the transaction context" but that kind of thing is frustrating no matter what.
Anyway, I think they're both valid ways to explore the runtime characteristics of your program. Depends on what you're comfortable with, how you think of the problem, and the type of codebase you're working on. I could see living in the debugger being more useful to me if I'm in some C++ codebase where every line potentially has implicit destructor calls or something.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
Another way (and probably a better idea) is creating a launch definition for VS Code in launch.json which attaches the IDE's debugger to Chrome. Here is an article describing how that works: https://profy.dev/article/debug-react-vscode
Breakpoints are nice because they can't accidentally get shipped to production like debugger calls can.
For Python I essentially do the same thing, minus involving Chrome. I run the entry point to my application from launch.json, and breakpoints in the IDE 'just work'. From there, you can experiment with how the debugger tools work, observe how state change as the application runs, etc.
If you don't use VS Code, these conventions are similar in other IDEs as well.
Even assuming print statements and debuggers are equally effective (they're not), debuggers are better simply because they are faster. With print statements, you might need to recompile a dozen times before you find whatever it is you're looking for. Even with quick builds this is infuriating and a ton of wasted time.
The trouble of course is that there is always money to be made debugging. There is almost no incentive in industry to truly eliminate bugs and, indeed, I would argue that the incentives in the industry actively encourage bugs because they lead to lucrative support contracts and large dev teams that spend half their time chasing down bugs in a never-ending cycle. If a company actually shipped perfect software, how could they keep extracting more money from their customer?
Built code largely works with source maps, but it fails often enough, and in bizarre ways, that my workflow has simply gone back to console logs.
React's frequent re-renders have also made breakpoints very unpleasant - I'd rather just look at the results of console logs.
Are there ways I can learn to continue enjoying the debugger with TS+React? It is still occasionally useful and I'm glad its there, but I have reverted to defaulting to console logs.
But yes, any code that is inside jsx generally sucks to debug with standard tooling. There are browser plugins that help you inspect the react ui tree though.
Conditional breakpoints help alleviate the pain when there are frequent re-renders. Generally you can pinpoint a specific value that you're looking for and only pause when that condition is satisfied. Setting watch expressions helps a lot too.
But the reason is simple: they are always 100x harder to set up and keep working than just type a print statement and be done with it.
Especially when you're in a typical situation where an app has a big convoluted docker based setup, and with umpteen packers & compilers & transpilers etc.
This is why all software companies should have at least one person tasked with making sure there are working debuggers everywhere, and that everyone's been trained how to use them.
There should also be some kind of automated testing that catches any failures in debugging tooling.
But that effort's hard to link directly to shipped features, so if you started doing that, management would home in on that dev like a laser guided missile and task him to something with "business value", thereby wasting far more dev time and losing far more business value.
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
Now I understand that “architecture” and “patterns” is a jobs program for useless developers. It’s this, or they’d be on the streets holding a sign saying “will write JavaScript for a sandwich”.
Yet, developers are always using patterns and are thinking about architecture.
Here you are doing so too, a pattern, "form submission" and an architecture, "request-response".
I'm curious what role you have where you're doing this repeatedly
The agencies are similarly uncoordinated and will pick up their metaphorical credit card and just throw it at random small dev teams, internally, external, or a mix.
Those people will happily take the credit! The money just… disappears. It’s like a magic trick, or one of those street urchins that rips you off when you’re on holiday in some backwards part of the world like Paris.
I get brought in as “the cloud consultant” for a week or two at the end to deploy the latest ball of mud with live wires sticking out of it to production.
This invariably becomes an argument because the ball of mud the street urchins have sold to the customer is not fit for… anything… certainly not for handling PII or money, but they spent the budget and the status reports were all green ticks for years.
Fundamentally, the issue is that they're "going into the cloud" with platform as a service, IaC, and everything, but at some level they don't fully grok what that means and the type of oversight required to make that work at a reasonable cost.
"But the nice sales person from Microsoft assured me the cloud is cheaper!"
In fact we're going through one of these SAP HANA migrations at present and it's very broken, because the prime contractor has delivered a big ball of mud with lots of internal microservices.
These new superdepartments all have the same issues to the point that I sometimes confuse them.
This alone earns my upvote.
One app I got brought in late on the architect had done some complicated mediator pattern for saving data with a micro service architecture. They'd also semi-implemented DDD.
It was a ten page form. Literally that was what it was supposed to replace. An existing paper, 10 page, form. One of those "domains" was a list of the 1,000 schools in the country. That needed to be updated once a year.
A government spent millions on this thing.
I could have done it on my todd in 3 months. It just needed to use simple forms, with some simple client side logic for hiding sections, and save the data with an ORM.
The funniest bit was when I said that it couldn't handle the load because the architecture had obvious bottlenecks. The load was known and fairly trivial (100k form submissions in one month).
The architect claimed that it wasn't possible as the architecture was all checked and approved by one of the big 5.
So I brought the test server down during the call by making 10 requests at once.
Back in the very early 2000s I got sent to "tune IIS performance" at a 100-developer ISV working on a huge government project.
They showed me that pressing the form submit button on just two PCs at once had "bad performance".
No, not it didn't. One was fast[1], the other took 60 seconds almost exactly. "That's a timeout on a lock or something similar", I told them.
They then showed me their 16-socket database server that must have cost them millions and with a straight face asked me if I thought that they needed to upgrade it to get more capacity. Upgrade to what!? That was the biggest machine I have ever seen! I've never in the quarter century since then seen anything that size with my own two eyes. I don't believe bigger Wintel boxes have ever been made.
I then asked their database developers how they're doing transactions and whether they're using stored procedures or not.
One "senior" database developer asked me what a stored procedure is.
The other "senior" database developer asked me what a transaction is.
"Oh boy..."
[1] Well no, not really, it took about a second, which was long enough for a human button press to to "overlap" the two transactions in time. That was a whole other horror story of ODBC connection pooling left off and one-second sleeps in loops to "fix" concurrency issues.
This is the real, actual conversation to be had about "AI taking jobs."
I've seen similar things a lot in the private sector.
There's just loads of people just flailing around doing stuff without really having any expertise other than some vague proxy of years of experience.
It's really not even exactly their fault (people have lives that don't revolve around messing about with software systems design, sure, and there's no good exposure to anything outside of these messes in their workplaces).
But, outside of major software firms (think banks, and other non-"tech" F500s; speaking from experience here) there's loads of people that are "Enterprise Architects" or something that basically spend 5 hours a day in meetings and write 11 lines of C# or something a quarter and then just adopt ideas they heard from someone else a few years back.
Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.
Most other fields are similar, frankly.
One of my pet-peeves. "We're doing DDD, hexagonal architecture, CQRS". So, when was the last time your dev team had a conversation with your domain experts? You have access to domain experts don't you? What does your ubiquitous language look like?
So no, some "senior" read a blog post (and usually just diagonally) and ran with it and now monkey see monkey does is in full effect.
And you get the same shit with everything. How many "manager" read one of the books about the method they tell you they're implementing (or any book about management) ? How many TDD shop where QA and dev are still separate silos? How many CI/CD with no test suite at all? Kanban with no physical board, no agreed upon WIP limits, no queue replenishing system but we use the Kanban board in JIRA.
"We're all-in on using Kanban here"
"Ah, great. What's your current WIP limit?"
"Um, what's a whip limit?"
As a consultant, I don't actually mind finding myself in the midst of that sort of situation - at the very least, it means I'm going to be able to have a positive impact on the team just by putting in a bit of thought and consistent effort.
If you mix in two services into a single database - no matter how good the logical and security isolation is — they will roll back their transactions together if the DBA presses the restore button.
Similarly they have the option (but not the obligation!) to participate in truly atomic transactions instead of distributed transactions. If this is externally observable then this tight coupling means they can no longer be treated as separate apps.
Many architects will just draw directed arrows on diagrams, not realising that any time two icons point at the same icon it often joins them into a single system where none of the parts are functional without all of the others.
My grug brain conclusion: Grug see good microservice in many valley. Grug see grug tribe carry good microservice home and roast on spit. Grug taste good microservice, many time. Shaman tell of good monolith in vision. Grug also dream of good monolith. Maybe grug taste good monolith after die. Grug go hunt good microservice now.
But you are right that the remote procedure call is a fraught concept for more reasons than one. On one hand there is the fundamental difference between a local procedure call that takes a few ns and a remote call which might take 1,000,000 longer. There's also the fact that most RPC mechanisms that call themselves RPC mechanisms are terribly complicated, like DCOM or the old Sun RPC. In some sense RPC became mainstream once people started pretending it was REST. People say it is not RPC but often you have a function in your front end Javascript like fetch_data(75) and that becomes GET /data/75 and your back end JAXB looks like
@GET
@Path("/{id}")
public List<Data> fetchData(@PathParam("id") int id) { ... }> I think it's best to break things down
Factoring system = break things down.
I think this is correct as an explanation for the phenomenon, but it's not just a false perception on their part: for a lot of organizations it is actually true that the only way to preserve boundaries between systems over the course of years is to stick the network in between. Without a network layer enforcing module boundaries code does, in fact, tend to morph into a big ball of mud.
I blame a few things for this:
1. Developers almost universally lack discipline.
2. Most programming languages are not designed to sufficiently account for #1.
It's not a coincidence that microservices became popular shortly after Node.js and Python became the dominant web backend languages. A strong static type system is generally necessary (but not sufficient) to create clear boundaries between modules, and both Python and JavaScript have historically been even worse than usual for dynamic languages when it comes to having a strong modularity story.
And while Python and JS have it worse than most, even most of our popular static languages are pretty lousy at giving developers the tools needed to clearly delineate module boundaries. Rust has a pretty decent starting point but it too could stand to be improved.
Many companies don't seem to do a good job coordinating between teams. Different teams have different incentives and priorities. If group A needs fixes/work from group B and B has been given some other priority, group A is stuck.
By putting a network between modules different groups can limit blast damage from other teams' modules and more clearly show ownership when things go wrong. If group A's project fails because of B's module it still looks like A's code has the problem.
Upper management rarely cares about nuance. They want to assign blame, especially if it's in another team or department. So teams under them always want clear boundaries of responsibility so they don't get thrown under the bus.
The root cause of a lot of software problems is the organization that produces it more than any individual or even team working on it.
— Melvin E. Conway, How Do Committees Invent?
Or developers are given a deadline and no slack to learn the code base. So developers will tactically take the fastest route to closing their ticket.
It's not developers that lack discipline. It's CTOs, VPs, etc.
I must scream but I'm in a vacuum. Everyone is fine with this.
(Also it takes a few seconds to process a 500 line test file and runs for ten minutes on the real 20k line input.)
That's still probably the wrong way to do it, but 10 minutes for a 20k line file? That seems like poor engineering in the most basic sense.
But the guy 'is known to get the job done' apparently.
And to be fair, if folks didn’t get paid for reinventing basic Unix utilities with extra steps, the economy would probably collapse.
Then you deploy the server program, and then you deploy the client program, to another machine, or machines, where they continue to talk to one another over http, maybe over the public Internet or maybe not.
Deploying can mean any one of umpteen possible things. In general, you (use automations that) copy your programs over to dedicated machines that then run your programs.
* Information hiding by exposing a closed interface via the API
* Isolated, scalable, fault tolerant service
* Iterable, understandable and super agile
I'm going to laugh pretty hard if it could just be done with: cat file1.csv file2.csv > combined.csv
So I guess something like this to skip the headers in the second file (this also assumes that headers don't have line breaks):
cp file1.csv combined.csv && tail -n+2 file2.csv >> combined.csv- Cannot be run without an orchestrator like K8S, which is a bear to install and maintain, which helps sell managed cloud.
- Uses more network bandwidth, which they bill for, and CPU, which they bill for.
- Makes it hard to share and maintain complex or large state within the application, encouraging the use of more managed database and event queue services as a substitute, which they bill for. (Example: a monolith can use a queue or a channel, while for microservices you’re going to want Kafka or some other beast.)
- Can’t be run locally easily, meaning you need dev environments in cloud, which means more cloud costs. You might even need multiple dev and test environments. That’s even more cloud cost.
- Tends to become dependent on the peculiarities of a given cloud host, such as how they do networking, increasing cloud lock in.
Anyone else remember how cloud was pitched as saving money on IT? That was hilarious. Knew it was BS way back in the 2000s and that it would eventually end up making everything cost more.
Don't forget various pipelines, IaC, pipelines for deploying IaC, test/dev/staging/whatever environments, organization permissions strategies etc etc...
When I worked at a large, uh, cloud company as a consultant, solutions were often tailored towards "best practices"--this meant, in reality, large complex serverless/containerized things with all sorts of integrations for monitoring, logging, NoSQL, queues etc, often for dinky little things that an RPI running RoR or NodeJS could serve without breaking a sweat.
With rare exceptions, we'd never be able to say, deploy a simple go server on a VM with server-side rendered templates behind a load balancer with some auto-scaling and a managed database. Far too pedestrian.
Sure, it's "best practices" for "high-availability" but was almost always overkill and a nightmare to troubleshoot.
It’s hard to hire people to do anything else. People don’t know how to admin machines so forget bare metal even though it can be thousands of times cheaper for some work loads (especially bandwidth).
You’re not exaggerating with a raspberry pi. Not at all.
The "entire generation of developers" paradigm is all over in different domains too--web programmers that seem to honestly think web development is only React/Angular and seem to have no idea that you can just write JS, python programmers that have no idea a large portion of the "performant codebases" are piles of native dependencies etc
Same goes for users. There are about the same number of computer literate users as there were back then. There’s just a new class of iPad/iPhone user who is only a casual user of computers and the net and barely knows what a file is.
The scaling problems we face could probably be solved by other solutions, but the company is primed and ready to chuck functionality into new microservices. That’s what all our infrastructure is set up to do, and it’s what inevitably happens every time
You could probably get most of this without the literal network hop, but I haven't seen a serious attempt.
Or maybe I’m missing what you mean - can you explain with an example an API boundary you can’t define by interfaces in Go, Java, C# etc? Or by Protocols in Python?
from submodule import pandas
microservices putting a network call in on the factoring is a feature in this case, not a bug. it’s a physical blocker stopping devs doing stuff like that. it’s the one thing i don’t agree with grug on.
HOWEVER — it’s only a useful club if you use it well. and most of the time it’s used because of expectations of shiny rocks, putting statements about microservices in the company website, big brain dev making more big brain resume.
You can do this in Python as well, but it does involve a bit of care; I like the pattern of a module named “internal” that has the bulk of the modules code in it, and a small public api.py or similar that explicitly exposes the public bits, like an informal version of the compiler-enforced pattern for this in Go
Of course the best is mono repo and monolith :3
If you prefer working on mono repo it s fine though I think micro services are more popular one now.
If so: mono repo is not an alternative to micro services. You can have both. You can have micro services in a mono repo. One single repository with many micro services. That is possible.
What is an alternative to micro services is “monolith”.
Curious how I’m misunderstanding this because I looked at it for a while and I really don’t see it.
I think about this line at least once a week
this grug keeps one on one invisible t-rex, grug cursed
Tyrannosaurs were built for speed. Their leg bones were hollow, like a bird’s. If he could break a femur …
The shovel connected, but not solidly. It hit without breaking anything. But, still, it got tangled up in those powerful legs. With enormous force, it was wrenched out of his hands. Leyster was sent tumbling on the ground.
Somebody was screaming. Dazed, Leyster raised himself up on his arms to see Patrick, hysterically slamming the juvenile, over and over, with the butt of the shotgun. He didn’t seem to be having much effect. Scarface was clumsily trying to struggle to its feet. It seemed not so much angry as bewildered by what was happening to it.
Then, out of nowhere, Tamara was standing in front of the monster. She looked like a warrior goddess, all rage and purpose. Her spear was raised up high above Scarface, gripped tightly in both hands. Her knuckles were white.
With all her strength, she drove the spear down through the center of the tyrannosaur’s face. It spasmed, and died. Suddenly everything was very still."
There is of course a time and place for sophistication, pushing for higher levels of abstraction and so on. But this grug philosophy is saying that there isn't any inherent value in doing this sort of thing and I think that is very sound advice.
Also I noticed AI assistance is more effective with consistent, mundane and data driven code. YMMV
Novice dev writes simple code
Intermediate dev writes complex code
Expert dev writes simple code
Something along the lines of "Hey, you're a great developer, really smart, you really know your stuff. But you have to stop reaching for the most complicated answer to everything"
He took it to heart and got promoted at the start of this year. Was nice to see. :)
Three examples:
DRY (Don't Repeat Yourself) sometimes leads to premature abstraction. We think, "hey, I bet this pattern will get used elsewhere, so we need to abstract out the common parts of the pattern and then..." And that's when the Complexity Demon enters.
We want as many bugs as possible caught at compile-time. But that means the compiler needs to know more and more about what we're actually trying to do, so we come up with increasingly complex types which tax your ability to understand.
To avoid boilerplate we create complex macros or entire DSLs to reduce typing. Unfortunately, the Law of Leaky Abstractions means that when we actually need to know the underlying implementation, our head explodes.
Our challenge is that each of these examples is sometimes a good idea. But not always. Being able to decide when to introduce complexity to simplify things is, IMHO, the mark of a good software engineer.
To modulate DRY, I try to emphasize the “rule of three”: up to three duplicates of some copy/paste code is fine, and after that we should think about abstracting.
Of course no rule of thumb applies in all cases, and the sense for that is hard to teach.
Student: I notice that you duplicated code here rather than creating an abstraction for both.
Master: That is correct.
Student: But what if you need to change the code in the future?
Master: Then I will change it in the future.
At that point the student became enlightened.
Just for fun, this more or less already exists as another acronym: WET. Write Everything Twice
It basically just means exactly what you said. Don't bother DRYing your code until you find yourself writing it for the third time.
I totally agree. For example having 5 variables that are all the same value but mean very different things is good. Combining them to one variable would be "DRY" but would defeat separations of concern. With variables its obvious but the same applies to more complex concepts like functions, classes, programs to a degree.
It's fine to share code across abstractions but you gotta make sure that it doesn't end up tying these things too much together just for the cause of DRY.
Usually I make web applications using Sinatra-like frameworks like Flask or JAXB where I write a function that answers URLs that match a pattern and a "screen" is one or more of those functions that work together and maybe some HTML templates that go with them. For instance there might be a URL for a web page that shows data about a user, and another URL that HTMX calls when you flip a <select> to change the status of that user.
Assuming the "application" has the stuff to configure the database connection and file locations and draw HTML headers and footers and such, there is otherwise little coupling between the screens so if you want to make a new screen you can cut and paste an old screen and modify it, or you can ask an LLM to make you a screen or endpoint and if it "vibe coded" you a bad screen you can just try again to make another screen. It can make sense to use inheritance or composition to make a screen that can be specialized, or to write screens that are standalone (other than fetching the db connection and such.)
The origin story was that I was working on a framework for making ML training sets called "Themis" that was using microservices, React, Docker and such. The real requirement was that we were (1) always adding new tasks, and (2) having to create simple but always optimized "screens" for those tasks because if you are making 20,000 judgements it is bad enough to click 20,000 times, if you have to click 4x for each one and it adds up to 80,000 you will probably give up. As it was written you had to write a bunch of API endpoints as part of a JAXB application and React components that were part of a monolithic React app and wait 20 minutes for typescript and Docker and javac to do their things and if you are lucky it boots up otherwise you have to start over.
I wrote up a criticism of Themis and designed "Nemesis" that was designed for rapid development of new tasks and it was a path not taken at the old job, but Nemesis and I have been chewing through millions of instances of tasks ever since.
I also recoiled at the complexity of React, Docker, etc. and went a different path: I basically moved all the code to the server and added a way to "project" the UI to the browser. From the program's perspective, you think you're just showing GUI controls on a local screen. There is no client/server split. Under the covers, the platform talks to some JavaScript on the browser to render the controls.
This works well for me since I grew up programming on Windows PCs, where you have full control over the machine. Check it out if you're interested: https://gridwhale.com.
I think pushing code to the server via HTMX and treating the browser like a dumb terminal has the same kind of advantage: you only have to worry about one system.
Fundamentally, IMHO, the client/server split is where all the complexity happens. If you're writing two programs, on the client and one on the server, you're basically creating a distributed system, which we know is very hard.
dont matter if complex or simple, if result not add value. focus on add more value than detract, worry complexity after
Premature abstraction is a thing. Doesn't help that every CS course kinda tells you to do this. Give a new grad a MySQL database and the first thing they might try to do is abstract away MySQL.
Oh boy, this is so true. In all my years of software engineering this is one of those ideas that has proved consistently true in every single situation. Some problems are inherently complex, yes, but even then you'd be much, much better off spending time to think things through to arrive at the simplest way to solve it. Again and again my most effective work has been after I questioned my prior approaches and radically simplified things. You might lose some potential flexibility, but in most case you don't even need all that you think you need.
Some examples:
- Now that reasonably good (and agentic) LLMs are a thing, I started avoiding overly complex TypeScript types that are brittle and hard to debug, in favor of writing spec-like code and asking the LLM to statically generate other code based on it.
- The ESLint dependency in my projects kept breaking after version updates, many rules were not sophisticated enough to avoid false positives, and keeping it working properly with TypeScript and VSCode was getting complicated. I switched to Biome.js, and it was simpler and just as effective. However, I'm recently having bugs with it (not sure if Biome itself or the VSCode extension is to blame). But whatever, I realized that linting is a nice-to-have, not something I should be spending inordinate amount of times babying. So I removed it from the build tool-chain, and neither do I even need have it enabled all the time in VSCode. I run Biome every now and then to check the code style and formatting , and that's it, simple.
- Working on custom data migration tooling for my projects, I realized forward migrations are necessary to implement, but backwards migrations are not worth the time and complexity to implement. In case a database with data needs to be rolled back, just restore the backup. If there was no data, or it is not a production database, just run the versioned initialization script(s) to start from a clean state. Simple.
And I don’t see how number 3 is simpler. In my maths head I can easily create bijective spaces. Emulating backward migration through others means might be harder (depending on details of course thats not a general rule)
For the code generation, note that some types are almost impossible to express properly, but code can be generated using simpler types that capture all the same constraints that you wanted. And, of course I only use this approach for cases where it is not that complicated to generate the code, and so I can be sure that each time I need to (re)generate it, it will be done correctly (ie., the abstraction is not leaky). Also, I don't use this approach for generating large amounts of code, which would hide the inherent structure of the code when reading it.
For the eslint example, I simply made do without depending on linting as a hard dependency that is always active. That is one of my points: sometimes simply some "niceties" would simplify thing a lot. As another example in this vein, I avoid too much complex configuration and modding of my dev environment; that allows me to focus on what matters.
In the migration example, the complexity with backward migration is that you then need to write a reverse migration script for every forward migration script. Keeping this up and managing and applying them properly can become complex. If you have a better way of doing it I'd like to hear it.
The article says that the best way to manage complexity is to find good cut-points to contain complexity. Another tool is an excellent cut-point, probably the best one there is. (Think about how much complexity a compiler manages for you without you ever having to worry about it.)
P.S: For those wondering what this refers to, here is his talk: https://youtu.be/SxdOUGdseq4?t=1896
sad but true: learn "yes" then learn blame other grugs when fail, ideal career advice
This doesn't mean OP is bad advice, just make a conscious decision about what to do with complexity and understand the implications.
grug whisper: “problem come later.”
grug see lone dev make clever code.
grug light torch for future archaeologist.
This is why I treat "complexity bad" as a mantra to keep me in the right mindset when programming. Complexity bad. Even necessary complexity. We may have to deal with it, but, like fire, it's still dangerous.
How do I recognise it?
When do I tell whether something is complex or I am just not very familiar with it?
but complexity bad
This part:
> back end better more boring because all bad ideas have tried at this point maybe (still retry some!)
I entered a Spring Boot codebase recently, and it was anything but boring or "simple" -- everything is wrapped by convention in classes/abstract classes/extending layers deep of interfaces, static classes for single methods. Classic OO design that I thankfully moved away from after college.
I think the author makes good points, but I don't think the author is any different than your average developer who accuses the thing they are not familiar with to be "complex".
The Grug Brained Developer (2022) - https://news.ycombinator.com/item?id=38076886 - Oct 2023 (192 comments)
The Grug Brained Developer - https://news.ycombinator.com/item?id=31840331 - June 2022 (374 comments)
Fun fact about Google: logging is like 95% of the job, easily... From tracking everything every service is doing all the time to wrangling the incoming raw crawl data, it's all going through some kind of logging infrastructure.
I was there when they actually ran themselves out of integers; one of their core pieces of logging infrastructure used a protocol buffer to track datatypes of logged information. Since each field in a protocol buffer message is tagged with an integer key, they hit the problem when their top-level message bumped up against the (if memory serves) int16 implementation limit on maximum tag ID and had to scramble to fix it.
Complexity not bad. Complexity just mean "thing have many consideration". Some thing always have many consideration. Not bad if useful and necessary.
Complexity still difficult and raise problem. So try avoid complexity when unnecessary and no add value. But shun complexity bad when it detract value or raise new problem.
Little grug benefit from not over-simplify. Not everything just good or bad, most thing both, it depend.
simpler to live naked in wood, use big rock hunt food. but grug really like pizza. use computer order pizza, leave tip on app. grug no have to social interact, make grug happy.
manage complexity well and it make good result, worth occasional pain. soon complexity become normal.
Where the type system isn't that expressive, you still have to fall back to testing to fill in the places where the type system isn't sufficient, and your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot by virtue of execution constraints anyway.
I'm not sure I agree with your prior that "your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot." Invalid input is at the very edge of the program where it goes through a parser. If I parse a string value into a type with no room for null, that eliminates a whole class of errors throughout the program. I do need to test that my parser does what I assume it will, sure. But once I have a type that cannot represent illegal data, I can focus my testing away from playing defense on every function.
You're asking about a circumstance that's just very very different from the one I'm in.
You seem to misunderstand. The question was centred around the fact that unexpected null cases end up being tested by virtue of you covering normal test cases due to the constraints on execution. Explicitly testing for null is something else. Something else I suggest unnecessary — at least where null is not actually part of the contract, which for the purposes of our discussion is the case. Again, we're specifically talking about the testing that is necessary to the extent of covering what is missing in the type system when you don't have a formal proof language in hand.
> You're asking about a circumstance that's just very very different from the one I'm in.
But one you've clearly had trouble with in the past given your claims that you weren't able to deal with it. Otherwise, how would you know?
Perhaps I can phrase my request in another way: If null isn't expected in your codebase, and your testing over where the type system is lacking has covered your bases to know that the behaviour is as documented, where are the errant nulls going to magically appear from?
Same thing on the front end in JS: change the shape of some record that has more places using it than I could remember. Better tests would have caught these. A compiler would be even better.
FWIW I’ve written a lot of tests of code written in all of the languages I like. You absolutely need tests, you just don’t need them to be as paranoid when writing them.
Right, but the condition here is that the languages that are expressive enough to negate the need for testing are, shall we say, unusable. In the real world people are going to be using, at best, languages with gimped type systems that still require testing to fill in the gaps.
Given that, we're trying to understand your rejection of the premise that the tests you will write to fill in those gaps will also catch things like null exceptions in the due course of execution. It remains unclear where you think these errant nulls are magically coming from.
I'm not convinced "Didn’t help that we were testing by hand.", "Better tests would have caught these." is that rejection. Those assertions, while no doubt applicable to your circumstances, is kind of like saying that static type systems don't help either because you can write something like this:
data MaybeString = Null | Str String
len :: MaybeString -> Int
len (Str s) = length s
main :: IO ()
main = do
print (len Null) -- exception thrown
To head off (hah) discussion of taking the head of [], we used a prelude where head returned a Maybe. As far as I know, there were no incomplete functions in the prelude. https://hackage.haskell.org/package/nri-prelude
"Better flag choices would have caught it" is a poor take given what came before. Of course that's true, but the same as your "better tests would have caught it". However, that really has nothing to do with our discussion.
Again, the premise here is that you are writing tests to close the gaps where the type system is insufficient to describe the full program. Of course you are as the astute observation of "My hair is too shiny and my muscles are too large to have to deal with [error prone things] everywhere. Can't do it." is true and valid. Null checks alone, or even all type checks alone, are not sufficient to satisfy all cases of [error prone things]. That is at least outside of formal proof languages, but we already established that we aren't talking about those.
So... Given the tests you are writing to fill in those gaps (again, not tests specifically looking for null pointer cases, but the normal tests you are writing), how would null pointer cases slip through, even if the compiler didn't notice? What is the concrete example that demonstrates how that is possible?
Because frankly I have no idea how it could be possible and I am starting to think your rejection was just made up on the spot and thrown out there without you giving any thought, or perhaps reiterating some made up nonsense you read elsewhere without considering if it were valid? It is seemingly telling that every attempt to dismiss that idea has devolved into bizarre statements along the lines of "well, if you don't write tests then null pointer exceptions might make it into production" even though it is clear that's not what we are talking about.
I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I insist on writing new C# code with null-checking enabled which mostly solves the problem, but there's still plenty of code in the wild crashing on null bugs all the time.
Of those who are concerned about type theory? 99%. With a delusional 1% thinking that a gimped type system (read: insufficient for formal proofs) is some kind of magic that negates the need to write tests, somehow not noticing that many of the lessons on how to write good tests come from those language ecosystems (e.g. Haskell).
> I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions.
I don't think I've ever seen a null exception (or closest language analog) occur in production, and I spent a lot of years involved in projects that used dynamically typed languages even. I'd still love for someone to show actual code and associated tests to see how they ended up in that situation. The other commenter, despite being adamant, has become avoidant when it comes down to it.
The "how" is almost always lack of discipline (or as I sometimes couch it, "imagination") but usually shit like https://github.com/microsoft/SynapseML/issues/405#:~:text=cl...
I've had to learn in my career not to open other people's projects in a real IDE because of all the screaming it does about "value can be null"
There is admittedly a lot in the update and I may have simply missed it, but I don't see any modifications, even additions, to tests to denote recognition of the problem in the earlier version. Which, while not knowing much about the project, makes me think that there really isn't any meaningful testing going on. That maybe be interesting for what it is, I suppose, but not really in the vein of the discussion here about where one is using type systems and testing to overcome their personal limitations.
Yeah, when you have extra tools like that it can certainly help. The thing is that you can ignore any warning! I like it to be a compiler error because then there's no way to put that on the tech-debt credit card and still pass CI. If you are able to put those warnings into your CI so a PR with them cannot merge, then that's like 99% of what I like in my code: make the computer think of the low-hanging-fruit things that can go wrong.
With all of that said, solving for null does not get you everything a tagged union type does, but that's a different story.
More careful use of the type system might have caught the null pointer case specifically, but the compiler still wouldn't have caught that they were doing the wrong thing beyond the null pointer error. In other words, the failure would have still occurred due to the next problem down the line from a problem that is impossible to catch with a partial type system.
While a developer full of hubris who thinks they can do no wrong may let that slide, our discussion is specifically about a developer who fully understands his personal limitations. He recognizes that he needs the machine to hold his hand. While that includes leveraging types, he understands that the type system in any language he will use in the real world isn't expressive enough to cover all of the conditions necessary. Thus he will also write tests to fill in the gaps.
Now, the premise proposed was that once you write the tests to cover that behaviour in order to fill in the gaps where the type system isn't expressive enough, you naturally also ensure that you don't end up with things like null pointer cases by way of the constraints of test execution. The parent rejected that notion, but it remains unclear why, and we don't yet have a concrete example showing how errant nulls "magically" appear under those conditions.
"I don't need testing" isn't the same thing, and has nothing to do with the discussion taking place here.
"Bad"
lol
https://craftinginterpreters.com/representing-code.html#the-...
I remember reading through it and not understanding why it had to be this complicated and then just used a tagged union instead.
Maybe I'm too stupid for OO. But I think that's kind of the point of the grug article as well. Why burden ourselves with indirection and complexity when there's a more straight forward way?
https://prog2.de/book/sec-java-expr-problem.html - Not the writeup I was looking for but seems to cover it well.
> Why burden ourselves with indirection and complexity when there's a more straight forward way?
Because each way has its own tradeoffs that make it more or less difficult to use in particular circumstances.
https://homepages.inf.ed.ac.uk/wadler/papers/expression/expr... - Wadler's description of the expression problem.
However, this is just not something that I typically perceive as a problem. For example in the book that I mentioned above, I didn't feel the need to use it at all. I just added the fields or the functions that were required.
In the first link you provided, the OCaml code seems to use unions as well (I don't know the language). I assume OCaml checks for exhaustive matching, so it seems extremely straight forward to extend this code.
On the other hand I have absolutely no issues with a big switch case in a more simple language. I just had a look at the code I wrote quite a while ago and it looks fine.
https://grugbrain.dev/#grug-on-parsing
but the visitor pattern is nearly always a bad idea IMO: you should just encode the operation in the tree if you control it or create a recursive function that manually dispatches on the argument type if you don't
The additional complexity doesn't add significant value IMO. I admit that's a subjective claim.
In languages influenced by ML (like contemporary Java!) it is common in compiler work in that you might have an AST or similar kind of structure and you end up writing a lot of functions that use pattern matching like
switch(node) {
type1(a,b) -> whatever(a,b)
type2(c) -> process(c)
}
Java doesn't support this though I thought?
https://blog.scottlogic.com/2025/01/20/algebraic-data-types-...
Stands to reason. Rust "enums" are tagged unions (a.k.a. sum types, discriminated unions).
In implementation, the tag, unless otherwise specified, is produced by an enum, which I guess is why it got that somewhat confusing keyword.
Given the syntax tree example from Wikipedia, I think I'd call it AstWalker, AstItem::dispatch(AstWalker) and AstWalker::process(AstItem) instead of Visitor, AstItem::accept(AstVisitor) and AstVisitor::visit(AstItem).
"The walker walks the AST, each items sends it to the next ones, and the walker processes them". That means something. "The visitor visits the AST items, which accept it" means basically nothing. It's more general, but also contains very little useful information. So the visitor might need different names in different situations. Fine. Just add a comment "visitor pattern" for recognizability.
I remember a situation where I needed to walk two object trees for a data comparison and import operation. I created an AbstractImporter that walked the two trees in lockstep in a guaranteed order and invoked virtual methods for each difference. It had a non-virtual doImport() for the ordered data walk, and doImport() called virtual methods like importUserAccount(), importUserAccountGrouMemberships() etc. There were two subclasses of AbstractImporter: ImportAnalyzer collected differences to display them, then there was a selection step implemented by a kind of list model + a bit of controller logic, then an ImportWorker to make the selected changes. All rather specific terminology and not exactly the visitor pattern.
more important app is like how programmer intended. web only for docs like unix files anyway no good to think too much.
And it’s inspired many more grugs like grug ceo https://www.sam-rodriques.com/post/the-grugbrained-ceo
many, many shiney rock lost to agile shaman!
Shamans suck for the same reason architects suck -- take "getting stuff done" out of someone's responsibilities, and they lose the plot pretty quickly.
https://www.lulu.com/shop/carson-gross/the-grug-brained-deve...
it's the same content + an index, so not worth buying unless you want the hard copy, but maybe a good convo-starter for the ol'dev team
Grug brained dev I am I guess.
Grug user find program that does that and more.
Grug user confused by menu.
Grug user wish tool only did keyword research.
grug make other work grugs read this after yellow circle arrive next
grug thank clever grug
I smell a formal grammar behind dumbiffied grug english.
nonetheless, I think that when it says:
> so grug say again and say often: complexity very, very bad
at the end of that section, it shoulud say instead:
> so grug say again and say often: complexity very, very, very bad
this disambiguates 3 instances of same concept/idea AND, even better, showcases 3 values of increasing strength like for warning, error, critical use. most compact.
end of groog
But it should be fairly quick, expect an updated version around end of summer or just after.
https://docs.google.com/document/d/1emldq9MovfYshOSkM9rhRUcl...
not as good as the original, i know!
"be organized" is, of course, good advice
Lies and slander, munificent is quite open about his grugness and his grug-brained approach is why his book is so good.
mmmm
get the mug!
And our customers seem very content with the way our product works.
(For the record: yes, it’s a stable, profitable, long-established product.)
It's very effective, simple and expressive to work this way, as long as you keep in mind that some client side rendering is fine.
There are a few bits I don't like about it, like defaulting to swap innerHTML instead of outerHTML, not swapping HTML when the status code isn't 200-299 by default and it has some features that I avoid, like inline JSON on buttons instead of just using forms.
Other than that, it's great. I can also recommend reading the book https://hypermedia.systems/.
(no affiliation, I enjoy the original and wish for it to reach as many people as possible)
Also like a lot of programming advice it isn't actually that useful. Advice like "avoid complexity" sounds like it is good advice, but it isn't good advice. Of course you should avoid complexity. Telling people to do that is about as useful as telling people to "be more confident".
We mostly learn to avoid complexity through trial and error - working on complex and simple systems, seeing the pitfalls, specific techniques to avoid complexity, what specific complexity is bad, etc. Because not all complexity is bad. You want simplicity? Better trade in you Zen 4 and buy a Cortex M0. And I hope you aren't running a modern OS on it.
Ok "avoid unnecessary complexity"? Great how exactly do you know what's unnecessary? Years of experience that's how. Nothing you can distill to a gimmicky essay.
If it's "unnecessary" or "premature", then of course it's bad. I don't need to be told that. What I do need is advice on telling apart the unnecessary and premature from the necessary and timely.
"Oh we don't need to think about performance when deciding the fundamental architecture of our system because everyone knows premature optimization is the root of all evil."
i have started to use repl debugging … just insert a repl statement anywhere in your raku code and you can interact with the code state
That's why it's so easy to rush in and agree with everything. In another article, you might have to read 3 points for microservices, and 4 against. But here? Nope.
Factoring? Just something you do with your gut. Know it when ya see it. Grug isn't going to offer up the why or how, because that's something a reader could disagree with.
> note: grug once think big brained but learn hard way
It's basically a lot of K.I.S.S. advice. Of course you should use the best tool for the job, but complexity often sneaks up on you.
> Factoring? Just something you do with your gut. Know it when ya see it. Grug isn't going to offer up the why or how, because that's something a reader could disagree with.
I would dispute this characterization. Grug says it's difficult, and it is, but gives specific advice, to the extent one can:
- grug try not to factor in early part of project and then, at some point, good cut-points emerge from code base
- good cut point has narrow interface with rest of system: small number of functions or abstractions
- grug try watch patiently as cut points emerge from code and slowly refactor
- working demo especially good trick: force big brain make something to actually work to talk about and code to look at that do thing, will help big brain see reality on ground more quickly
https://x.com/intercoolerjs/status/1935190940903424241?t=WuM...
and clojure.
mmmmm.
Thinking you are too smart leads to all sorts of trouble, like using C++ and being proud of it.
If you think your intelligence is a limited resource however you'll conserve it and not waste it on tools, process and the wrong sort of design.
Even if we decided to use Zig for everything, hiring for less popular languages like Zig, lua, or Rust is significantly harder. There are no developers with 20 years experience in Zig
Any decent engineer must be able to work with other languages and tools. What you're looking for is someone with experience building systems in your area of expertise.
And even then, experience is often a poor substitute for competence.
You may in trivia quiz languages that have more features than anyone can learn in a lifetime
And yeah, its a mess. Even when you think you have a good understanding of at least one part of the language, you'll stumble across some small section of cppreference that makes you question your entire career.
"We're going to need to fit parts of this into very constrained architectures."
"Right, so we need a language that compiles directly to machine code with no runtime interpretation."
"Which one should we use?"
"What about Rust?"
"I know zero Rust developers."
"What about C++?"
"I know twenty C++ developers and am confident we can hire three of them tomorrow."
The calculus at the corporate level really isn't more complicated than that. And the thing about twenty C++ developers is that they're very good at using the tools to stamp the undefined behavior out of the system because they've been doing it their entire careers.
Sure C was a huge advance in portability but C and C++ represent a transitional form between an age where you could cleanly spec a special purpose language like COBOL or FORTRAN but not quite spec a general systems programming language and one in which you could. C++, thus, piles a huge amount of complexity on top of a foundation which is almost but not quite right.
I think it's still little appreciated how revolutionary garbage collection is. You don't have maven or cargo for C because you can't really smack together arbitrary C libraries together unless the libraries have an impoverished API when it comes to memory management. In general if you care about performance you would want to pass a library a buffer from the application in some cases, or you might want to pass the library custom malloc and free functions. If your API is not impoverished the library can never really know if the application is done with the buffer and the application can't know if the library is done. But the garbage collector knows!
It is painful to see Rustifarians pushing bubbles around under the rug when the real message the borrow checker is trying to tell them is that their application has a garbage-collector shaped hole in it. "RC all the things" is an answer to reuse but if you are going to do that why not just "GC all the things?" There's nothing more painful than watching people struggle with async Rust because async is all about going off the stack and onto the heap and once you do that you go from a borrowing model that is simple and correct to one that is fraught and structurally unstable -- but people are so proud of their ability to fight complexity they can't see it.
I do think that there are a lot of circumstances a garbage collector is the right answer where people, for whatever reason, decide they want to manage memory themselves instead.
(Also, the lack of a package manager for C is much more due to historical cruft around build systems than it is difficulty getting C libraries to work together. Most of them can interoperate perfectly well, though there is a lot more faff with memory book-keeping)
I suspect that in a generation or so Rust will probably be where C++ is now: the language business uses because they can quickly find 20 developers who have a career in it.
They never wanted to be in a situation in the embedded architecture where performance was dependent upon GC pauses (even incremental GC pauses). Their higher-level abstraction has tightly constrained lifecycles and is amenable to static analysis of maximum memory consumption in a way Lua is not.
Relateable.