Do Developers Read Compiler Error Messages? [pdf]
static.barik.net
static.barik.net
Hopefully error msg is at the end so you can go to the end of the 100kB worth of logs to find what failed. If it's no at the end though it's kinda funny how you just mindlessly scroll through a garbage and sort of instinctively identify what 'looks like normal garbage' vs 'looks like a bit more suspicious garbage', just by it's shape
Like the App Store helpfully upgraded Xcode over the weekend, and the new version won’t compile the 67th dependency in your project. So you learn to have Xcode dump its version number at the start so it’s in the log. And you tell the junior DevOps engineer about how to prevent automatic upgrades on the Mac mini he never touches. And also how it’d be good to have downloaded a few versions of Xcode and make selecting the one you need a step in the build process… but I digress
Needing to sift through a giant log is annoying, but you can get pretty fast at it for "normal" errors. Needing to debug an issue that is not adequately logged is a nightmare.
Sure there are things such as 'too much data' and 'trash data'. The problem is discerning what you need and what you don't need.
The answer isn't always obvious, but it is possible and 'easy' to filter data, whereas getting missing data can be impossible/real hard.
There must be a better abstraction than here is all the things, have fun.
Looking for a dependency issue shouldn't be remotely comparable to analyzing an airplane's crash logs in difficulty.
Since all logs should be immediately persisted off box, they are protected from tampering.
Of course there’s no reason the default view of logs in the CI/CD system needs to be so verbose.
Yes, this data should be recorded.
Yes, it should be viewable.
Should it be viewed as a continuous 34000 line log file?
Of course not. But this is what many (most?) CI tools do. The correct way is to group output at least by step, make log output from various parts of the system collapsible etc., for example, dumping all environment variables and component versions at every step CAN be helpful to diagnose issue, but usually isn't. So it should be collapsed by default.
Edit: I have the exact same gripe with all of the log4j inspired logging libraries, which in my opinion provide very little value over a print. To little structure, no way to hold back output until you have an error, temporary redirects are difficult and have to modify global state and so on. Uninspired design providing minimal utility.
Which means test tools should conform to some agreed-upon universal output format for their console (and make sure none of the underlying libraries print anything to stdout) or have an out-of-band way to communicate with the CI system in real-time (XUnit/Junit isn’t good enough as those are only typically written at the end of the process).
IIRC, Travis has been doing this since at least 2016 where they collapse most of the setup steps by default - and if you are interested, it only takes one click.
This of course only applies to the UI, the raw log is complete.
This comment isn't serious. I read that. You just tail the file to find the error. But it's very useful to go through it all. It's not garbage to me. I find most logs quite readable.
ENV variables at every step? print them, build has a parser/codegen step? print everything it does, sanity check scripts? I always need to know what they did, setup scripts? That's the most important part, every .o file compiled and linked? show me, with the compile command for a good measure so all the -Ilibs's are there? of course, set -x? why not, need to wget/curl something? output goes to the log, is it in parallel? just append as it goes, build failed? start the teardown but make sure you add everything it does to the log
And as I wrote, I hope it all just terminates at the first error so I can tail the logs... but it's not always that pretty. I don't believe you've read those instead of tailing for an error or doing the fast scroll to find something looking somewhat off
I mean, I'm used to it now, I'll grep, less, tail and I'll find what I need, I'm just baffled that it has so much unnecessary data for every single build. At one company I've got good enough at it that I could press-and-hold pageup from the bottom in less and staring at the streaming bytes Matrix-style looking for a place of interest
So you end up debugging everything, and when it works you're afraid to remove the debugging because the last time you did it took hours, and you ended up leaving it in.
Edit: also, because all the important messages are being sent to a group chat or other appropriate place.
Warning: http timeout to package server 1
Warning: could not find x in package server 2
..(several lines later)..
Warning: could not find dependency x in cache
....
Compile Error: unknown reference to "x"
Because the actual problem by itself isn't fatal -- it doesn't yet know if the package is in server 2, and it might already have a suitable package in cache -- it's not a fatal error. Later, the part of the compiler they needs the dependency doesn't have context to the other failures that happened; all it knows is something is missing.This is a universal problem in almost every application: very rarely can a single log line provide enough information to fix a problem or identify a bug.
In my contrived example, the partial fix is that the package bit should fail. However the final log message needs all the context of previous errors as well or it isn't useful either. It's turtles all the way down.
It's not that this isn't a solvable problem, it's that people building compilers and other tools make assumptions about context, people putting together the build don't spend the time to learn the nuance of every tool they use (rightfully so - who had time for that?), and everyone falls back on the crutch of giant log files. In other words, through the whole stack (language, compiler, tools, build system, and individual application build itself), no one has optimized or built anything based on the idea the CI server or build system should have a single, useful error message.
The problem is the pages and pages of "did that, everything is fine". Since Linux distros stopped optimizing disk usage by omiting the update time of files a couple of decades ago, I haven't seen build tools lose any single step. The failure is never because something that should be done was ignored. Instead, the failure is often because of some non-fatal error that you have no hope of finding because it's mixed with 10k like of "build tool got here" spam.
....(thousand of lines)...
Info: Package X version 1.2.3 successfully added
....(thousand of lines)...
Compile error: Unknown type Z
Actual problem: Z was added in Package X version 1.3.0, but some other dependency or reference that wasn't updated caused the incorrect version to be loaded.Fixing this on your own computer is quite straight forward, you just need to check the dependencies, you don't need the info line. Fixing that kind of problem on the build server is one of the reasons we have all those practices about reproducible compilation. On this specific case, it's a matter of freezing your dependencies.
What developers do is to check where the errors show up, usually just the first error, and use pattern recognition techniques, aka 'experience', to fix the error in its context. Only if the fast track error recovery process fails, repeatedly, do developers spend minutes to carefully read the compiler error messages and deeply make sense of it.
This matches the article thesis: the 'better' compiler messages are, the more likely is that pattern recognition techniques can use the error message as part of the fast track error fixing process.
This depends fully on the ~language~ compiler.
Some compilers create extremely long verbose but useless messages, other create use-full messages most times only as long as necessary with nice formatting making it easy to skip over the parts you don't care about.
Like e.g. I always read error messages when programming rust, sure I fix a error at a time so I don't read all error messages at once, but I do go through them step by step.
However, we do have a policy of expecting that all compiler errors (except those in in-tree 3rd party libs) should be fixed. We take a very negative view of any commits that create new compiler warnings. Essentially, we prefer to have a development culture that treats warnings as errors rather than have the compiler enforce warnings as errors.
When building Ardour, we this as a minimum:
'-Wall', '-Wpointer-arith', '-Wcast-qual', '-Wcast-align', '-Wno-unused-parameter'
but prefer to use this:
'-Wall', '-Wpointer-arith', '-Wcast-qual', '-Wcast-align', '-Wno-unused-parameter', '-Wcast-align', '-Wextra', '-Wwrite-strings', '-Wunsafe-loop-optimizations', '-Wlogical-op', '-Wnon-virtual-dtor', '-Woverloaded-virtual', '-fstrict-overflow'
For example, if there's an error in a macro that was meant to emit a trait impl, the first error will tell you about the error in the macro, and then the fifty errors after that will be fifty different and increasingly confusing ways of telling you that the type doesn't impl the trait.
If you don't start at the top and fix one error at a time you'll just end up chasing and being confused by red herrings.
In effect, good writing can have negative returns.
It doesn't happen that often, but sometimes I get frustrated searching for an error that shares its wording with way too many unrelated issues.
[I am thinking more of runtime errors, but still]
So I updated my test to grab their OS user name and called them.
"I thought you were testing something so I didn't call"
While they may read the message, it didn't mean the purpose was clear.
Error messages can be wrong or misleading sometimes, but how do you not at least think about what they said and if that could actually be what's wrong?
I have made the same experience running tutorials and simple example code, but as soon as I took on anything more intricate, I got thrown right back into "GCC compiling a C++ template" territory.
Basically, the compiler told me "constraint Foo<<<<trait::?>'<mut&>><other> does not meet bounds Bar<trait::dist>?<other>" and that was it.
Feel free to file a ticket next time you come across a case like these! For complex or uncommon cases we rely on people's reports to handle them better.
I have to admit I was pretty discouraged and rage-deleted the code that made me give up, so I don't have it at hand any more.
I also am really early in the process in learning about Rust, and dove in very deep by trying to use warp to write a web application.
I'm usually the type to dive in by copy-pasting examples and modifying them to my needs, and I just hit a brick wall of compile errors here that discouraged me from continuing.
When I gather the patience to continue, I will keep in mind what you said.
I've seen multiple colleagues new to Rust ask me for help with their code that has `&'static mut`, and when I tell them that's never going to work and ask them how they got there, it's always because the compiler told them to do it.
Like [1] says, you eventually get the experience to ignore the compiler error message (or at least the "suggestions"), and instead understand the real problem and apply the real fix yourself.
Similarly, when you have a case of value of type A being assigned to a binding of type B, there's a 50% chance that the compiler gets the "expected type" and "actual type" the other way around from what you want. This isn't really a fixable problem, because only you the human know which one you expected and which one you didn't. But it's yet another case where you eventually gain the experience of ignoring the "expected type" "actual type" parts of the compiler error because they don't matter, and just looking at the types.
Most of the time the compiler complains about an unmet 'static lifetime, what it's actually talking about is about wanting an owned type. Sadly, I haven't gotten around to making those diagnostics more accurate (from the user's points of view).
> I've seen multiple colleagues new to Rust ask me for help with their code that has `&'static mut`, and when I tell them that's never going to work and ask them how they got there, it's always because the compiler told them to do it.
> Like [1] says, you eventually get the experience to ignore the compiler error message (or at least the "suggestions"), and instead understand the real problem and apply the real fix yourself.
I would appreciate bug reports at https://github.com/rust-lang/rust/issues/ when encountering these kind of situations. If there's something worse than missing suggestions is inaccurate or misleading suggestions.
Please, compiler engineers, ALLOW the developer to output stack traces in a structured format. It doesn't matter if it's XML, JSON, YAML, whatever. Please allow the developer to configure the compiler so that when a compiled program yields an error, the error isn't some series of random tabbed lines, but something structured that can then be easily processed.
Any format you have a preference for?
That doesn't mean it never happens (most of us inherited a shitty codebase lacking things like code standards and autoformatting at some point), but I always find it strange that people keep talking about this thing I've never experienced in years writing JS professionally
Actually, I remember Jerry Pournelle asking exactly the same question about Pascal in Byte magazine in the 1980s.
Template errors have gotten better but can get very verbose. I rarely read the whole thing. You learn how to skim and find where I can click to get to the line which has the issue.
One thing you can do (that Rust does) is keep a stack of opened delimiters and their positions, and pop them as you find the closing one. If you encounter a mismatch or any other parse error, you look through the open delimiters to match for indentation level or valid alternatives. The more "flag posts" or redundancy the grammar has, the more likely you are to be able to recover from malformed code. If you are in a language with a sparse grammar, recovery is less feasible.
The classic example is forgetting to end a typedef struct foo with a semicolon, at the end of an #include file.
The compiler will happily combine that with whatever follows the #include# statement in the file including it, and report an error with a line number in the file doing the #include.
One of the top questions on stack overflow was “how do I fix this?”
How about… upgrade?
The other thing “how do i fix this?” might mean is that the person asking doesn’t know how to update their foobar. Our system state the exact command the developer has to run to get their foobar to the correct version. No guesswork needed. Saves a ton of time.
1) there are too many irrelevant, knock-down errors caused by a single source
2) the errors are not clear in their description and attribution
3) the quality of errors varies a lot, meaning that ahead of time you don't know whether reading the error will be helpful or not, which leads to
4) you've been trained by experience to ignore errors in all tools
These are a problem even if you do provide good diagnostics because people will not expect them to be of any use.
Do you speak from anecdotal experience? Do you have any hard data (that you can share)?
> Do you speak from anecdotal experience?
I think we can agree that people don't read things properly.
I'd suggest a hash or a heuristic hash or a 3rd party algorithm run on the error linked to a crowed sourced location the user can chose.
So the 3rd party can say ignore this it's stupid. Something the original error message writer could never tell their boss. Or when the error was written their info was wrong, now it's this. Or they can say use root permissions on the code. Or undo this security patch. And just like Stack, people could explain why and why not. Teach around the error.
Their conclusion "that the difficulty of reading error messages is comparable to reading source code" can't be solved at the error message writers level. Politically or even for the ultra high users who need the hard core message.
Even by allowing the user to rule out the easy solutions it might give, they can turn their brain back to source code mode to tackle it.
I have been programming professionally for over thirty years and most mentors used to shrug at warnings if the code worked.
It inspired me to go back and clean up my C# and Java stuff and as a learning process I highly recommend it. It seems especially valuable in fast changing environments, generally you will upgrade a tool and get new warnings. I used to think they were just a pain but now I treat them like a way of gaining new insights.
Rather than
"Do Developers Read Compiler Error Messages? [pdf]"
a more descriptive title may be more appropriate E.G.
"Study of JAVA errors and resolution effectiveness [pdf]"
Then again, dang has also broken this rule and had a justification for it, so who really knows for sure.
Double-click on the error-message, look at the code, fix the code, recompile.
Can't understand what's wrong with the code? read the error message for clues. Maybe SDK or language docs, maybe google to figure it out.
https://github.com/zloirock/core-js/issues/708
I'll read the compiler errors when I don't have to scroll past solicitations for employment.
https://github.com/zloirock/core-js/issues/936#issuecomment-...