Hacker News Reader

Comment by mattnewton | Hacker News Reader

The strings used for debugging at Google were supposed to include “DO NOT SUBMIT” which, a bot to check for this was added after a debug strings made it to production (It was before my time but I believe it was after the legendary WD incident- “western digital” was remapped to “woodly doodly” in test code that accidentally shipped for those curious)

View on HN

> "DO NOT SUBMIT"

Kinda verbose, ain't it? Just speaking from my own personal experience, usually when I resort to print-debugging I'm already pretty punchy and more likely to use a quick "ASDFASD" or similar.

Yes, but you can check for "DO NOT SUBMIT" with automation.

You can't automate checking for random strings, right?

Perhaps an abbreviation would be the best of both worlds, and debug strings should be prefixed with "DNS"

You won't need to submit that particular string working at Google, right?

We could also have an acronym for the more severe "DO NOT SUBMIT - SECURITY EXPECTATIONS COMPROMISE".

DNS-SEC...

Confused sysadmins wondering if this is SOX code...

The automation which can check for do not submit itself is hard to submit. Or at least updates to it are hard to submit.

forbidden_string = "DO NOT " + "SUBMIT"

Seems easy enough?

Just disable that particular linting rule (or however it's implemented) in that repo.

Well in the LLM era, you could. I’m not sure you should :)

> You can't automate checking for random strings, right?

No, but you can make the string configurable.

Related in terms of being easy to search for, I use the abbreviation "TK" as a placeholder for text or incomplete code. Took this from the publishing industry (my partner worked in magazines) -- it's a combination that does not appear in regular English and so is easy to both see and to use search tools for.

I hope you never built a rooTKit.

TK is a semi-well known widget toolkit/GUI toolkit which came from TCL and today it's the default toolkit for Python. Nothing fancy, just basic menues/buttons and widgets, but they get the job done in a hurry.

> > "DO NOT SUBMIT"

> Kinda verbose, ain't it?

I always used the word "doberman" for this purpose. I've never written code for a project that legitimately included the name of a dog variety. A simple grep for "doberman" in the production release CI pipeline catches it. If one ever did slip thru I figured it wouldn't be too offensive to anybody.

I've used NOCOMMIT. Less verbose, equally clear.

I'd be scared to spell it with one M or two Ts.

Not if a layperson comes across it and you miss an omission.

depends. If you are paranoid and afraid of dogs, ...

I use xyzzy

- Nothing happens

- Easy to find string in code, output, wherever

And in GNU info manuals. https://www.gnu.org/software/emacs/manual/html_node/elisp/Se... Just so happened to be reading that in the next tab.

XXX is already highlighted by most editors by default (or at least mine) and seems suitable. Any comment to be committed to a shared branch should probably contain more specifics and not contain that, if you wanted to institute a policy.

I use the strings "XXX" and "999" for this (the latter because you sometimes need a dummy value in a numeric context), and have a global git hook which stops me committing a changeset which includes them.

I occasionally need to override the hook, for example when using mktemp -t, or when some floating-point data actually contains a run of 9s. But mostly, it is quite specific at catching stuff that shouldn't be checked in.

It's about writing code that your peers can read. "DO NOT SUBMIT" is clear as day. "ASDFASD" probably does not mean "this is a debugging string" to most people.

Not really? it’s close to the minimum string you’d be okay with never actually wanting to commit. Never had a problem with it in logs, but it could be in a comment next to the log if you did. Easy enough to add a shortcut for if you really have a problem typing it out, but at my typing speed my brain has always been the bottleneck, not the number of characters.

I worked on a codebase that had a special logging function with a name like NoPushLog that was just a direct wrapper of the base log function. A githook checked that this string was not in pushed code.

This solves many of the concerns raised in this thread about readability, automation, avoiding typos in the magic string.

The tricky thing you have to solve is how to push the code that defines the custom logging function, but there are solutions.

It doesn’t allow you to put that in comments, config files or other languages though like a plain text string.

Why wouldn’t it? Sounds like the git hook is probably just grep…

Then why have a function?

Because it distinguishes the normal logging function from the ones that are for local debugging.

Because of you made a typo, you're more likely to notice than in a comment or string, because it will fail, even at build time if you're using a compiled language

Right, but the whole point of DO NOT SUBMIT is that it doesn’t need to be in the compilation step. It can be in data files, comments, etc. You can change your syntax highlighting theme to spot the typos x (I added them with a keyboard shortcut anyways)

You need to compile the code with all the debugging logging in it so it can run and print out all the “got here 3”s and so forth.

I use "NOMERGE" plus the habit of grepping for it before I merge a branch.

"GOT_HERE_1"

They pay you enough that you'll respect the need to be professional and write "DO NOT SUBMIT" over "ASDFASD" or similar.

Pro tip: never add a "funny comment" especially an error message that's "never supposed to happen" unless you're happy with that message showing up in a board meeting, in front of your boss or worse (and it being traced back to you)

Listen. If it’s never supposed to happen then you definitely want a customer to tell you if it happens. Best way to make sure that happens is to insult their mum. It’s just good engineering discipline.

In my younger and more foolish years I worked for a company called Redgate and, whilst there, I had to do some licensing jiggery pokery that stopped people using our tools if the licensing code had been tampered with or removed.

I didn't want to make it too easy for people trying to crack our tools so, if such a problem were detected, the error message spat out to the console bore no relation to what had gone wrong.

By "bore no relation" I mean the error message would be one of a number of lines of dialogue from a scene in a House episode where House is trying to figure out who Wilson's girlfriend is... which led to this:

https://gist.github.com/gregoryyoung/871736

Yes, Greg Young (the event sourcing guy) ran into an error that said "Because I wanna ask you about your girlfriend. I must know who she is, or you would've told me her name," because he'd moved Smartassembly minus the licensing DLLs. Of course, he didn't realise what he'd done "wrong" so he tweeted it to his considerable following and it went viral.

Because I'm exactly the sort of walking cliche you'd expect I was blissfully unaware of the unfolding drama, being away on sabbatical snowboarding at the time. I hadn't taken my laptop with me, I'd disabled data roaming on my phone to avoid a whopper of a bill after a couple of months away, and had only a gen 1 iPad to access email and internet. So of course I came back in the evening to discover that it had all kicked off, with this ludicrous chain of emails where people were trying to figure out what was going on, until somebody had the sense to go and look in our source control system (which at the time was still svn for most things), and figure out that it was my fault.

People saw the funny side and it wasn't that big a deal but the reason I bring this up is that the last line of dialogue in that scene is, "Your mama," delivered by Wilson to House, and it was also the error message for the final failure scenario I was looking for. I'm glad that one didn't get printed.

Back in the nineties, 3D Studio used to be pirated by every kid in their dorm and more. The software had license checks all over the place apparently, but crackers tried to remove all of them.

However, if they missed one, it would move a random vertex by some random amount which is obviously very visible when rendering a 3D scene.

When people complained about this on forums (usenet mostly), they outed themselves for using pirated software.

You should be 100% sure that such code never ever gets executed by legitimate user. As seen above, it doesn't have to be a bug in licensing code, it could be something external.

Yup. Long ago, join an online Command and Conquer game. Boom, everything blows up. Piracy watchdog that reacted to my repairing Windows and not realizing I needed to reinstall the game after that.

System.exit(0); in Java is probably the most evil thing invented, when the license check fails (preferably launch a thread and do it without stacktrace, after a random time).

To this day I still think Google’s motto only referred to “Do no System.exit(0)”.

Sadly I mostly agree with this. File it with "the first time people get the password right tell them they failed, making them enter it a second time" and other janky but highly effective stuff.

I believe you can get someone’s entire list of passwords this way.

I write alot of #![no_std] code in Rust and forgetting debug messages is luckily impossible because compiler won’t compile in the end if you forget something.

The nice thing about DO NOT SUBMIT is that it worked in comments, text protos and other configuration files, and every language.

I did this once with the deliberate intention that be traceable. Holding down a few keys while clicking on a specific small set of pixels in an application we were developing for a customer would produce a specific message that was attributable to me personally. I did this sort of thing in most things I worked on back then, partially as a "calling card" and partially as a means of being able to prove my involvement in a given product if the need ever arose. This saved our bacon when a sales rep sent an "evaluation" copy of the nearly completed product to the customer, without authorization, that consisted of our entire source tree. (This was the 1990s and we were running MS-DOS, no file security)

Suddenly the customer tells us they have "repurposed" an "internally developed program" and they cancel our contract. The sales rep was upset that he would not be getting a commission on the sale and mentioned that he had "even sent them a demo", which led to the discovery of what had happened. My boss and I went to the customer's site to "finalize the paperwork", and when we were shown the application (which had extremely minor UI changes) I checked if the hidden click spot still worked, and it did! The customer was horrified. My boss was outraged.

We ended up getting paid the full contract amount plus a large bonus in exchange for agreeing not to sue. The sales rep decided to leave voluntarily rather than be fired.

> Holding down a few keys while clicking on a specific small set of pixels

Did you draw inspiration for it from Bullock's The Net (1995) by any chance?

I don't think so; I don't remember ever seeing it

I like the way you phrased this. Maybe someday academics will refer to Bullock's net the same way we talk about Occam's razor, or Maxwell's demon, today.

There's a fine tradition of this, especially at early Microsoft. It's documented here: https://www.pagetable.com/?p=43 , and includes hidden messages that Gates allegedly added himself to prove that code the company licensed could still be traced back to him. (I hit the Ric Weiland one mentioned on my first computer, the Ohio Superboard II - https://www.pagetable.com/?p=43#:~:text=the%201977/1978-,Ohi... ).

This was all in the era of Gates' Open Letter to Hobbyists[1], when "Micro-Soft" was licensing their code to companies, perhaps because direct sales to hobbyists didn't work out.

[1] https://en.wikipedia.org/wiki/An_Open_Letter_to_Hobbyists

Yeah, back in the DOS days I would normally embed something that would respond to specific nonsense.

Was the debug string that made it to production embarrassing?

I remember a swearword appearing in a very big national company login form once, and I'd have felt sorry for the dev if it wasn't so hilarious.

Yea, I have seen a very inappropriate debug statement being left in the led to the dev being promptly fired.

I was leading a startup development team in the late 90's and one of our developers told me that some very inappropriate test data had been sent to IBM to be included in a product demo and burned onto tens of thousands of discs that they would distribute.

Eventually it turned out that it was the single word "sheep" - which taken out of the relevant alt.* context was pretty harmless.

Talk about a moment when I thought my career had crashed and burned...

> Eventually it turned out that it was the single word "sheep" - which taken out of the relevant alt.* context was pretty harmless.

Please, do contextualize it?

Likewise; they tried to pin it on me, and I had to go to the Git commit history to show it was another employee. My administrative leave was promptly revoked.

“Western Digital” was replaced with “Woodly Doodly” on the srp. Not really thag embarassing but I am sure WD didn’t appreciate it

When I worked as a technical writer earlier in my career, we had a debug/prod check: If you used a series of words it would fail you in prod (refused to build). These were usually codenames of projects and such.

One afternoon, I added the phrase "[Ww]ombo\s[Cc]ombo" (it supported regular expressions) and an expansion set (&prodbomb; evaluated to &wca; &wcb; to create "womb combo") but only in prod. In debug builds, it would produce "Wombat Combat"in bright red text.

This was a useful way to mark pages as "Don't build this in prod yet" - a habit some teams had of just yolo'ing their development branches right into production instead of doing a rebase to main. More than one build was saved by it within a few months.

It's amazing how many processes evolve because of incidents like these.

All of the comments seem to talk about automation for detecting sloppy logging.

Is nobody reviewing code before merging it upstream?

Belt and suspenders.

Every line was reviewed but that doesn’t mean it would be caught every time. And if it’s automatable to catch things like this it should be.

Right, but I’m pointing out that everybody is using a belt and nobody suspenders.

> "DO NOT SUBMIT"

That's pretty good advice if you find yourself dealing with a large tech company.

Reply on news.ycombinator.com