My PhD advisor rewrote himself in bash (2010) | Hacker News Reader

My PhD advisor rewrote himself in bash (2010)

matt.might.net

257 pointsvismit2000104 comments

Two resources which helped me improving my writing, when I was writing my thesis were "How to Write Mathematics" by Paul R. Halmos and "Mathematical Writing" by Donald E. Knuth et al. I would always start with Halmos to get into the spirit of perusing clear and precise communication. The "Bad/Better/OK" suggestions especially reminded me of the discussions in the lecture notes from Knuth et al. And at a third step a linter such as the proposed one is probably helpful, if something slips through.

I think these resources are essential for anyone who writes on any subject which at least involves definitions here and there.

I second those recommendations and add one: The Craft of Scientific Writing by M. Alley.

If anybody likes this article and wants to know more about the process of writing effective PhD papers they should watch [1]. In fact, anybody who desires to improve their communication skills should watch it. It is so good that I would have paid to have access to this video!

[1] https://youtu.be/vtIzMaLkCaM

I wish there was a repo of "all the things you need to know when starting a PhD", include the famous phd grind.

For the 10,000 of today: http://linyun.info/phd-grinding.pdf

I find myself disagreeing with many of the examples. E.g. according to the article:

Bad: It is quite difficult to find untainted samples. Better: It is difficult to find untainted samples.

Bad: We used various methods to isolate four samples. Better: We isolated four samples.

Something being quite difficult reads significantly differently than just being difficult. You haven't made the sentence better, you've changed the meaning.

And the fact that you used various methods instead of a single method is information missing from the second sentence.

And this is why I now have to read 30 page design docs that could have been 3 pages and said the same thing.

Please try to understand why people have such strong dislike of floral writing, especially in technical texts. If you read a lot of papers or designs, it makes your life miserable.

When it comes to technical writing the only thing I can really discuss is documentation, and the key thing I'm personally looking for there is structure.

It could be about basically anything, just please, pretty please, for the love of god, make it structured. And I don't mean sections with catchy headings, I mean as structured and reference-like as possible.

I want to minimize the amount of time I spend reading prose and searching around, as well as the chance of missing things. I want to hit CTRL+F and be put where I need to be stat and have that be enough. Structure alone can convey a lot of the idea behind how something works - please trust me to able to utilize it to make basic leaps in logic.

A bad example for this is AWS documentation. It's a mish-mash of prose and structured reference. A good example is the AWS CLI documentation (although if they lead with example usages first, that'd be even better).

Writing good technical text is an art. There is a certain amount of fluff that helps, and it’s almost unnoticeable when it’s there. Without it, it’s too terse. Quite often, my complaint of technical documentation is “it did exactly what the docs said it would do, except in a situation that I didn’t expect it to do that”.

Yes, it's the usual advice of how artists/authors/scientists make something: 1) Make the thing, 2) Try removing each part, 3) If the work fails without that part, put it back.

For example, adverbs are good when readers might have the wrong image without them. E.g., "Alice [quickly] walked." Most of the time, writing is better without words like "very" or "quite."

> Something being quite difficult reads significantly differently than just being difficult. You haven't made the sentence better, you've changed the meaning.

The problem there is that the meaning quite carries can vary significantly depending on the reader or the context where the word is read (so it can read differently depending on how previous sentences have primed the reader, which means the same person might read it differently with that context than if they start at that sentence. Quite differently, in fact!

This is because in spoken form the word changes a lot with tone of voice. Technically "quite difficult" means "slightly difficult" but in many places actually means "damn near impossible".

I'd day that while removing the word isn't wrong, replacing it with a more specific comparison would be better.

And as I understand it, Brits are particularly fond of using "quite" in a sarcastic fashion, so "quite difficult" in England might mean "not difficult at all, you sodding idiot" or something along those lines.

Brit here, this isn't something I'm familiar with. "Quite" usually means "somewhat" as in "I found the test quite hard". In upper class speech it can mean "very" as in "that was quite the challenge" or "agreed" when said on its own as a response to a statement.

Interesting. Maybe it's a regional thing, or a generational thing. Or maybe I'm just flat out mis-remembering. Or maybe some of my British friends told me that, but they were just taking the piss. :-)

It's something I've come across references to more than a few times over the years though.

EDIT:

OK, FWIW, I can't find any solid reference at a quick glance to the form I was thinking of, but Google's "AI Search" GenAI thing does reflect what I was getting at, so I don't think it's completely something I made up. Unless me and the Google AI both hallucinated the same thing.

Here's what Google has to say:

    In British English, when someone says "quite" with a 
    slightly sarcastic tone, it usually means they are 
    implying something is "not at all" or "very much the 
    opposite" of what they are describing, essentially 
    downplaying a positive quality to express mild 
    disapproval or skepticism. 

    Example:

    "Oh, that new restaurant was quite good." (Meaning: it 
    was actually pretty bad)
    "He's quite the brilliant mind." (Meaning: he's not very 
    intelligent at all)

I probably did overstate the degree of emphasis of this though.

Regardless, your point is valid. Adding a valueless word like “quite” does not improve clarity or meaning and can only have a negative impact. Not worth the risk.

> when someone says "quite" with a slightly sarcastic tone

The sarcastic tone is the secret sauce which makes the difference with a lot of words, including qualifiers like "quite". Try applying a sarcastic tone to "definitely" in the Earth is "definitely" flat and you'll see how people react.

Teacher: "There's plenty of languages where two negatives will negate each other and create a positive, but no languages where two positives will make a negative."

Student, sarcastic tone: "Yeah, right."

(I have no idea where I got this from, read it online ages ago)

A member of the British upper crust can correct me if I'm off the mark, but the definition there is of a really existing usage, and then the example doesn't match it at all. Did you make the example up yourself, by any chance?

There is, in ordinary people's language, "yeah, it was quite good", when talking about a movie or something, which could easily mean, it was moderately ok, not amazing in any way. It'll depend entirely on tone, you could say it in a chirpy tone and you'd mean that it was actually pretty good. This is the most common usage, and familiar to our brothers and sisters and non-binary-siblings across the pond, I suppose.

And then there's your mathematics teacher saying, "Oh, this lemma really is quite trivial", meaning it's very, very trivial, or a "quite difficult proof", meaning you've to drag yourself across hot coals for hours before it hits you.

Then there is also the meaning you describe above! E.g., a bunch of aristocrats are having dinner, and the candelabra suddenly breaks loose, flies through the air, and smashes into a thousand pieces with a crash. Luckily, no one is hurt.

Everyone looks around, shocked, there's a few shrieks of course, and then one of them says: "Oh, what a smashing evening!" and the other says, in a bored drawl, "Quite". It's like an additional layer of being removed from and above the mere idea that the original thing could have been worthy of a positive comment (in this case, the dinner).

This is another thing that is captured in tone more than anything though the Brits do have a well deserved reputation for sarcasm. Difficult to convey in print what meaning you want the recipient to get.

It may have been Jimmy Carr who pointed out that:

Americans think that Brits can be quite patronizing ("pæ-trə-naɪ-zɪŋ").

Actually it's patronizing ("peɪ-trə-naɪ-zɪŋ").

> And as I understand it, Brits are particularly fond of using "quite" in a sarcastic fashion,

Brit here. Many of us are fond of using _any_ word/phrase with sarcasm, irony, or both.

> so "quite difficult" in England might mean "not difficult at all, you sodding idiot"

Depending on tone and other context "quite" can mean anything from a little to a huge amount. It can also mean exactly, as in "Well, quite.".

This is why you need to be careful in professional and academic contexts, or anywhere in writing for that matter, and use domain specific terminology as much as possible.

> Technically "quite difficult" means "slightly difficult"

Really? You must come from a different literary tradition. Quite means exactly.

See the first definition in the Cambridge Dictionary at https://dictionary.cambridge.org/dictionary/english/quite

completely:

The two situations are quite different.

The colours almost match but not quite.

I enjoyed her new book though it's not quite as good as her last one.

UK formal Are you quite sure you want to go?

Quite honestly/frankly, the thought of it terrified me.

Great examples, but you should probably remove the "completely" header - as the following examples don't fall under it. I'll delete this comment in 15 minutes ( • ‿ • )

I didn't add completely, that's a quote directly from the Cambridge Dictionary.

Not only is “completely” the definition they're quoting from the dictionary, it is also exactly what is exemplified by the examples, so I'm not sure what you mean by “don't fall under it”.

Ooooh, now I get it- I completely misunderstood that! Indeed, I can substitute every quiet with completely and it's meaning never changes from how it would've been interpreted!

(And I totally forgot to delete the comment too)

I just didn't realize that and only considered how I'd interpret the meaning of completely on is own. And that meaning doesn't translate to every example, hence my previous confusion

Your examples are excellent but I agree with the GP that "quite" can also be used in the sense of "somewhat but not entirely".

Quite difficult = Not impossible.

So, completely difficult. But completely difficult doesn't sound quite right, probably as less syllables are preferred over many unless there's a quite good reason to prefer the latter.

You've omitted the definition on the lower part of the page:

quite

adverb, predeterminer

"a little or a lot but not completely:"

I'm quite tired but I can certainly walk a little further. There was quite a lot of traffic today but yesterday was even busier. It was quite a difficult job. He's quite attractive but not what I'd call gorgeous. It would be quite a nuisance to write to everyone.

The same dictionary also includes a grammar article clarifying that quite [usually] means "a little, moderately, not very", when the adjective or adverb it modifies is gradable (e.g "good" or indeed "difficult") and it being an intensifier in [generally rarer] situations where the adjective or adverb isn't (e.g "it is quite wrong to say that 'quite' invariably means 'exactly')

https://dictionary.cambridge.org/grammar/british-grammar/qui...

My takeaway from all this is that "quite" means literally nothing and belongs on the banned words list.

I agree. It's quite useless.

> Quite means exactly.

Quite.

[Or sometimes “Quite so”.]

Though in informal communication the word's meaning can be quite different depending on context and tone.

> the meaning quite carries can vary significantly

Yes, it can be read as a signal that there is some variability in the amount of difficulty.

Putting more uncertainty in the wording is not necessarily bad, and in a scientific context it can be actually good.

Yeah most of his examples looked terrible to me. It's actually part of why reading papers is so damn difficult even when the paper says something simple. They're obsessed with this stilted formal tone that no one actually likes and leaves out subtle but important context clues.

Strong disagree. If you want to make a point about using different methods, say what you did.

“We isolated four samples using the following methods…”

If you are looking for more of this, check out vale.sh. It lets you add all kinds of style guidelines, including avoiding weasel words.

Here's the GH action we use to run vale on our website at PR time: https://github.com/FusionAuth/fusionauth-site/blob/main/.git...

and our config: https://github.com/FusionAuth/fusionauth-site/tree/main/conf...

We've found it helpful to enforce style but probably aren't using it to the full extent.

Vale.sh looks pretty good, thanks for the reference.

I’m currently editing the third edition of my book and a bunch of articles for my website. This will come in handy.

I got Claude to port those Bash scripts to a web UI so I could paste code directly into it and see what came out: https://tools.simonwillison.net/writing-style

Claude transcript here: https://gist.github.com/simonw/e9902ed1cbda30f90db8d0d22caa0...

I don't think it is working properly. The second(?) pattern match is bad. For example, for

    It is quite difficult to find untainted samples. We used various methods to isolate four samples.

it gives

    Found "quite" in: "is quite difficult to find untainted samples."
    Found "various" in: ""

Otherwise great work, I like these very fast html/js tools.

Looking at the code, the issue is that both `findWeaselWords` and `getContext` construct a list of words but do it differently:

  const words = text.toLowerCase().split(/\b/)

  const words = text.split(/\s+/)

I pasted these two comments into Claude and had it update the tool: https://gist.github.com/simonw/dc79f6adcdb189469890bc0a44331...

New version is now live at https://tools.simonwillison.net/writing-style

Why not just use Claude itself to review the writing, instead of having it write a much less capable and brittle and limited bash script to do it? You could even ask Claude to write a prompt for itself that performs the same or better function than the bash script. Bash and Perl are like duct tape and chewing gum, so terrible for that kind of stuff, and it's just what Claude does best. It can go so much further by actually weighing alternatives and suggesting changes, instead of just flagging problems. And no weird regular expression inconsistencies that cause false positives and negatives and parsing errors.

My goal here wasn't to build the best possible writing analysis tool, it was to try out these 2010 bash script rules in a slightly more convenient format (and to play more with Claude Artifacts).

I've actually had a tiny bit of trouble using LLMs for writing analysis in the past, though that was more about spell checking. I found they often missed obvious errors, probably because the tokenization step means they don't "see" the exact original prose in a way that makes those errors as obvious as they are to me.

diction and style commands were present in early Unix, which did similar jobs.

You can get them (or versions of them) from GNU. They're in homebrew.

https://www.gnu.org/software/diction/

From what I remember, they weren't great; this bunch of programs probably does just as well.

In my field, writing quality was on the very lowest rung of importance, below even teaching evaluations. As much as I value clear, concise prose, I’d say a grad student would be better served working on public speaking especially when faced with hostile questioning, and, sadly, with brown-nosing. Yes I am bitter :p

how can you tell? likely this is field-dependent but even if because reviewers dont tend to comment on writing quality it doesnt mean it doesnt play a factor in acceptance. if you annoy people, (sometimes even especially) in ways that dont feel substantive enough to merit mentioning in a review, it can make them more inclined to be critical about aspects they might otherwise gloss over.

(i acknowledge this is unsatisfyingly unfalsifiable, and that it can also go the other way, in that selectively bad writing can be used to attempt to paper over holes)

I think even more than public speaking is just seeking a therapist. People get in over their head but its by their own doing. The stakes are never as high as you have built them up in your head. People want you to pass.

> To market a paper, the author must make a compelling case for why her idea deserves access to that resource.

In other words, journals are filled with papers that were sold the best, not the most important ideas. And as the author also says, superficial things like hard to detect typos are often a deciding factor because the reviewers can detect them.

we should stop pretending there is objectivity and embrace journals that reflect taste and opinion of the editor.

Or have standard places like arxiv for publishing everything. There is no scarce resource for uploading pdfs.

The editor is not the one making the call really. Its the reviewers. If they give it few marks or no marks then that’s it, its published. If they dog that paper down then the editor has to sit up in their chair and actually decide whether or not the authors made sufficient change to address the reviewers complaints. But even then its still the reviewers who are demanding the standard, not the editors.

I desire the power to be with the editor.

The editor is the one who decides if to goes to review. Lots of 'top tier' journals the editors will quickly reject the manuscript.

While we're at, improve the citation system, so I the references become hyperlinks, and code artifacts can be accessed.

Useful for those who, for some unfathomable reason, don't use emacs.

Might be hard to get into a good workflow as running them and then re-editing seems tedious.

Author gives credit to emacs "writegood", but my all-time fave style-nazi plugin is "artbollocks-mode".

Shell scripts to improve your writing - https://news.ycombinator.com/item?id=13295530 - Jan 2017 (55 comments)

Shell scripts to improve your writing, or "My advisor rewrote himself in bash." - https://news.ycombinator.com/item?id=1529166 - July 2010 (31 comments)

Good intentions, indeed. Creating lots of steering committee slides, I know about the wish from the audience of a simpler language. But ‘very close’ is different from ‘close’. It’s not just salt and pepper but trying to articulate a complex and nuanced reality. And yes, research papers then sound a bit less solid and complete- sorry, but often this is the reality you should not hide.

> But ‘very close’ is different from ‘close’.

What's the difference?

it's a way to express steps in a continuum (given by the context): - not close (1) - close (2) - very close (3) - arrived (4) That's how language works. It's not mathematics and uses "salt and pepper" to convey the message as accurately as possible.

Can confirm the value of checking for repeated words such as ‘the the’. This is the final example I give my Econ PhD students in a session on RegExps: https://users.sussex.ac.uk/~gm268/iphd/regexps/regexps.pdf

This is a great summary how to use regexps, complete with some very nice examples. Thank you!

Just so long as it doesn't blindly modify "that that".

Right. You can just search for /\b(\w+) \1\b/ and see what comes up. There are usually false positives of one kind or another.

I once wrote a sentence with a double “in” in it — it was something like “let's see what state this is in in two weeks’ time” or similar — and at least two people commented thinking that it was wrong and needed an “in” removed...

As a humanities sort I appreciate this but the scripts sort of go against the general thrust of the text since the scripts can not understand context or semantics, it feels like they would push many towards blindly following prescription instead of what he is advocating for. I think elaboration would have served better than bad, better and good examples which do not explain the issues and assume the reader will intuitively understand. We get some elaboration but not enough.

Gardner's Modern English Grammar should also be the primary recommendation for further reading, Gardner has a gift for explaining the nuances of these things. Style guides are guides for the style of a given publication or writing within a discipline, not guides on writing well or with style.

Yes, I think the tool should highlight usages to double check, not blindly apply changes and submit.

Put "AI" in the name and sell it for $5M?

I found this an interesting read: https://www.cambridge.org/us/universitypress/readers-brain

It gives a neuroscience perspective on what makes certain writing styles clearer.

    I'd even go so far as to say that the removal of all adverbs from any technical writing would be a net positive for my newest graduate students.

I've heard this before. Stephen King hates adverbs. However, it can be very difficult to remove all adverbs from your writing. "randomly" for example is an adverb, and if your sentence uses it, it can be difficult to rewrite the sentence without "randomly" that isn't long and complicated. Many adverbs are emphasis words that don't need to be in the sentence ("extremely" for example), but other adverbs are critical to the sentence.

I only heard the recommendation to completely avoid adverbs for the first time the other day, although in the context of creative writing rather than technical. I think there's validity to trying to define a uniform style without fluff for technical writing, but I can't help but think that trying to studiously follow personal stylistic decisions of famous try to improve as a writer won't usually end up with something particularly original. Does anyone think that Stephen King got where he is today by just blindly following idiosyncrasies from writers who came before him rather than developing his own style?

Funny thing is, he doesn't even follow his own advice. Many of his books include lots and lots of adverbs (The Shining, for example).

Besides, adverbs make writing more emotional. Would you rather read the sentence "I'm extremely thirsty" or "I'm thirsty" - the former has more punch and describes the state of thirst better. Try re-writing that sentence describing extreme thirst levels without adverbs, you start getting into an overly technical, hard to read writing style. "My thirst is extreme" (a sentence which doesn't describe the feeling of being thirsty) or "I am in a state of extreme levels of thirst" (a sentence that sounds too technical).

The advice should be "don't use so many adverbs," not "don't use adverbs at all." Adverbs exist for a reason.

This what Claude/Chat do when you ask them to make text more concise.

Have to say, Chat does produce what I am looking for in one go more often. Claude always makes lists or makes things way to concise. Maybe I need other prompts.

duplicate words:

    sed -z -r 's|(\b[^ ]+\b)(\1)+|~Dupe!\1~|g'

The script appears to be missing a check for "long series of single-sentencr paragraphs", which quite harms readability.

I use a version of these scripts all the time.

though all-in-one spell-checkers like vale.sh are more convenient.

Next: create a bash script to find out if the PhD advisor reads HN

Wouldnt need a bash script, phd advisor probablu gave you the password to everything already. Use the code they gave you into their office. Log into that laptop on their desk and use the gui to parse the history and see for yourself. If they catch you they probably wouldn’t even care.