Hacker News Reader

I often write a script even if I only have to do something twice. In this case, the point is not to save time, it's to be secure in the knowledge that I did the thing in exactly the same way both times, because it was the same script that did it both times.

(And then it saves time anyway, because it turns out I had to do the thing 20 more times after all.)

It’s not full proof, but the heuristic that your stakeholder will ask you for the same thing next week (or immediately ask you to do the same thing but slightly differently) has held pretty true for me

yes, I've got a little heuristic I use to tell the beginner analysts at work when to think about automating their work or putting it into a script.

if you're doing something once and once only, maybe it's a candidate for the manual or gui way.

if you think you might do it twice, it's almost certainly time to automate or start programming it.

Anything encountered in business that you encounter more than 1 time is likely to be encountered N times more.

The good old "allow it never, once or n times" :-)

Exactly. A little {insert favorite language} snippet is immediately inspectable, also weeks or months or years in the future.

You underestimate my ability to write code that I myself can’t make sense of years (or sometimes even months or weeks) in the future.

Same for me – but that's usually an indicator that the spreadsheet situation would be even worse. We just suck, apparently, but we suck less with code :-)

This is spot on! I think sometimes if you do same thing 3 times it is already better to script it, unless you are sure that no further modifications will take place. Scripting will ensure reproducibility and it is invaluable when you can follow your logic later when you already forgot how you come up with that chart.

this is obviously laziness, the first of the three virtues http://threevirtues.com/

Also, if you need to do something (even once) and have to know how you did it three months later, write a script.

I usually save my command history to version control every time I git commit. It helps a lot for less frequently visited projects.

You can set your HIST_FILE per project. That’s what I do in vscode (terminal env), so I have a history per project.

Other than that, I usually document some ops/quick read me about how to use or prepare the project

Yes, the lack of reproducibility is really Excel’s biggest weakness in data analysis. Is this 5th workbook that is a 4th generation derivation from the original data statistically valid? Can you go back to each step and verify it was valid? A script that does all processing and analysis starting from raw data can be validated and audited.

This!

> I want to move a thousand files beginning with 'UTR-77' and ending with '.csv'

1. Use a file manager with stable sorting [1] (I use Thunar which does this, but I suspect lots of file managers keep the sort order stable).

2. Sort by type.

3. Sort by name.

4. Select the first file named UTR-77.

5. Scroll to the last file, and Shift+Click it.

6. Cut then paste to your desired directory.

[1] - https://en.wikipedia.org/wiki/Sorting_algorithm#Stability

> 5. Scroll to the last file, and Shift+Click it.

With thousands of files, this won't be particularly fast or easy.

If you know how to use a scrollbar, it is fast and easy. Grab the thumb and drag it down. If you overshoot or have trouble landing on the last file, use the page up/down areas of the scrollbar or its up/down arrow to fine tune the scroll position. Should only take a few seconds.

Another way is to hold down the Ctrl key and use the PgDn/PgUp keys to scroll. If you hold down Ctrl+PgDn you can whip through thousands of files quickly. Because you have the Ctrl key down, it won't affect your selection of the first file.

If all you know is two finger touchpad scrolling, then it will be very tedious.

Unfortunately, from observing a number of people - and apps that hide the scrollbars - it seems to me that two finger scrolling has largely taken over from scrollbar scrolling or the scroll keys.

There are many circumstances in which it's not.

- Remotely accessing a system (terminal or GUI). - An overloaded system (GUI response is ... inconsisstent) - Remotely accessing a system from a touch-based device. An increasingly common scenario. - Walking someone through a process (text is unambiguous). - Repeated operations (something that has to be done multiple times, in multiple directories, on an ongoing basis, on a scheduled basis, reliably, consistently, provably, with documentation and debuggability).

Additionally, the scrollbar seems to be increasingly unpopular. I'm on record as not being happy about this.

https://ello.co/dredmorbius/post/0hgfswmoti3fi5zgftjecq (HN discussion: https://news.ycombinator.com/item?id=21356511)

I enjoyed your flippin' scrollbar rant! I couldn't have put it better myself.

Thanks :)

This feels pretty squarely in the realm of "but you could still do it manually". The CLI method still seems cleaner and faster, with less likelihood of error since you don't select the files manually.

I had the same initial reaction, and I think conversations like this often include desperate justification of GUI workflows driven by an inferiority complex.

But the parent comment was responding to a portion of its parent that specifically said "there's no option other than scripting". It sounds like he's responding to that, not making some general claim about the GUI being a better option.

How is not selecting it manually but implementing code that could select the wrong files any better?

rstudio is an excellent gui for R. So you can already start with R.

So is https://exploratory.io - I almost always start with it and export the R script once I have some intuitive confidence in my data.

> I would benefit from learning `mv UTR-77.csv $folder`

In far manager (freeware, open source) and total commander (commercial, not too expensive, trial available), numpad `+` key open “expand selection” popup, where you can type “utr-77.csv”, and it will select just these files.

Unlike the command line, you can inspect what had been selected before moving these files. You can also use insert/numpad +/numpad - keys to modify the set of files going to be moved/copied/deleted/zipped/etc.

Most file systems and file managers don’t have undo support. It can be important to review what going to happen before actually moving any files.

This is a great comment and the line of reasoning for work at different scales makes so much sense.

Not sure I agree with this. Most statistical analysis is a 'do once' thing, but using Excel would not be a good choice. Reproducibility and immutability of the source data are two good reasons to write a script.

And then there is the fun tasks, which I have always considered to pit the CLI guys against the GUI: Copy all cat pictures to different directory. (No metadata or names)

But why not use the scripting languages built in to Excel?

Agreed.

I found that "Automation"[1] by xkcd (R Munroe) closely captures the lifecycle of my "clever" scripts.

I have a few of them glancing at me from the corner of my desktop and I hope to do something "over the summer".

[1] https://xkcd.com/1319/

> If I want to move a file from one folder to another then I just drag it across. Easy.

Even this is easier with scripting, especially if you're already used to thinking in wildcards and tab-completion. Scrolling, hunting for files, dragging, clicking: these are inherently clumsier and slower steps, optimized for new-user intuitiveness and simplicity over efficiency. The upfront investment of making your brain think in CLI is fairly high, but once you've done it, there's vanishingly little reason to bother with file browsers. I don't think I've used one in a decade, even with (eg) Nautilus's ability to match wildcards with ctrl+s.

For those who haven't used it, ctrl-r in the command line is the best thing in the world. It does a reverse-search over previous commands. Set your max history size to unlimited, and your CLI will remember /all/ commands you ever type, making it very easy to find long, strange incantations that you figured out once, so long as you can remember some small part of the command.

For example, How did I do that magical ffmpeg thing last time? Just hit ctrl-r, type ffmpeg, and keep hitting ctrl-r to find previous examples until I find what I'm after.

For more complex commands I use often I sometimes comment the end of my line with a keyword I can search on

Does this work on Mac OS?

I’ve got a bunch of saved notes with various incantations that would become redundant if so!

Seems like it should!

https://stackoverflow.com/questions/41780746/searching-your-...

Good to double check how to turn off the history size limit before throwing away the post it notes entirely.

> I’ve got a bunch of saved notes with various incantations that would become redundant if so!

Throw them in scripts! Another advantage of using a CLI is that it provides a friction-free path from ad hoc commands to saved commands to messy scripts to properly supported tools. I've ridden this gradient multiple times prfoesssionaly, especially now that I work with a bunch of PhDs who are less comfortable around OSes than I am. It's a good feeling to have such a clean pipeline from "this command/series of commands feels awkward to me" to a reviewed, checked-in tool that becomes a critical part of the team's workflow. The best part is that there's value at every incremental step, so you don't need to invest any organizational effort, which is especially useful in a chaotic execution environment.