Ask HN: Shell-Based Research Workflow?

Yesterday I was trying to figure out how to scrape odds from a betting website. This sort of thing involves a combination of trial and error, and a lot of intermediate files which are easy to lose track of. Also sometimes you will overwrite a result produced with other command parameters which you later need again. You will also lose track of the command and parameters you used to produce a file.

I am wondering how to organise this sort of job better.

Here is what my job involved:

1. Use Python and Selenium to give me the source of a page after it finishes loading. 2. Put the source through xmllint, look at it in an editor, find a script tag where the odds lie in some very obfuscated JS. 3. Extract the text of the script tag with xmllint. 4. Put the output of that through a JS parser, and extract the desired odds after trial and error to find the right combination of parameters. 5. Use another tool to format the odds as a TSV, 6. etc., etc., etc.,

As an example, Mathematica has notebooks which allow you to organise your job and go back to previous results. Normally everything is ordered by time, but you can also rearrange things. Mathematica notebooks actually allow you to construct a tree out of your results.

You could also view my job above as a tree. Each call of a command takes some input files and gives an output file, which maybe is used by another command. Maybe I make multiple invocations of a command before realising that I want to go back to a previous result and use that as input for further research.

How do you organise your shell-based research jobs? Does it involve some sort of file-naming convention? Is there a GUI for this? Do you have some sort of shell-based tool which shows a nice tree relating your commands, their parameters and their result files? Maybe there is an Emacs package written for this? Do you put things in separate directories?

I'm interested to learn about your methods.