Hacker News Reader

Same goes for something as simple as setting up a server. You will forget, and if you don’t write it down you’ll have to figure it out again.

Where did I put those configs again? Where did Bob put that script? Fuck, why didn't I write an ansible script. It's never a one off, and it serves as documentation. I'll remember after I make the same mistake next time.

Also, environment modules for the win

https://modules.readthedocs.io/en/latest/

One time my boss asked me to upgrade some servers and was surprised I put all my work in a script in version control. Then there was a second batch of servers.

Use nix and you’ll at least will be able to deploy the same again.

So what happens in nix if a package version is found to have some huge vulnerability? And my app expects that exact version?

Because this is literally what docker, venv, and nix claim to fix but after getting burned by three other systems I'm not willing to invest the time in getting in to nix.

I keep machines powered off that have a working configuration of older AI and other software tools because there is no other way to run them, regardless of the code being available on github.

There's other solutions too, this is a big part of systemd. Which also has nspawn and vmspawn do more explicitly. But everything has some containerization capabilities and ideally you'd give your program access to only what it needs. Privatetmp should always be on.

But if versions are vulnerable you usually want to remove those versions, not put them in containers

> remove those versions, not put them in containers

I don't know how to fix this, but perhaps i can ai it and release something on my github if i manage to cobble something together.

These aren't "services" that anyone has access to, except myself; "clients", UIs, and things like whisper.

IF someone were to pay me, I'd figure it out. I'm friends with maintainers and that isn't my style. archiving is.

to wit, i expend no more energy than necessary maintaining other people's code.

I really do not follow what you are trying to convey here.

If there are vulns, and you are using software from nixpkgs, there are tools to get yourself notified about vulnerable packages.

If you want to run vulnerable software on-demand, you can just boot the machine/vm up when needed? If you want to patch stuff yourself, nix makes it trivial to apply your own patches to already packaged software.

I didn't write, nor do i maintain the software i use. for instance, i have a couple of old, but perfectly working web-UI for stable diffusion and LLM. If i, say, copy the folder to an ubuntu 24.04 server install, there's a real good chance it won't load at all. I can't fix that. I mean, i can, but i can't fix it for free. It's cheaper and lazier for me to just maintain the machine/VM for my own purposes.

So in the case of Nix, i am still unsure if, for example, i had a Nix config for some software (say whisper-ctranslate2), is that permanently portable? Is there a way to make it so? Right now, with stuff that needs a GPU, the only guarantee i've found is to either do pcie passthrough and use a CoW VM that does incremental backups to NAS, or, if the publisher provides, a dockerfile / container.

For nearly everything else, i use a rolling release server OS and keep stuff up-to-date. bleeding-edge sci-tech, being publish or perish, doesn't much care about the published code working much past the git push.

For comparison, every time i've tried to use a node.js distribution of a package, there's some blocker because a pulled-in library version is vulnerable, and i'm even less likely to be able to fix that than some C++ or python thing.

If your work is tracked in an issue-tracking system, you can put your in-progress thinking notes as comments there (if they don't go in the code or some other artifact).

It helps to have a safe environment, among whomever might access that issue comment history. If people don't feel safe exposing their thought process, then they won't do it, or they'll be stressed by presenting vulnerability, and even modify their problem-solving for appearances.

(I have some more complicated options involving a wiki, but explaining requires too much context. The issue-tracking comments solution is obvious.)

What I try not to do is to introduce new places that important information goes. If you don't rein this in, there will be an explosion of employees plastering your IP all over a bunch of random SaaSes, to be undiscovered or even lost to your company entirely (also, those other SaaS companies and hackers might get more use out of stealing your IP than you do).

Blame -> pr -> issue is a great way to learn a codebase if your team is good about keeping a log of their work, which they generally should be.

Chaotic energy haha, I like it. Thanks for the tips re: keeping a journal, I will do this more in the future. I usually keep development notes, though normally in markdown files scattered across the codebase or in comments, never by date in the README. In the future, I might make JOURNAL.md a standard practice in my projects? re:w&b, I used w&b when it first came out and I liked it but I'm sure it's come a lot further in the time since then. I will have to take a look!

Also lol "pretentious perfectionist" I'm glad to finally have some words to describe my design aesthetic. I like crisp fonts, what can I say.

  > Chaotic energy haha, I like it

My boss says I'm eccentric. I say that's just a nice word for crazy lol

> normally in markdown files scattered across the codebase or in comments

I used to do that too but they didn't end up helping because I could never find them. So I moved back to using a physical book. The wandb reports was the first time I really had something where I felt like I got more out of it than a physical book. Even my iPad just results in a lot of lost stuff and more time trying to figure out why I can't just zoom in on the notes app. I mean what is an iPad even for if it isn't really good for writing?

But the most important part of the process I talked about is the logging of all the parameters and options. Those are the details you tend to lose and go hunting for. So even if you never write a word you'll see huge benefits from this.

  > re:w&b

Wandb's best feature is that you can email them requesting a feature and they'll implement it or help you implement it. It's literally their business model. I love it. I swear, they have a support agent assigned to me (thanks Art! And if wandb sees this, give the man a raise. Just look at what crazy people he has to deal with)

  >  lol "pretentious perfectionist" I'm glad to finally have some words to describe my design aesthetic

To be clear, I'm actually not. Too chaotic lol. Besides, perfectionism doesn't even exist. It's more a question about personal tastes and where we draw the line for what is good enough. I wish we'd stop saying "don't let perfectionism get in the way of good" because it assumes like there's universal agreement about what good enough is.

Parameters and options, got it. I try to keep all configuration declarative and make building and running as deterministic as possible. Then I can commit whenever I do something interesting, that I can just checkout to revisit.

I think these are the two main headaches with experimenting. No matter what kind of experiment you're doing (computation, physics, chem, bio, whatever)

  - Why the fuck aren't things working
  - Why the fuck are things working

The second is far more frustrating. The goal is to understand and explain why things are the way they are. To find that causal structure, right? So in experimenting, getting things working means you're not even half way done.

So if you are "organized" and flexible, you can quickly test different hypotheses. Is it the seed? The model depth? The activation layers? What?

Without the flexibility it gets too easy to test multiple things simultaneously and lose track. You want to isolate variables as much as possible. Variable interplay throws a wrench into that so you should make multiple modifications at once to optimally search through configuration space but how can you do any actual analysis if you don't record this stuff. And I guarantee you'll have some hunch and be like "wait, I did something earlier that would be affected by that!" and you can go check to see if you should narrow down on that thing or not.

The reason experimenting is hard is because it is the little shit that matters. That's why I'm a crazy pretentious "perfectionist". Because I'm lazy and don't have the budgets or time to be exhaustive. So free up your ability so you can quickly launch experiments and spend more time working on your hypotheses, because that task is hard enough. You don't want to do that while also having to be debugging and making big changes to code where you're really just going to accidentally introduce more errors. At least that's what happens to my dumb ass, but I haven't yet met a person that avoids this, so I know I'm not alone.

Converged to something similar after spending 2 days bissecting a repo to reproduce a training run, having to wait 3hr on each commit before conclusive results. I couldn't get myself to use hydra though, it felt like a lot of bloat vs loading a yaml with pydantic

The problem with keeping a journal is that the distraction of doing so may break the state of flow.

OTOH there are natural breaks in the process of working; writing things down during these works fine. The fidelity is a bit lower, but it's still much better than nothing.

Agree, it's much better to write up a journal at times when your colleagues would be https://xkcd.com/303

  > the distraction of doing so may break the state of flow.

Sure, but like you said, don't do it when in the state of flow.

Or better, make it part of your flow state. To me, it is part of my flow state, so not a real issue.

I mean whatever works for you. You gotta time manage and I can't manage for you. I'm sure your boss is asking for more writeups than I am and just send them your notes. They don't care whats in it half the time, they just don't know how to figure out if you're working or not and just want something.

Hell, we're on HN on a workday... I can guarantee you aren't in a flow state the whole time and can't be bothered with a few minutes to write some stuff down. I mean you have to eat and go to the bathroom, right?

From my two minute skim of the docs, not encouraging that hydra only officially supports up to Python 3.11.

I use it in python 3.12, and 3.12 just got out of bug fix. I haven't tried 3.13 but I would be surprised if there was a break. Most of it works through OmegaConf[0].

Idk why they haven't pushed an update in 2 years but neither has this been a problem. FWIW, they're still updating the repo[1]

[0] https://omegaconf.readthedocs.io

[1] https://github.com/facebookresearch/hydra

OK so I try to do that. But then I'll have some big problem or add too many big features and just give up. (My sleep is nearly nonexistent so I don't have a lot of time for logging things anyway.)

You might be doing it wrong. Make your code more modular. Think about how powerful functions actually are. But also make them simple and self contained. All those programming books suggest this not because "it's pretty", "good form", or whatever. You do this because you know that things never work out like you expect them to. Only a naive programmer or a literal god thinks they'll get the program right on the first go. Even just the fact that the world changes underneath our feat means it will change.

So you write expecting things to change. You write so a change can be added quickly and not break everything else. You write so you don't have to pull apart a bunch of tangled mess. There's a lot of complexity no matter what, so even a little goes a long way.

  > My sleep is nearly nonexistent so I don't have a lot of time for logging things anyway

I'll make a bet.

I'll bet that if you log you'll get more sleep. This is a classic negative feedback loop and is honestly why I started doing this in the first place. A little extra work upfront saves me a lot of work down the line. You need to be concerned with today but that doesn't mean you can ignore tomorrow.

The point of my strategy is in how positive things compound. A little here, a little there, do this for a bit and you got something beautiful while it seems like you did no extra work (because you spread it out)

But the negative effects compound too. They create more work. The less sleep you get the more mistakes you make. Worse, the more subtle hard to catch mistakes you make! You just end up missing sleep chasing down bugs and issues introduced because you wrote while being sleep deprived. We all do this! But we need to recognize it and try to break this cycle as soon as we recognize it happening.

My bet is you are the one creating most of the work that is keeping you up and making you feel over burdened.

My bet is if you take a break you'll actually get more done.

My bet is you're caught in a destructive loop.

I'll make this bet because I have so much experience with this same self-destructive behavior. Been there. Done that. I don't want to be there nor do I want you to be. But to get out, you have to fight that impulse that got you there in the first place.

Thanks so much for the thoughtful and encouraging response. Unfortunately proper logging would do nothing for my sleep, which has been abominable for 60 years.

My current workflow is to keep a wiki, would you say hydra would replace/complement especially that if you're used to note keeping the wiki way?

Hydra is part of the documentation process imo. Truthfully, the most important stuff that goes in your experiment journal is all those pesky parameters and things that can surprisingly change results.

So I love that hydra uses OmegaConf and I essentially get 3 copies: the experiment config yaml, the wandb log, a dictionary in the checkpoint. Multiple times my dumbass has had to try to match the checkpoint to the wandb log, so the redundancy is incredibly helpful. Sometimes just a library version has unexpected changes on performance and this makes it trivial to trace. The yaml file is more helpful when passing off the code to someone else or releasing to public.

So yeah, I would say that it'll benefit no matter how you document. Use whatever documentation method works for you. Reports can still offer some benefits in just throwing some charts together quickly and organizing but I think you'd still benefit from hydra. It's too easy to lose track of those little things and this helps me automate. But you can also just straight up use OmegaConf or even dictionaries. Whatever works for you.

The real help is logging. So whatever tools help you log, use them. This is just what I benefit from (there's a lot I can talk about too and I'd love to see what others do as well)

Thanks, appreciate the detailed response. I'm going to have to check it out.

Hydra and omegaconf are almost officially abandonware, you shouldn't really depend on them.

Anyone use a digital notebook, like the reMarkable, for this kind of thing?

Started but stopped. most of the things are commands, code snippets, urls, all of which are tedious to hand-write and just easier to copy&paste. Often I do 'typescript' for shell sessions, asciinema or stuff like that, and file those.

Also, use git, commit everything, never care about doing tidy commits, just commit commit commit and use tons of branches to try out stuff. if you need clean history later on, you can always do interactive rebase, squash merge or whatever. but having a documentation of all the things tried and failed is far more important.

I thought the entire point was that it syncs cross platform really quickly and lets you have the best of both worlds?

Only handwriting in a proprietary format. It isn't at all like one would wish for. It works as a replacement for a paper notebook. But it largely ignores the things one could do when adding more digital embeddings. In that way it is even worse than OneNote.

If you want to do something like that, my recommendation would actually be something like OneNote on some Windows tablet.

Work logs are generalized at my company and they are AWESOME

In my company also. We put everything into our github issues. Some things also into the README.md or the github wiki, but it's usually lost there

All of this agreed.

Now in the age of AI, many students entering into CS need do this NOW, otherwise any answer they come up in the interview, will be assumed that it was from an AI and they need to show that they something useful came out of their blogpost or research.

It is what it now means to know how to experiment, understand and build knowledge, rather than spitting out the answer because it it from stack overflow or ChatGPT.

The mistakes are raw, all the learnings in a blog post which is what makes us human yet, 90% of candidates do not do this which is why most of them cannot explain an AI's mistakes an interview if they use it.

I actually really like this idea. I've often found it odd we don't show off reports or how we run experiments during interviews. Certainly this has far greater influence over your aptitude than leetcode.

  > 90% of candidates do not do this or cannot explain AI's mistakes an interview.

I have a growing concern that people do not see mistakes. This seems to be a bigger divide than "uses AI to code" vs "doesn't".

So here's the thing I struggle with. I do a lot of work in jupyter notebooks. I come up with a new model or approach to some problem, and I want to fork out and test a hypothesis in the background (which might be some set of hyperparameters, and might take several minutes, or hours; call it Run A) while continuing to work down some other path in the same notebook, and maybe kick off a Run B that explores some other change (like a restructure of the code that's not "compatible" with the hyperparameter search of Run A).

Then at some point when Run A finishes, I want to incorporate the changes I made in Run B and kick off Run C, and so on.

The hard/important things are:

1) Being able to do this while staying in a Jupyter notebook context the whole time. Even something as simple as multiprocessing sucks because I've found it's too hard to manage in a Jupyter context (e.g. how do you handle where stdout and stderr go?). It's easier if you move to scripts where you have full support for this sort of thing and you are expecting to look at multiple log files on disk and whatnot.

Also the sequential nature of notebooks doesn't help when you want to occasionally fork out or conditionally run stuff.

2) Keeping track of all these changes and hypotheses and merging the results/code together as you learn. It's like you need a VCS for your hypotheses. Maybe hydra & wandb help with that, I haven't used them. But this idea of keeping track of hypotheses seems like the more fundamental thing.

3) The main reason I prefer to stay in a notebook context is because I have all my objects easily accessible. My models, all my dataframes, functions to do some ad-hoc charting etc, all super easy to access in a REPL-like form. That is invaluable for doing ad-hoc sanity checks or digging/drilling down. So a big part of the workflow is you basically have this in-memory database of a bunch of relevant objects and you're querying it and constructing new objects & visualisations using Python as your tool, without having to load things from disk or build up the context from scratch. It's all "just there".

4) And then sometimes you want to take the results X1 of that notebook and plot them against some entirely different set of data X2 that requires a whole bunch of other code that you've defined in some other notebook somewhere, or maybe even as a real Python module. Like maybe that data lives in a database and you transform it or something. So OK, you call some functions to load X2 within your original notebook, but BOOM you get an OOM and you're like ok now I have to write some code to serialise X1 to disk, and make YET ANOTHER notebook so I can go analyze X1 and X2. It all just seems so... unnecessary, if only the right tooling existed.

My current best approach is to use semantic versioning on the filename, just copy the whole notebook each time I make a fundamental change, and try to keep track of my hypotheses, preconditions, learnings etc within comments and have a few of those on the go running, but it's often hard to engage in critical thinking when everything you know is sprawled across multiple notebooks.

Maybe a simple global journal is the only thing for this sort of use case. And that doesn't even address (4) which is often a huge pain point. Can anyone think of something better?

For me, those side-investigations are often physical experiments, which run on their own time scale. Plus they often run on another computer, to reduce the risk of crashing and losing data, or just physical proximity to the experiment.

How I tie those threads together is by the data that they generate. I use ASDF because it works for the kind of stuff I'm going, but choose your poison. Once the data are in the bag, the cells that analyze or report the results can stay in the same notebook, or be copied into your main notebook. My data aren't so huge that there's much of a penalty in re-loading them.

For me, reproducibility is more important than organization, because I'm not all that organized anyway. So, a single master notebook at the end of a study isn't my top goal.

  > I do a lot of work in jupyter notebooks.

I don't think I'll have good advice for you if you want to use jupyter notebooks, hopefully someone else will. I *hate* notebooks. I think they are great for reports or for demos (especially when teaching), but I honestly do not get how people use them in research or general programming.

I will use vim and ipython thought. If on a remote machine I'll use tmux (preserve sessions) but local I use Ghostty[0]. I can iterate through code this way without the notebook and am far less likely to get caught up by with out of order executions. I can get all that ad-hoc, persistent memory, auto-updating function benefits with ipython (autoreload). But vim and ipython don't require me to lose my modularity, organization, automated record keeping, and the rest.

If it works for you, keep it! I'm just saying what works for me (any switch will cause some disruption). But I do want to stress that there's tons of other options for keeping things in memory without using jupyter notebooks (pdb is also a fantastic tool!). But also be careful because persistent memory can easily bite you in the ass too. Easy to forget what's still in memory. I'll also add, that having also been the person that maintains our lab's compute systems, I'm wildly annoyed with notebooks and VSCode users leaving their workloads in memory. This is a user thing more than a tool thing but there's a tendency here and it eats up resources that other people need. Just make sure to disconnect when you leave the desk.

  > I want to fork out and test a hypothesis in the background

But my process does greatly help with this! IMO you should be trying to run experiments in parallel. Operating in this style there's no forking, you're just launching another job. I like using a job launcher like slurm when I have multiple machines but just a simple bash script to launch is often more than good enough.

The point is to not fork. You should clone. With changes, I suggest using git branches. But if your code is written to be modular and flexible it is often really quick and easy to add new functions to handle different tasks, add new types of measurements, or whatever.

The two big reasons to write like I do is that

  1) It is (partially) self documenting. You don't have to think about writing down and remembering all your hyperparameters. I'm going to forget and so I need to automate that to prevent this
  2) I'm running experiments! I may be dumb, but I'm not so dumb I think I am not going to change details of my experiments as the project matures.

That's why I say it is about being lazy. I'm writing the way I do because I know that whether it is tomorrow, next week, or 6 months from now, I'm going to need to make changes that are going to make things very different from where they are today. I don't think of it so much as having foresight about the future so much as I'm just frustrated at having to constantly dig myself out of a hole and this makes that a lot easier and lets me get back to the fun exploration stuff faster. It is 100% about having version control over my hypotheses and experiments.

So I'd argue you should move away from notebooks and use other better tools more suited for the job. It'll definitely cause disruption and you're definitely going to be slower at first but find what works for you. The reason people love tools like vim or love working in the cli is because they are modifiable. There's no one tool that works for everyone. I'm not sure there's even a tool that out of the box works for any one person (maybe the original dev?)! But there's a ton of power in having tools which I can adapt to me and the way I work. I can make it help me catch my common mistakes and highlight things I care about. You don't need to spend hours doing this stuff. It develops over time. But go into any workshop and you'll see that everyone has modified the tools for them. We're programmers, we have way more flexibility over customization than people working with physical stuff. Use that to your advantage. And truthfully, you should see how that idea becomes circular here. I'm just designing my code and experiments to be like my tools: environments to be shaped.

[0] https://ghostty.org/