Actually the question was more around "how do you create your models and what do you mean treating them as code", "why slurm and not something like airflow" , "what is the test/performance setup - backtesting, smoke test" etc etc
The Gitlab stuff is easier to understand.
> how do you create your models and what do you mean treating them as code
we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not taken too seriously tbh)
> why slurm and not something like airflow
because the HPC resources we have access to are built with Slurm, which is super fast, supports DAGs of jobs, schedules our jobs reliably and quickly. I don't really want the other stuff on the Airflow feature list to be honest.
>we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data
This is the part that everyone seems reinventing. Have you looked at PyML (https://eng.uber.com/michelangelo-pyml/). What are some of your learnings around jupyter -> production code. A lot of these are around conventions - "write a function called train(), fit(), test()". Is that the basis of your pipeline as well ?
Usually when we are doing more of the train/fit/test cycle, there’s an argparse script to quickly try different parameter values succinctly (which is run and tracked by the above CI setup)
I wouldn’t say we’re reinventing since a better solution isn’t very clear (though PyML et al look interesting)
edit forward simulation isn't a frequent thing in posts on generic ML algorithms, so just as an example: suppose you run a model and see an oscillatory component along a temporal dimensions in your residual error, and you add a oscillatory component to your model, and rerun it but still see a residual with an oscillation. You can run a forward simulation of your model to see what frequency it's predicting and check against what's seen in the data, and fix it. This is a contrived example but when you have multiple competing priors or model components, this is an effective way to debug their behavior.
Maybe a missing detail is that our models are run-once, once results are QA'd, they are sent to relevant practitioner, so Uber's query-per-second stuff is irrelevant for us (for now), which I can see simplifies the deployment question enormously.
do you have specific questions?
Would love to know more about your packaging setup - the branch name to divide datasets is a nice trick (I'll use it as well).
How does your CI know where to find models ? Im betting you are using some kind of convention here - one model per py file...so package each py file in a docker container.
If it is possible, would love to see the skeleton structure of one of your pre-packaged files.
Tldr - it seems you invented something like pyml as well. Are the deployment scripts+model skeletons open source ?
In the ML projects, it serves mainly to package dependencies, and to ensure some basic security constraints: raw datasets are accessible read only, ensuring that if we suspect some issue with cached results (cause our inner orchestrator is Make..) we can nuke all the results and start over from scratch, sure the raw data is intact.
The models and arguments are in the CI config. No magic there, but since it’s all in the repo I’m ok with it.
This whole setup was put together for an upcoming clinical trial as steps toward ISO quality norms compliance, and I can’t share it now. I do intend to reproduce it in an open form alongside our existing software (GitHub.com/the-virtual-brain) when it’s ready.
In any case I appreciate your questions a lot: they drove me to think a little harder and see why stuff like Michelango and PyML is stuff that even we (academic/clinical) group should be using... if we can find the time to do it.
Pachyderm is another one I’ve looked at but we don’t have the sys admin bandwidth for that stuff right now.