What’s the advantage of the file system/repo/bespoke diag database over storing the numpy arrays in the existing database infrastructure?
Doesn’t implementing this system with HDF5 cause headaches for concurrency in either direction?
Doesn’t implementing this system with HDF5 cause headaches for concurrency in either direction?
1) if you've got models that are re-generated periodically based on new inputs/algorithm tweaks, then you can potentially end up with quite a few of these as you scale.
2) if you want to track the details that/debug the reason your production system made a given decision, you need to log not just your model but all of the parameters that went into that decision. If that type of decision happens many times a day, then you can end up with some pretty massive logs to go through.
In either case, storing that historical data in your transactional database can be a bit of a load, so it's ideal to keep it separate if you get any kind of volume. I've actually bumped into 2) at one job.