This is totally outside of my world. What OS would you run on this and what applications take advantage of this system? Or, is this for in house custom application development?
You'd usually run CUDA applications that can take advantage of the GPUs. Deep learning is all the rage these days, and the new A100 GPUs are particularly well-suited. Many HPC applications like computational fluid dynamics simulation take advantage of GPUs nowadays. There are also a number of visualization applications that GPUs excel at.