CP67 ... security provisions appeared to be
absolute -- run any code at all ... with full
safety and security.
There have been decades of high end production
systems running on VM. I haven't heard about
any security holes.
Once upon a time, perhaps 40 years ago (it's been very long, my memory is hardly exact) some IBM employees/researchers started with either CP67 or VM source code (I forget which) and found literally dozens of bugs and/or usable exploits for it.
Source code was not an unreasonable starting point, since IBM published all this code on microfiche, and also probably on mag tape.
IIRC many/most of the exploits revolved around emulating the I/O channel architecture, channel programs, and corresponding SIO or SIOF instructions.
They wrote the whole thing up in a prestigious publication, perhaps the IBM Systems Journal or Communications of the ACM or maybe ACM Computing Surveys. I did a very quick search but couldn't find that particular discussion.
Edit: I may have found the original paper. First I found:
A Taxonomy of Computer Program Security Flaws, with Examples
https://cwe.mitre.org/documents/sources/ATaxonomyofComputerP...
in which they reference a 1976 IBM paper that discussed many VM vulnerabilities. Here's just one of multiple times they cite that paper:
Case: I2
Source: C.R. Attanasio, P.W. Markstein, and R.J. Phillips, ‘‘Penetrating an operating system: a study of VM/370 integrity,’’ IBM Systems Journal, 1976, pp. 102-116. System: IBM VM/370
Description: By carefully exploiting an oversight in condition-code checking (a retrofit in the basic VM/370 design) and the fact that CPU and I/O channel programs could execute simultaneously, a penetrator could gain control of the system. Further details of this flaw are not provided in the cited source, but it appears that a logic error (‘‘oversight in condition- code checking’’) was at least partly to blame.