Interesting. Far from a domain expert in materials science, but I've worked professionally in this area. What's your method for identifying valid "novel" compounds? Certainly, anything actually novel has been included in the models' training set already, unless you're doing a CASP-like coordinated blind test...right?

As an aside, the "Fable lies and cheats" section made me laugh -- have encountered this same failure mode, albeit for much simpler models. Polymerization (I realize this is not exactly the same thing) is an unbounded 3D playing field for constraint escape. You can try to put an additional constraint on polymerization, but then it will just make minor variations on the monomers...