Don't be simplistic. Mold on old bread grows and is natural, yet you should not eat it all day.
How sure are we about this? How certain are we that those specific species of mold have a net negative effect, rather than a net positive (like for example mushrooms)? Penicillium grows on stale foods and I doubt eating it would have a net negative effect.
Feel free to eat it.
"Penicillium Species and Their Associated Mycotoxins" - https://pubmed.ncbi.nlm.nih.gov/27924532/