So.... I'm guessing if the AI messes up spectacularly, or if some reddit group manages to trick it to overinflate some penny stock, the stock markets will undo all the trades, so that billionaires don't lose a few million?
Any firm worth their salt has risk systems in place to prevent runaway trading too, and those aren’t black boxes like AI/ML. They’re limits set for specific strategies or teams etc, and typically require approval from firm partners to increase past certain levels.
Collapses of firms are far more often caused by lapses in proper risk management than bugs in code. The KCG collapse looks like a bug on the surface but was ultimately a misuse of accounts that sidestepped these risk systems, so they couldn’t stop the runaway caused by the bug.