Regulators are aware and are creating rules to prevent this. For example, the upcoming EU AI law requires companies to provide a detailed explanation for eg credit scoring if requested.