Of course it can. Like 50% of efforts in safety are to stop the model outputting popular lies, oft-repeated software bugs and slanderous hallucinated falsehoods.
If Mistral puts out a model it's only downloaded by enthusiasts who understand its limitations, and doesn't get a load of scrutiny from journalists in the way Apple and Google products do. If sometimes it falsely claims Richard Stallman was arrested for lassoing a police officer - Mistral won't see any blowback from that.
On the other hand, if Google released that model, and pushed it hard in search and whatnot, so search results started including such false claims - that's going to get a lot more notice.
The user must be responsible in the model's usage, or else some law will apply. The company is not liable for misuse.
And for things like "repeating bugs", maybe the model won't do that anymore if we focus on making it more capable and smart? I believe OpenAI stated that the guard rails in GPT-4 made the "sparks of AGI" phenomenon disappear. That's just bizarre reasoning for these companies, in my opinion.
The media would be all over google once someone used it to create something "disturbing"..
Everyone says that until they are in civil/criminal court getting millions/billions in fines.
Put frankly, the law gives zero fucks about that. Looks like child = jail is the law in many countries in which Google operates.
To be graphic as possible, if your commercial AI application generates childporn you will be fistfucked by the long arm of the law in at least the US, Canada, and Japan. While chasing individuals is more difficult for law enforcement a big juicy target like Google could rack up billions in fines.
Of course, an actual lawyer could probably explain why this would be a bad idea, but at least I as a non-lawyer don't see where the problem lies (aside from the fact that the company wouldn't want their stuff to be used to create CSAM, which is fair).
But these AI systems do not just generate CSAM by happenstance or the user tricking them into doing it. These systems have their training data obtained by scraping everything off the internet. There is a lot of CSAM (and whatever you will call text/literature describing such things) on the internet, and so, in the training data.
OpenAI had some of their Kenyan data tagging partners quit the contract because they had to tag which images and text were CSAM. (With none of the already-insufficient mental health resources afforded to social media moderation staff)