But maybe that was still enough time for them to instruction tune it based on ChatGPT feedback, or at least to focus more of their fine tuning iteration in the areas they learned were strong or weak for 3.5 based on ChatGPT usage?
Training data is publicly available internet (and accessible to everyone). It's the SFT step w high quality examples which determines how well a model is able to answer questions. ChatGPT's virality played a part in that in the sense that OAI got the real world examples + feedback others did not have. And yeah, it would have been logical to focus on 3.5's weaknesses too. From Karpathy's videos, it seems they hired a contractual labelling firm to generate q&a pairs.
As far as data, OpenAI haven't just scraped/bought existing data, they have also on a fairly large scale (hundreds of contractors) had custom datasets created, which is another area they may have a head start unless others can find different ways around this (e.g. synthetic data, or filtering for data quality).
Altman has previously said (on Lex's podcast I think) that OpenAI (paraphrasing) is all about results and have used some ad-hoc approaches to achieve that, without hinting at what those might be. But, given how fast others like Anthropic and Google are catching up I'd assume each has their own bag of tricks too, whether that comes down to data and training or architectural tweaks.
To get that dataset now would take significantly more expense.
Most other model fail on basic stuff like the python creator on stack overflow question, they identify Guido as the python creator, so the knowledge is there, but they don't make the connection.
You're saying that when Mistral Large launched last week you tested it on (among other things) explaining jokes?