The full dataset can be found here: https://huggingface.co/datasets/conceptofmind/FLAN_2022
I worked with Shayne Longpre the main author of the FLAN collection to recreate his great work and publicly release high-quality instruction tuning data. We fixed encoding issues and also increased the sequence length to 4096: https://twitter.com/EnricoShippole/status/166175616624899686...
Each of the individual submixes is also available on huggingface to download. The sub-mixes are T0, FLAN2021, CoT, NIv2, and Dialog. Each contains relevant metadata such as Inputs, Targets, Task Source, Task Name, and Template Type.
T0 submix: https://huggingface.co/datasets/conceptofmind/t0_submix_orig... Flan2021 submix: https://huggingface.co/datasets/conceptofmind/flan2021_submi... CoT submix: https://huggingface.co/datasets/conceptofmind/cot_submix_ori... NIv2 submix: https://huggingface.co/datasets/conceptofmind/niv2_submix_or... Dialog submix: https://huggingface.co/datasets/conceptofmind/dialog_submix_...
You can find the original FLAN repository and all of Shayne Longpre's incredible work here: https://github.com/google-research/FLAN/tree/main/flan/v2#do...
Be sure to also read through Shayne's paper on the FLAN collection to get better insight into how the data was created: https://arxiv.org/abs/2301.13688
We are going to soon be releasing a massive causal language modeling dataset containing hundreds of GBs of high-quality instruction data. Look out for that release in the near future.
Our work on an open reproduction of FLAN V2 and related projects is all thanks to the generous sponsorship by CarperAI and StabilityAI. You can find out more about CarperAI here: https://carper.ai/ And StabilityAI here: https://stability.ai/
A big thank you to Jason Phang and Fabrizio Milo for helping build the dataset as well. You can find Jason Phang's twitter here: https://twitter.com/zhansheng And Fabrizio Milo's here: https://twitter.com/fabmilo
You can check out Shayne's new paper on building pre-training datasets here: https://github.com/shayne-longpre/a-pretrainers-guide/blob/m...
This is not an official Google or StabilityAI product.
If you have any questions about the data be sure to reach out and ask! I will try to respond promptly: https://twitter.com/EnricoShippole