Also, it would be really cool if I can personalize FauxPilot by feeding it with all my repos on GitHub. Sometimes I just need to reimplement a function I've written before but it's really hard to find where my old code is.
deepspeed --num_gpus 1 --num_nodes 1 run_clm.py --model_name_or_path=Salesforce/codegen-6B-multi --per_device_train_batch_size=1 --learning_rate 2e-5 --num_train_epochs 1 --output_dir=./codegen-6B-finetuned --dataset_name your_dataset --tokenizer_name Salesforce/codegen-6B-multi --block_size 2048 --gradient_accumulation_steps 32 --do_train --fp16 --overwrite_output_dir --deepspeed ds_config.json
It might be doable to set this up on an AWS machine with a beefy GPU or two. I haven't tried it yet though.
Once you have a model trained in Huggingface Transformers you'd be able to convert it using this script:
https://github.com/moyix/fauxpilot/blob/main/converter/huggi...
{"text": "contents_of_source_file_1"}
{"text": "contents_of_source_file_2"}
...
Better alternatives: Google Colab, Paperspace Gradient, Lambdalabs Cloud, Vultr GPU instances
Colab will give you a T4, K80, V100 or P100 (alternatively their own TPUs) for free - $50 for 24h uninterrupted background jobs, Gradient will give you free A6000s and sometimes even free A100s for a $40 subscription for 6 hours (repeatable ad infinitum), Lambdalabs gives you a RTX 6000 for 0.50/hour and A6000 for 0.80/hour and Vultr GPU will give you 1/7th of an A100 for 0.37/hour
There was a project here some time back that allowed to call function using its hash. This avoided replication. It was entirely different paradigm to resolve dependencies.
Anyone remembers its name? It was controversial here but it was interesting nonetheless.
I'm fine with them accessing public repositories, as long as they respect the license. Which they probably aren't for private?
On a related note, Google apparently treats their free customers in a way where all their data is mixed in for the algorithm, but apparently not the data from accounts who use Google Suite (paid business accounts), instead they have their own recommendations built for them on a per-user basis, then has the public algo mixed in.
They don't respect the license.