mikeravkine

This is exactly what happened when I tried this approach personally: on a non trivial codebase, the tests were just as likely to be wrong as the code itself and during "debugging" the wrong would converge until the test passed but the results were useless.

I still have my prompt from before and a Modal account, I'm going to give this one a shot to see if I missed something or if it really only works in the mickey mouse case.

TensorDock Marketplace is another option: https://marketplace.tensordock.com/order_list

It's unique in that you can set your own prices, it's a true spot marketplace.. I've grabbed 2x3090 for $0.02/hr before.

Probably no good for training (can be interrupted any time with zero warning ssh just drops and that's it) but for my inference usecases it lets me spot heavy compute for pennies.

Sorry where are these .50c many core servers you speak of exactly?

Outliers only begin to appear around 3B parameters (as per the original LLM.int8 paper) so unfortunately not consumer GPU in an afternoon kinda stuff to prove you've managed to suppress them.

Maybe mention that GGML is under the hood. I see you've forked the library, did you pick up the GQA changes needed for llama2-70b?

Also seems like you are picking quants on people's behalf, which ones?

Llama2 7B and 13B GGML are up and work with existing llama.cpp no changes needed! The 70B does require a llama.cpp change, but I'm sure it won't take long.

No, no you can't! When the admin borks the server that holds your data, it's just gone.

The GGLLM fork seems to be the leading falcon winner for now [1]

It comes with its own variant of the GGML sub format "ggcv1" but there's quants available on HF [2]

Although if you have a GPU I'd go with the newly released AWQ quantization instead [3] the performance is better.

(I may or may not have a mild local LLM addiction - and video cards cost more then drugs)

[1] https://github.com/cmp-nct/ggllm.cpp

[2] https://huggingface.co/TheBloke/falcon-7b-instruct-GGML

[3] https://huggingface.co/abhinavkulkarni/tiiuae-falcon-7b-inst...

This smells very badly of quantization - the extra commentary is a failure mode I observe frequently when dropping down from FP16 down to 4 bits.

This is so terrible, almost hilariously so:

https://heypi.com/s/gf72UPDDacbLwxTHEQLzg

Errors are so common with these endpoints, not having retry out of the box just makes it harder to hack on which I think is against the spirit?

What's your position on cannabis photo/review content?

Your ToS mentions Australian law so going to guess I am just as unwelcome here as everywhere else

Its a pretty terrible model, I wouldn't use this for anything at all. Vicuna 1.1 outperforms it in all my tests.

I have been burned by Fediverse so badly I will never go back.

I joined a Mastodon instance with 3k+ other users and made several hundred posts.. until one day when login failed. Turns out the admin lied about a lot of things (like backups) and an unpaid bill wiped the machine.

Migrations are not possible when original server is offline, so I lost many months of content.

Running an instance by myself is no fun, community is the point. I see no obvious solution but wont personally ever touch another ActivityPub based social media.

This is exactly how the serverless guys "get you": Low traffic is nearly free but you pay for it on the slope of the scale cost ramp.

The search plugin is bad. I have done some exploratory work [1] on specializing search agents to understand the query and result syntax and use that to structure results but I started with an explicitly structured data source to prove the concept and then kinda wandered off.

It was my hypothesis that the variety of trash returned by things like serpapi need to be massaged into something consistent and potentially run through a result retrieval and fine tuning stage to be useful to a high level agent like autogpt, but didn't make it far enough to have anything working to show.

[1] https://bitbucket.org/mike-ravkine/sara/src/master/

I've been working on a project evaluating the Python and Javascripr programing performance of open-licensed large language models [1] but don't currently have the resources to run the larger 65B ones.

Any chance you'd be able to help run some evaluations?

[1] https://github.com/the-crypt-keeper/can-ai-code

4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ

Inference is very slow right now but it works!

This looks great! Sharing your Open AI key is probably a bad idea however.

Here's a project I've been working on for the last couple of weeks that compares baseline programming performance across multiple LLMs.

ChatGPT is king, which is unsurprising.

What is surprising however is that several smaller models are quite capable: Vicuna-7B 1.1 is pretty decent at Python, while Wizard-Vicuna-13B-Uncensored beats several much larger models at JavaScript (while being the worst at Python).