Hacker News Reader

Personally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, you can get down to 2), but it would still require a higher-end Mac. Highly recommend Together, it runs quite quickly and is $0.9/million tokens

View on HN

What’s the advantage of Together? The price is about the price of GPT 3.5 Turbo ($1/mil tokens is $0.001/thousand tokens), which has the advantage of wide ecosystem and support.

are there any docs on setting up togetherAI with continue.dev? would be interested in checking that out as an alternative to OpenAI for experimenting with larger models that won't run/run well on a m1 max.

Definitely, here is a brief reference page for the Together provider: https://continue.dev/docs/reference/Model%20Providers/togeth..., and a higher-level explanation of model configuration here: https://continue.dev/docs/model-setup/overview

Reply on news.ycombinator.com