Show HN: Exafunction NLP – Run models like GPT-J for 6x cheaper than OpenAI | Hacker News Reader

Show HN: Exafunction NLP – Run models like GPT-J for 6x cheaper than OpenAI

exafunction.com

45 pointsvarunkmohan19 comments

Hi HN, I’m Varun, cofounder of Exafunction. We’re excited about all the use cases of large NLP models like text generation and language comprehension, but found the existing offerings to be quite expensive. Our goal is to serve these models in a cost effective way at scale.

The first model we’re starting with is GPT-J, the recently released open source large language model by EleutherAI. It’s comparable to OpenAI’s GPT-3 Curie in performance. With lots of optimizations, we are able to serve the model for 6x cheaper per token compared to OpenAI.

We have a simple HTTPS API that you can try out for free. Here’s a link with the details - https://www.exafunction.com/nlp-api

We’re also looking into supporting custom and fine-tuned models since it’s required to get good performance for a lot of applications. Especially for smaller models like BERT, we’re excited about the cost savings we can deliver for users who use many different models. In this case, we’re 100 - 1000x cheaper than others offering similar inference APIs like Hugging Face.

We hope this is interesting to you all and please email me at varun@exafunction.com if there’s a model you’d like to see supported.

Hi!

Please could you explain some use cases?

OpenAI is very strict with GPT-3. Say a company wanted to use Exafunction for email interactions or Discord interactions or even SMS interactions.

What would be the setup? How would that work with Exafunction?

Thank YOU!

Unlike OpenAI, which is serving a closed-source model, anyone can validate for themselves (even without using our service) whether open-source models like GPT-J are suitable for a particular application. We'd like to work with the open source community to make this kind of technology more accessible, essentially as an infrastructure provider.

If you have some specific use case in mind feel free to shoot us an email at hello@exafunction.com

https://nlpcloud.io/effectively-using-gpt-j-gpt-neo-gpt-3-al...

Here's a wonderfully written breakdown of some functionality that can be achieved with gpt-j and gpt-neo - it's not ExaFunction specific, but the language models are the same.

The exafunction api is fairly simple and powerful. Kudos!

But how fast? I see other companies advertising 1.5 second response times for GPT-J, but a now assume that’s average per token, as for, say, a 200 word prompt response times can be well over a minute during heavy use periods like weekends when everyone is hitting their side projects.

For the public API, we should be under 100ms per token but don't have a strict guarantee. If you have a strict SLA, you can talk to us and we can get it to as low as 20ms per token at high load.

Have you considered adding support for CodeT5? I believe they're supported in Huggingface as well. https://github.com/salesforce/CodeT5

Great suggestion! We will look into it.

Wow! How are you able to achieve the cost reductions? Is it different hardware, software optimizations, or both? Also does this suggest that OpenAI is charging 6x markups on their models... >:(

Mainly efficient usage of cloud hardware right now, with some software optimizations. Looking at more cost-effective hardware is on our roadmap too.

Awesome. I am testing HuggingFace's API and evaluating if I should pay for a dev account. I will evaluate Exafunction also, it looks interesting. Specifically, I'm interested in using the multilingual LaBSE model. Do you support this model?

We currently do not support this model but are rolling out support for more models. Please add your request to the waitlist and we'll try to support your model as soon as possible!

Awesome, I definitely will do! Congratulations for the project, it looks promising.

This is really remarkable! How hard do you think it will be to support new models, i.e. does the tooling you’ve built generalize to you being able to serve other large scale models easily?

Wow this is really cool! How did you guys get such big speedups?

There's some secret sauce behind it, but mostly just using relatively inexpensive cloud inference hardware very effectively. It turns out most of the common NLP frameworks leave a good deal of performance on the table, not to mention the importance of minimizing cloud costs through general methods like using spot instances.

Cool! 7500 words free per month…does anyone have any interesting ideas about what you could do with this?

I have a couple of NLP side projects that I'm working on and I'll definitely try using this!

amazing cost ... will try it out for some personal projects

Reply on news.ycombinator.com