atylerrice

You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)

This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.

Well the batch size also helps with making training more stable. So a larger batch size computes the gradient across more examples so it ends up being more stable than only using a batch size of day one. A batch size of one is only fixing the error on one example rather than many.

a larger Batch size can help keep the gpu fed but usually that’s not a problem with these larger models.

I don’t usually comment, but aws is actually rather expensive and i’ve hit this annoying quota problem as well. Been back and forth and still hasn’t been raised. I can also recommend coreweave banana.dev and pipeline.ai as great alternatives. This service looks awesome good luck with the launch!

I’ve seen this a lot. I think sometimes in my experience it comes down to what is work?

As a developer I would classify a lot of the meetings I’ve had at past companies where it takes up 40% of my time as not really working. Sometimes these meetings push over the hours for the week but weren’t really productive in any meaningful way.

I totally agree but I’m excited for some recent initiatives to fix this. https://www.researchallofus.org/ This research project is open even to at home scientists although I think some of the genomic data may reasonably not be as open as other parts.

I built Hatchedwith.com mostly because I wanted to mess around with language models. I can’t seem to find any customers even though I got two from a fake landing page but realistically it’s not the product I advertised in the beginning I’m going to lower the pricing to extremely cheap and call it done unless google starts picking up on the tens of thousands of pages.