Hacker News Reader

Comment by beebaween | Hacker News Reader

This is what services like Vast.ai are for - super cheap GPUs you just use as long as you need etc etc.

Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.

Of course you cannot use these machines "for confidential workflows for enterprise use", at least with AWS you know whose computer you're working with, but also keep in mind that it's really hard to steal your data as long as your data stays in memory, and you use something like mTLS to actually get it in and out of memory via E2EE. You can figure out the rest of your security model along the way, but anything sensitive (i.e. confidential) would surely fall way out of this model.

I read through the FAQ and the answer is "no", but they say it basically as "nobody really cares what your data is".

I wouldn't put anything confidential through it.

What's your workload and timeline? I'm wondering how much of that workload could be handled in-house.

Just currently exploring how custom AI workflows (e.g. text to sql, custom report generation using private data) can help given the current SOTA. Looking to develop tooling over the next 3-6 months. I'd like to see what we can come up with before dropping $50-100k on hardware.

I threw together a toy project to see if it would help me understand the basic concepts and my takeaway was that, if you can shape your input into something a dedicated classification model (e.g. YOLO for document layout analysis) can work with, you can farm each class out to the most appropriate model.

It turns out that I can run most of the appropriate models on my ancient laptop if I don't mind waiting for the complicated ones to finish. If I do mind, I can just send that part to OpenAI or similar. If your workflow can scale horizontally like my OCR pipeline crap, every box in your shop with RAM >= 16GB might be useful.

Apologies if this is all stuff you're familiar with.

i use them a lot and constantly forget to turn mine off and it just drains my credits. i really need to write a job to turn them off when it's idle for longer than 20minutes

Hmm, that looks interesting -- I might have to explore a bit. The low-end GPU pricing is pretty competitive.

Reply on news.ycombinator.com