Hacker News Reader

Comment by bix6 | Hacker News Reader

Thank you! Is there one you’d recommend in particular when working with sensitive data?

You might want to consider a local setup? Personally I do model + llama.cpp + Pi + tmux, that's as barebones as you can get and everything is great. I also use opencode for bigger task, I could probably hook that to a local model but I'm not sure (still new to this). But if you work with sensitive data you need a local setup purely.

I am interested in a local setup but whenever I read about it, it seems the money is better spent on cloud inference since even with thousands of dollars of my own hardware I wouldn’t have frontier like capability? I am bad at choosing models though (I just throw everything at Opus atm).

Well yes you won't have frontier capability but that doesn't mean you'd have nothing, I'm not sure what you have but I know there are tools that let you see what models (and their quant variants) you can safely run, I'd look into that first and see what options you have and what you're willing to delegate to a smaller model vs use opus for. Maybe opus to plan vs smaller models for focused implementation, or what works for you workflow and needs.

I think this would depend upon the level of sensitive data.

For OpenRouter, you can setup an API key and limit it to only models that claim to not train on data, but that is just a claim. You can then use trust to judge which providers will honor that claim.

But if the data is really sensitive, you might want either a local model or a business subscription with some big name in the US that legally promises no data training.

So are we talking some app idea you are playing around with, or files filled with PHI/PII that you have legal mandates to safeguard? If the latter, I would stick to only provider with enterprise agreements to not store/train on the data. Even the ones who promise no training are likely storing the data for monitoring for abuse or such short term.

For anything sensitive I have Claude business (team?) which (claims) no training but I would like to find an alternative for them. I’m just really tired of supporting Anthropic each month (same feelings for all the major providers).

I’m biased (my wife works there) but Fireworks or another US-based inference provider is going to be a better fit than OpenRouter. Fireworks has zero data retention by default[1] and has tools to make it easy to run your existing harness against open models [2]

OpenRouter isn’t a provider, they route to other providers, so you would need to specify which ones you’re comfortable with anyway.

[1]:https://docs.fireworks.ai/guides/security_compliance/data_ha...

[2]:https://fireworks.ai/nexus

Fantastic thank you!

Openrouter lets you pick your provider and see their policy re retention and training. You can pick US providers e.g. digitalocean that don’t retain or train on your data. Openrouter is also now owned by stripe, so I don’t see any reason why using them with a trusted provider is riskier than using the big two (especially in light of some of the whole Navier Stokes thing)

If it’s really sensitive then don’t use a cloud provider.

Depending on what your threat model is, and how much you trust OpenRouter and their upstream providers, they offer zero data retention (googleable: ZDR) APIs which come at a higher cost. You can find other zdr providers as well.

Reply on news.ycombinator.com