Comment by anonacct37 | Hacker News Reader

Comment by anonacct37 | Hacker News Reader

Parent Full thread

I'll offer a different pov from many other comments.

I work for a fortune 50 basically doing web server stuff. Right now our security team would like to run some startups code synchronously as a module in our web server.

Their code could easily cost us millions off dollars (if the outage was small). I need to make sure their sdk is free of race conditions, and has proper timeouts and throttling and has proper metrics.

If your product can interrupt billions of dollars in revenue, I'm going to need some assurances and "we're really careful" doesn't count.

Of course they are free to say "no", and we can go our separate ways.

Because we're public we can't always tell vendors how much money they've cost us so when they accidentally point their prod to Dev and cost us millions of dollars all we can do is try and get a credit on this months bill.

I've services described a pretty specific use case, but there are other more generic possibilities. It's great that you have magic data science sauce for scoring customers, but when I get sued for racial profiling "I didn't know they used race as a factor" might not save my ass in court.

Just run it in a separate process with a seccomp sandbox, using pipe/socket IPC with a timeout (if it's some anti-DoS/intrusion thing, just fail allowing the request, since it's going to have false negatives anyway).

The IPC cost should not be significant compared to the rest of the web server code.

Can also ask them to do the work and provide a small open source in-process shim that sets up and talks to the sandboxed process.

I'm surprised your comment is showing up grey (downvoted). It' a perfectly reasonable solution. I'm considering adding a new "remote" plugin type to our server for running hooks in sidecar processes. It wouldn't be my first or second choice but it makes lots of sense if you have to talk to something that might have bad p99 response times.

Please feel free to reach out to me (info in profile) if you ever want to have a chat or are looking for some fun work.

It's not only the idea of "running this code might be dangerous". It's also "depending on this code to do something we need, it might suddenly drop dead and we'll still need it".

It's a question of devops- if this thing dies, or changes in negative ways, do I have the power to fix it internally? Or am I calling an outside company during business hours only to request that they please look at my problem?

That's the danger large companies face when deciding if they should depend on an external vs build it internal.

After my first skim of your post, I thought you might have had a huge misunderstanding of what a "race condition" was.

Are you going to audit all of their code changes from now into perpetuity as well? It seems like you kinda alluded to the thing that would actually be effective, which is not synchronously calling external services in high-uptime-requirement applications...

Sure, why not?

In such environments you anyway wouldn't deploy a new version from a vendor without extensive testing, so it's not something you do frequently and you can re-audit at the time. If you're paying a few million per year for a licence, then it's not an issue to spend $10k worth of manpower per upgrade to ensure that it goes properly.

I'm not sure what you mean here by external service, the solution you describe is just "build everything in house" which is possibly not reasonable.

The solution is to build in-house or to acquire the vendor for any systems you need to run synchronously inside your multi-billion-dollar revenue stream. Which doesn't seem unreasonable, and seems like a good motivation to architect that stream so third-party products don't have to be run synchronously.

If I read the comment right (a big if, I'm terrible at interpreting sentences that end in the middle with storm of dots and an implication), then he/she is implying that obviously we never considered running code asynchronously.

If so, this implication was inaccurate.

I’m not saying that it wasn’t considered, I just don’t know that a one time code audit offers sufficient downtime protection for me. If you’re worried about an external service being in the critical path then taking it out of the critical path seems like the only foolproof approach.

Not if it really is responsible for keeping billions of dollars flowing.

fantastic question. Just because the software complied with the requirements at one point doesn't mean it will in the future.

True, but I'm thinking if you can tell it was written by people who know what they're doing, odds are the company will continue to have people who know what they're doing.

In this situation at least it’s a 5 person company so the flux that they’re going to have to go through both with people and tech is gonna be pretty large. If you’re trying to gauge the future you could probably do the same / better job figuring that information out by having technical deep dives and researching their key employees and their past track record. Personally the source code is not the way I would want to gauge that information in the first place anyways, it’s a logistical nightmare and overkill.

Also to be more helpful it’s not unreasonable to ask to speak with current customers, seek proof of claims on stuff like SLAs and uptime, etc. There’s literally a million ways people do this in the real world without obtaining source code.

Reply on news.ycombinator.com