Hacker News Reader

Comment by elktea | Hacker News Reader

Love the look of all the tools netflix have at their disposal

A quick to use remote profiler does wonders to an org. By fast to use I mean it takes me 10 seconds to kick off a remote profiling scrape. Someone entrepreneurial please make a purchase-able service out of that, I'll be feeling lost once I quit my corp (not Netflix). Stuff is 100x more useful than remote debugging which ultimately almost nobody uses in practice.

Shameless plug: We just launched Prodfiler last week -- it's continuous in-production profiling based on eBPF, handles C/C++/Go/Rust/JVM/Python/Perl/Ruby (with NodeJS and .net in the pipeline), and it does not require symbols on the machine, nor recompiles with framepointer, nor special JVM flags.

https://prodfiler.com

:-)

Datadog has continuous profiling that is pretty useful. I'm pretty sure they follow Brendan religiously.. I hope they release something like flamescope next.

And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts.

Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it.

I had a lot more respect for the ops people there than I had for my fellow SWE's.

Most issues we hit are solved by the monitoring GUIs (Atlas, FlameCommander, PerfDash, S3 log collection, etc.), but some aren't, and there's a ton more information available from the CLI tools. Imagine losing your five nines because no one can SSH on and try a few ad hoc tools.

But yes, CLI can be dangerous, even for observability. E.g., people running strace(1) on production apps and causing outages due to the strace overhead (I wrote a prior post about that). You need to understand the risks and overheads of all tools. It's why I have a "pull no punches" policy when writing eBPF tool man pages [0]: If the overhead can be bad, it should say so clearly.

[0] https://github.com/iovisor/bcc/blob/master/CONTRIBUTING-SCRI...

> log the configuration

That's a change, inherently much more risky than the "cat /.../env" the colleague wanted to do.

Also the change might well cause the problem to go away, and now you know nothing instead.

The principle is good, but it sounds like it has taken on a life on its own. The ops guy could also have executed the cat command on the spot with the right privileges. Sure, it's gatekeeping, but so is the four eyes principle, and a little of that gatekeeping can be necessary to keep those nines rolling.

> Ops team told him how to do his job - log the configuration - instead of give him access

"do your job" is a really rude way to refer to debugging by logging instead of debugging interactively.

I suspect this usage is an awkward phrasing of “told him how to accomplish the result he wanted”; even though in common usage it implies a lack of overall competence, here I think it just implied a lack of a specific recipe/technique.

Doesn’t seem so, as GP specifically said this caused them to lose respect for their colleagues.

That’s a possible interpretation of their comment, but not the only interpretation and certainly not something they specifically said.

(Another interpretation is that the author previously held Ops in higher regard than SWE and that this event did not change that.)

I imagine if overzealous rule enforcement is the cause of respect that the OP would've always had more respect for Ops :D

Sounds weird , logging env vars would be even more risk prone to leaking credentials, than just letting an employee ssh into the machine and see the env vars.

Also like others noted here , If one guy sshing into one machine can screw your seven nine uptimes.

Then you never had a seven nine uptime (you’re company is probably just gambling on that uptime metric until some component fails)

You had a mandate for 5-7 nines uptime and you couldn't tolerate a problem on a single node?

yup, that sounds strange. if the environment is that advanced, then there shouldn't be a problem of taking the node in question out of the service pool and let the devs loose on it (assuming there's no info that they aren't supposed to see). when they're done, just recycle it.

Where did they say it was a single node issue?

i think they mean, in case he mess up a single node, it should still be fine

I see, that makes more sense, thanks for the clarification.

I’d still rather not have to debug arbitrary mutations to the env or file system in a production container though.

It seems to me that shelling into a prod container/VM is discouraged not because you might cause it to fail, but because you might produce undefined behavior while claiming it is healthy (more like Byzantine faults).

For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail. Debugging this kind of issue is a nightmare.

Not to mention that developers can often run arbitrary SQL from a prod shell when the app is backed by a DB.

> For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail.

Or far worse, not fail.

I disagree.

There are always bugs that will happen only on the prod machines. Sure, they are rare, but they exist.

> just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration

Well, that's not risk free neither. That needs a code change and you risk exposing secrets, or logging more than you should. (Though I agree with the general idea of you shouldn't be SSHing if you can do it another way)

Side node: Why do we have to talk about these things in such highly charged terms in our profession? It sounds like all that happened here is that one person made a reasonable request, and another person gave a reasonable explanation of why the request couldn't be granted.

Because people establish protective totalitarian control over their corporate turf because budgeting, it become full of dogma (the gods said thou shalt not ssh and we said amen).

Dogma results in religious wars

I know at least one place where they do allow ssh as a last resort, but the VMs that gets ssh'd to get flagged to be recycled automatically.

So you can log in and debug, but once you're done, the VM is replaced with a clean one.

If messing up a single instance is going to take your site down, you are nowhere near seven 9's.

This is the sort of fanboyism that I usually avoid when working. It is perfectly reasonable to SSH into a node for debugging. In fact I do it very often in a heavily automated infrastructure as code environment.

Not to mention that if you're using Ansible to manage machines, it depends on SSH to do its job.

Exactly. Any time this topic comes up the people who are arguing for no SSH the ones never operated a production environment successfully and they are busy chasing ghosts.

Reply on news.ycombinator.com