Hacker News Reader

Comment by ISL | Hacker News Reader

Think from the beginning what will be the end: "I thought your security policy was too overbearing, so I used tor."

IT departments make their choices for reasons. The key is to help them understand your use-case, and they'll probably help you through the problem in a way that might limit collateral damage.

Source: have seen firewall bypasses (with a pre-shared key) get leveraged as a way to hack an entire university lab/department.

View on HN

IT departments make choices that benefit their own needs and for their own convience, often forgetting that the entire point of their department is to make the rest of the organization more effective. Sadly, it often goes the other way.

Shadow IT is a signal that the IT organization is doing things wrong. People use shadow IT because the IT department is not doing it's job properly, serving it's customer base based on the needs they show via their actions.

For example, if you see someone like azalemeth do the things he does, it shows that the IT department needs to become responsive enough and cooperative enough to not push him to do such things in the first place. You notice he's tried to do thing the IT department standard way first, and spent considerable effort before he started his shadow IT method.

“Policy made my job slightly harder so because I know better than the netsec team who clearly has or should have unlimited time and resources to help me I will do what I want anyways, and put the organization at risk.”

Also known as “how to make the netsec team hate you 101”

I agree with you about why shadow IT exists, but most IT departments are spread so thin that expecting them to be super responsive to anything but the most critical business projects is often totally unreasonable.

Then they have to waste even more time hunting down idiots setting up Tor nodes on their internal networks.

If the IT department can't do its job because of resource constraints likely the whole organization is a failure.

If you find something like that, run…

If you can't run, do whatever makes your live better. The org is doomed anyway.

A recent example from me -- one VPN client of mine suddenly refused to connect one day for no discernible reason when they made a configuration change to their cisco vpn "concentrator" without documenting it or announcing it. Cisco AnyConnect GUI clients were fine and some magic happened behind the scenes to push the configuration change and, in typical Cisco style, avoid saying what exactly it was.

I had some esoteric monitoring machine that couldn't run anyconnect (for reasons I forget but almost certainly relating to it not having a linux arm64 client at that time) and naturally couldn't connect randomly one day with openconnect (which previously had worked perfectly). I asked what the configuration change was to prevent me having to reverse-engineer it. The response was "if you want to use unsupported clients we cannot offer any assistance [...] we are currently operating two heads down and we simply do not have the resources [...]." It took me about four or five hours to work out what change they had made, change the (122 line long) configuration file for openconnect, and then, boom, everything good again. A friendly "Hey, sorry about that -- we just $FLICKED_THIS_SWITCH because $REASON" would have been massively helpful and arguably take less words than their original response. (Edit: For context, approximately 10-20k people use that specific VPN. And their team is such that losing two members of staff temporarily is a major inconvenience.)

I totally understand it from the other side. IT departments have everything from state-sponsored ransomware attacks to important people loudly going "why doesn't the printer work any more". It's a different set of skills to being a C-junkie, a programming wizard, or, in my case, a young academic with one big grant and three PhD students trying to both do work, publish work, and get money to do more work where "work" is poorly defined and highly flexible. Over time I've noticed universities get far more corporate and many academics absolutely hate this, of which I am one. The "we control the network, bug off" may be technically true but at times it does feel a bit like an imposition of some sort of academic freedom, to be honest. At the very least, it's a nice little "dog egg" to find added to the pile of administrative crap to do for that day.

What you’ve just described is most post secondary institutions, public utilities, government, etc.

I'm working in an organization where we have one laptop from work, and another laptop to do work on. Because the one sized fits all IT policy doesn't work for our org, but it's forced on us because of the IP security needs of another parallel org.

We went from an organization moving towards BYOD, to, now the exact opposite.

> because I know better than the netsec team

For anyone who's been around the block a few times, there's a good chance this is true.

Most organizations' netsec teams are too busy throwing money at vendors to keep up.

“Because I think I might know better I will act in a disrespectful way, and make someone else’s job harder instead of working with them to solve the problem”

You’re not the one who’s phone is going to ring at 3am on Saturday when that Tor node gets compromised. You’re not the one who has to manage the security incident. You’re not the one who has to explain why your security controls and policy did not prevent this from happening. Nor are you the one who has to clean up the damage if something goes badly.

I also think you’re vastly overestimating the average developers awareness of security issues. Perhaps you are very well versed in this topic, but many developers are utterly clueless, even when it comes to basic application security practices.

I'm curious how you think an SSH service exposed over TOR is going to create a security issue? SSH is exposed all over the public internet.

The idea of allowing any kind of inbound connection into a secured network (other than to/via its DMZs) is anathema.

I don't even disagree with the logic, but the BigCorp Infosec Team heavy-handed approach to working with developers invites the developers to produce creative circumventions.

It bypasses all proxies and interception, and hides all of the traffic contained in the tunnel. This means no traffic logging of the tunneled traffic, no IPS/IDS in front of the SSH service, and no visibility into the SSH traffic itself. If the box with the SSH service isn’t in a DMZ it also compromises network segmentation.

The problem isn’t SSH over TOR being insecure. It is sidestepping all of the security controls in place at your org and not talking to the netsec folks first.

Honestly I would be amazed if any competent netsec folks would even allow TOR outbound by default. I certainly wouldn’t allow it by default in an enterprise environment.

I tried doing that, and largely succeeded, but the specific area of the university in question will not have a bastion SSH host anywhere on their network. They will not allow SSH access in at all. They will however allow SSH access to other parts of the university, with different people in charge, which explicitly do allow an SSH bastion host to exist (and provide several for that purpose). So, the net result is that they've effectively out-sourced the control and responsibility of their environment to someone else.

Normally this is fine, but my job involves programming and controlling large, expensive, and strangely fragile lab equipment. There's a resilience problem, and it's got to the point where others have suggested putting a GSM modem on a pci-e card inside some of the boxes in question, as the relevant IT department decides on a whim to block ports with no warning or justification. Some manufacturers of the devices in question do this as standard if you have a support contract. Trying to complain results in responses like "you have been used to doing things one way and this change now prevents you from working as before."

I completely accept that this is a political problem and best solved as one, but ultimately SSH is an industry standard for a reason -- it's secure, and it's flexible. The machines in question are valuable, prone to breaking in the middle of the night, and we are an international bunch who cannot always connect from a well-defined ipv4 address, or from the university's VPN. (The latter is blocked by the IT department automatically, as it has too large a pool of potential users). The thing I find most frustrating is that this sort of political decision creates days worth of work instantaneously, for little benefit. All of the actually confidential or sensitive information is held in a completely separate network at any rate...

To quote Dr. Manhattan, "Without condemning, or condoning, I understand".

I am in network security. I have stopped shadow IT, and been a part of it.

Your situation seems so ungodly stupid and anathema to the point of IT, that the remaining courses of action should be the following.

Thoroughly document via email your attempts at explaining requirements to Netsec, to document in writing their objections, to do your best with what they provide you... and WHEN things catastrophically break, point the finger at them and thoroughly document how if you had the proper, industry-standard tooling, you could have prevented the loss of research/time/money.

THIS. Don't paper over the issues with shadow IT. Make them painfully obvious to the point where IT has to do something or answer to it. Otherwise it will not change.

I've given teams the option to turn off their pagers when this sort of thing happens with the justification that they can't fix it anyway. And then documented the crap out of why they can't fix it so when someone asks I can point to existing policy. It's very effective if done right.

"What did you accomplish during your time with X research group?"

"Nothing since all our equipment broke, but we documented how it was all IT's fault. You shoulda seen the looks on their faces when we called them out on it to the dean!"

It seems weird to dump on IT when they’re a department responding to the incentive structure they’re placed under like everyone else. You going to the Dean/someone with actual authority to get top down approval for IT to give you what you want is basically how IT operates in large orgs. I have nigh infinite technical power but in return I am bound politically by polities that I’m explicitly not allowed to have any authority over (i.e. I can’t approve my own policy exceptions). I want to give you literally anything you ask for. As long as my ass is covered it literally doesn’t matter at all to me. When I worked Uni IT if someone wanted something we couldn’t give them because $dumb_reason weren’t in a position to have that fight with the higher-ups on their behalf. It doesn’t mean much coming from us and since it’s not impacting our work it falls on deaf ears.

From your tone you make it seem like you were proud to waste everyone’s time and money when one single meeting with the Dean and the CIO/Director of IT when the problem happened would have opened every door for you.

The parent comment was literally suggesting that GP should have allowed the equipment to fail so that IT could be blamed. I didn't get the impression GP's situation was the result of not bothering to sit down and talk to a higher-up.

Everyone else is responding to incentive structures too, it's no less legitimate for lab workers to circumvent IT due to their incentives than it is for IT workers to be unhelpful due it their incentives.

This is a diverging motivations issue.

Many people are not in a stable career such that they can hang around and do upper management's job for them by "expensively failing so as to demonstrate IT's failures".

Academics and PHD students in particular live from grant to grant. They can't afford to waste grant money "to make a point that IT doesn't work." Reputations - and by extension careers - can be made and unmade with such stuff.

Aside, I think the academic life being so fragile is ALSO silly but that is another story.

I was assuming that the OP was someone who worked in a department IT role, that had to abide by more centralized IT security requirements. You're right that someone who is more transient or less full-time has less motivation to make a long-term point to the administration.

Quite. IT people in universities have jobs for life. The overwhelming majority of academics -- often including many professors -- do not.

You might try Zerotier or Tailscale running either natively or using an RPi as a bridge. Assuming your IT rules don't forbid it, both should be fairly resilient to simple/random port blocking. They're actually used by a lot of enterprises to provide secure p2p networks with automatic port punching and nat traversal.

The problem in my experience is not that the security policy is too overbearing, but rather that the security policy is too rigid and designed with assumptions that are false. A common policy for example is that port 22 must always be closed. One can use a hardware secured two token authentication over ssh, and still the policy is that the port must be closed and that is that. That the policy allow remote desktop with just a password is completely irrelevant because the policy doesn't forbid that.

I have tried so many times to help people understand security and the purpose of a security policy when it is designed correctly, but it doesn't work. The policy exist so people don't need to think, not to make people understand why it exist and what use-cases should be given exceptions.

Often times these policies are driven by industry compliance. Exceptions have to be documented, and depend on the compliance regime, may carry liability. Lastly when exceptions are made the user often doesn’t know what they signed up for, and it ends up holding the bag for a breech.

It’s usually better to not make an exception.

> IT departments make their choices for reasons.

In a perfect world, yes. But I've worked with/at places where ineptitude is rampant, and any attempts of understanding their reasoning is seen as insubordination.

A simulated conversation with IT:

"Hey, IT department...I was wondering..."

"No."

Lucky me.

Our IT department goes out the of their way to help us stay sane and productive

- they're making sure most of us can continue to use our favourite Linux distro (I think most Debian/Ubuntu, Fedora and Arch is supported)

- make sure VPN etc works on Linux even if it is not officially supported

- taking time to sit down and debug hard problems (weird issues with WSL2 on one particular Windows laptop) instead of just blaming us engineers

Especially when, since it's Tor, potential attackers cannot be traced

This.

This makes no sense. Onion services don't hide the source, they hide the destination -- a destination that, in this threat model, you run. If the client connecting wants to hide their source, they can use Tor, a VPN, an existing botnet, etc.; whether you're running SSH over an onion service or with vanilla exposed IPs/DNS is immaterial.

Reply on news.ycombinator.com