Hacker News Reader

Comment by azalemeth | Hacker News Reader

In many ways I think this blog post really makes quite compelling arguments and honestly opened my eyes a bit.

One (perhaps mad) idea for more secure access to a machine deep behind many levels of NAT where you, the sysadmin, have lawful access but are fed up with having to have a 12 KB ~/.ssh/config file in order to access it because of your university's overbearing IT department^W^W^W^W network topology, would be to "just" run an onionsite with onion services authentication [1], preventing it being publicly accessed without the pre-shared key. If your onion service just redirects to ssh (presumably with certificate-only auth) I can't help but think that this is almost an example of security by obscurity done right.

[1] https://support.torproject.org/en-US/onionservices/client-au...

View on HN

Think from the beginning what will be the end: "I thought your security policy was too overbearing, so I used tor."

IT departments make their choices for reasons. The key is to help them understand your use-case, and they'll probably help you through the problem in a way that might limit collateral damage.

Source: have seen firewall bypasses (with a pre-shared key) get leveraged as a way to hack an entire university lab/department.

I tried doing that, and largely succeeded, but the specific area of the university in question will not have a bastion SSH host anywhere on their network. They will not allow SSH access in at all. They will however allow SSH access to other parts of the university, with different people in charge, which explicitly do allow an SSH bastion host to exist (and provide several for that purpose). So, the net result is that they've effectively out-sourced the control and responsibility of their environment to someone else.

Normally this is fine, but my job involves programming and controlling large, expensive, and strangely fragile lab equipment. There's a resilience problem, and it's got to the point where others have suggested putting a GSM modem on a pci-e card inside some of the boxes in question, as the relevant IT department decides on a whim to block ports with no warning or justification. Some manufacturers of the devices in question do this as standard if you have a support contract. Trying to complain results in responses like "you have been used to doing things one way and this change now prevents you from working as before."

I completely accept that this is a political problem and best solved as one, but ultimately SSH is an industry standard for a reason -- it's secure, and it's flexible. The machines in question are valuable, prone to breaking in the middle of the night, and we are an international bunch who cannot always connect from a well-defined ipv4 address, or from the university's VPN. (The latter is blocked by the IT department automatically, as it has too large a pool of potential users). The thing I find most frustrating is that this sort of political decision creates days worth of work instantaneously, for little benefit. All of the actually confidential or sensitive information is held in a completely separate network at any rate...

To quote Dr. Manhattan, "Without condemning, or condoning, I understand".

I am in network security. I have stopped shadow IT, and been a part of it.

Your situation seems so ungodly stupid and anathema to the point of IT, that the remaining courses of action should be the following.

Thoroughly document via email your attempts at explaining requirements to Netsec, to document in writing their objections, to do your best with what they provide you... and WHEN things catastrophically break, point the finger at them and thoroughly document how if you had the proper, industry-standard tooling, you could have prevented the loss of research/time/money.

THIS. Don't paper over the issues with shadow IT. Make them painfully obvious to the point where IT has to do something or answer to it. Otherwise it will not change.

I've given teams the option to turn off their pagers when this sort of thing happens with the justification that they can't fix it anyway. And then documented the crap out of why they can't fix it so when someone asks I can point to existing policy. It's very effective if done right.

"What did you accomplish during your time with X research group?"

"Nothing since all our equipment broke, but we documented how it was all IT's fault. You shoulda seen the looks on their faces when we called them out on it to the dean!"

It seems weird to dump on IT when they’re a department responding to the incentive structure they’re placed under like everyone else. You going to the Dean/someone with actual authority to get top down approval for IT to give you what you want is basically how IT operates in large orgs. I have nigh infinite technical power but in return I am bound politically by polities that I’m explicitly not allowed to have any authority over (i.e. I can’t approve my own policy exceptions). I want to give you literally anything you ask for. As long as my ass is covered it literally doesn’t matter at all to me. When I worked Uni IT if someone wanted something we couldn’t give them because $dumb_reason weren’t in a position to have that fight with the higher-ups on their behalf. It doesn’t mean much coming from us and since it’s not impacting our work it falls on deaf ears.

From your tone you make it seem like you were proud to waste everyone’s time and money when one single meeting with the Dean and the CIO/Director of IT when the problem happened would have opened every door for you.

The parent comment was literally suggesting that GP should have allowed the equipment to fail so that IT could be blamed. I didn't get the impression GP's situation was the result of not bothering to sit down and talk to a higher-up.

Everyone else is responding to incentive structures too, it's no less legitimate for lab workers to circumvent IT due to their incentives than it is for IT workers to be unhelpful due it their incentives.

This is a diverging motivations issue.

Many people are not in a stable career such that they can hang around and do upper management's job for them by "expensively failing so as to demonstrate IT's failures".

Academics and PHD students in particular live from grant to grant. They can't afford to waste grant money "to make a point that IT doesn't work." Reputations - and by extension careers - can be made and unmade with such stuff.

Aside, I think the academic life being so fragile is ALSO silly but that is another story.

I was assuming that the OP was someone who worked in a department IT role, that had to abide by more centralized IT security requirements. You're right that someone who is more transient or less full-time has less motivation to make a long-term point to the administration.

Quite. IT people in universities have jobs for life. The overwhelming majority of academics -- often including many professors -- do not.

You might try Zerotier or Tailscale running either natively or using an RPi as a bridge. Assuming your IT rules don't forbid it, both should be fairly resilient to simple/random port blocking. They're actually used by a lot of enterprises to provide secure p2p networks with automatic port punching and nat traversal.

> IT departments make their choices for reasons.

In a perfect world, yes. But I've worked with/at places where ineptitude is rampant, and any attempts of understanding their reasoning is seen as insubordination.

IT departments make choices that benefit their own needs and for their own convience, often forgetting that the entire point of their department is to make the rest of the organization more effective. Sadly, it often goes the other way.

Shadow IT is a signal that the IT organization is doing things wrong. People use shadow IT because the IT department is not doing it's job properly, serving it's customer base based on the needs they show via their actions.

For example, if you see someone like azalemeth do the things he does, it shows that the IT department needs to become responsive enough and cooperative enough to not push him to do such things in the first place. You notice he's tried to do thing the IT department standard way first, and spent considerable effort before he started his shadow IT method.

“Policy made my job slightly harder so because I know better than the netsec team who clearly has or should have unlimited time and resources to help me I will do what I want anyways, and put the organization at risk.”

Also known as “how to make the netsec team hate you 101”

I agree with you about why shadow IT exists, but most IT departments are spread so thin that expecting them to be super responsive to anything but the most critical business projects is often totally unreasonable.

Then they have to waste even more time hunting down idiots setting up Tor nodes on their internal networks.

If the IT department can't do its job because of resource constraints likely the whole organization is a failure.

If you find something like that, run…

If you can't run, do whatever makes your live better. The org is doomed anyway.

A recent example from me -- one VPN client of mine suddenly refused to connect one day for no discernible reason when they made a configuration change to their cisco vpn "concentrator" without documenting it or announcing it. Cisco AnyConnect GUI clients were fine and some magic happened behind the scenes to push the configuration change and, in typical Cisco style, avoid saying what exactly it was.

I had some esoteric monitoring machine that couldn't run anyconnect (for reasons I forget but almost certainly relating to it not having a linux arm64 client at that time) and naturally couldn't connect randomly one day with openconnect (which previously had worked perfectly). I asked what the configuration change was to prevent me having to reverse-engineer it. The response was "if you want to use unsupported clients we cannot offer any assistance [...] we are currently operating two heads down and we simply do not have the resources [...]." It took me about four or five hours to work out what change they had made, change the (122 line long) configuration file for openconnect, and then, boom, everything good again. A friendly "Hey, sorry about that -- we just $FLICKED_THIS_SWITCH because $REASON" would have been massively helpful and arguably take less words than their original response. (Edit: For context, approximately 10-20k people use that specific VPN. And their team is such that losing two members of staff temporarily is a major inconvenience.)

I totally understand it from the other side. IT departments have everything from state-sponsored ransomware attacks to important people loudly going "why doesn't the printer work any more". It's a different set of skills to being a C-junkie, a programming wizard, or, in my case, a young academic with one big grant and three PhD students trying to both do work, publish work, and get money to do more work where "work" is poorly defined and highly flexible. Over time I've noticed universities get far more corporate and many academics absolutely hate this, of which I am one. The "we control the network, bug off" may be technically true but at times it does feel a bit like an imposition of some sort of academic freedom, to be honest. At the very least, it's a nice little "dog egg" to find added to the pile of administrative crap to do for that day.

What you’ve just described is most post secondary institutions, public utilities, government, etc.

I'm working in an organization where we have one laptop from work, and another laptop to do work on. Because the one sized fits all IT policy doesn't work for our org, but it's forced on us because of the IP security needs of another parallel org.

We went from an organization moving towards BYOD, to, now the exact opposite.

> because I know better than the netsec team

For anyone who's been around the block a few times, there's a good chance this is true.

Most organizations' netsec teams are too busy throwing money at vendors to keep up.

“Because I think I might know better I will act in a disrespectful way, and make someone else’s job harder instead of working with them to solve the problem”

You’re not the one who’s phone is going to ring at 3am on Saturday when that Tor node gets compromised. You’re not the one who has to manage the security incident. You’re not the one who has to explain why your security controls and policy did not prevent this from happening. Nor are you the one who has to clean up the damage if something goes badly.

I also think you’re vastly overestimating the average developers awareness of security issues. Perhaps you are very well versed in this topic, but many developers are utterly clueless, even when it comes to basic application security practices.

I'm curious how you think an SSH service exposed over TOR is going to create a security issue? SSH is exposed all over the public internet.

The idea of allowing any kind of inbound connection into a secured network (other than to/via its DMZs) is anathema.

I don't even disagree with the logic, but the BigCorp Infosec Team heavy-handed approach to working with developers invites the developers to produce creative circumventions.

It bypasses all proxies and interception, and hides all of the traffic contained in the tunnel. This means no traffic logging of the tunneled traffic, no IPS/IDS in front of the SSH service, and no visibility into the SSH traffic itself. If the box with the SSH service isn’t in a DMZ it also compromises network segmentation.

The problem isn’t SSH over TOR being insecure. It is sidestepping all of the security controls in place at your org and not talking to the netsec folks first.

Honestly I would be amazed if any competent netsec folks would even allow TOR outbound by default. I certainly wouldn’t allow it by default in an enterprise environment.

A simulated conversation with IT:

"Hey, IT department...I was wondering..."

"No."

Lucky me.

Our IT department goes out the of their way to help us stay sane and productive

- they're making sure most of us can continue to use our favourite Linux distro (I think most Debian/Ubuntu, Fedora and Arch is supported)

- make sure VPN etc works on Linux even if it is not officially supported

- taking time to sit down and debug hard problems (weird issues with WSL2 on one particular Windows laptop) instead of just blaming us engineers

Especially when, since it's Tor, potential attackers cannot be traced

This.

This makes no sense. Onion services don't hide the source, they hide the destination -- a destination that, in this threat model, you run. If the client connecting wants to hide their source, they can use Tor, a VPN, an existing botnet, etc.; whether you're running SSH over an onion service or with vanilla exposed IPs/DNS is immaterial.

The problem in my experience is not that the security policy is too overbearing, but rather that the security policy is too rigid and designed with assumptions that are false. A common policy for example is that port 22 must always be closed. One can use a hardware secured two token authentication over ssh, and still the policy is that the port must be closed and that is that. That the policy allow remote desktop with just a password is completely irrelevant because the policy doesn't forbid that.

I have tried so many times to help people understand security and the purpose of a security policy when it is designed correctly, but it doesn't work. The policy exist so people don't need to think, not to make people understand why it exist and what use-cases should be given exceptions.

Often times these policies are driven by industry compliance. Exceptions have to be documented, and depend on the compliance regime, may carry liability. Lastly when exceptions are made the user often doesn’t know what they signed up for, and it ends up holding the bag for a breech.

It’s usually better to not make an exception.

Yup, and it's easy to make server and client side tooling use Tor to make this mostly transparent. Latency/bandwidth isn't _that_ bad when communicating with an onion service. And it can be even faster if server anonymity isn't a goal (server set HiddenServiceSingleHopMode and HiddenServiceNonAnonymousMode and create ephemerial onion service with NonAnonymous).

I use Tor plenty to self-host services from my house that are reachable anywhere (and often have a web interface I can access via Orbot). No hole-punching necessary.

Could you share more about your setup?

Sure. I wrote https://github.com/cretz/bine (though I admittedly don't work on it much these days). I just have a few-line daemon that starts an HTTP (or gRPC or whatever) server on ephemeral onion service. Then I use that onion ID to access it (via TorBrowser or Orbot or a client built with the same library).

Thank you!

at our lab the tor traffic would be noticed by the cyber security group's ids and all traffic from your host would start dropping at the border so fast your head would spin. you'd get an unpleasant phone call or visit to your office and be warned never to try side stepping the bastion ssh hosts that log all the things ever again.

Obviously, you should plan around this by gathering all the MAC addresses of every machine in the office, and then have your machine spoof through them in rotation. /s

It makes me sad every time I think about it, but Aaron Swartz did this during his saga. Well, sort of: he incremented the MAC address by 1.

Point being, it's not foolproof. If some clever undergrad is thinking about dodging the suits, win by fooling them, not by fighting them.

If you do insist on fighting, though, start at https://www.whonix.org/wiki/Mental_Model and then read the entire Whonix wiki https://www.whonix.org/wiki/Documentation. It's what I used when I was serious about dodging the cartels, and that knowledge will protect you as much as anything will.

(You'll hopefully conclude that the protection is too brittle to risk your life, as I did.)

building a new computer. want to be able to trust it 100% for at least a moment. i can't figure out how to "buy" a trusted copy of any linux and don't have any machines i have 100% trust in (who does), so can't burn it. current plan is to buy a chromebook solely for the purpose of downloading and burning ubuntu. alternatively, buy MSWindows, install on the new machine, burn, and then replace

but this mental exercise has convinced me that security is almost impossible in this day and age

One thing that helps a lot in this situation is to plan based on threat model. There’s no such thing as 100% trust, but you can have a computer which is safe for e.g. <thing>. It’s pretty crucial to pick one or two specific <thing>s and focus only on those.

If you just want to browse the darknet and see what the markets are like, for example, Tor on your current computers is fine.

If you’re wanting to make a purchase and you’re worried that your existing computers will narc on you, your plan of buy laptop + use ubuntu is A+.

If you want a computer to store information on, Edward Snowden style, you’ll need to take increasingly serious steps. Use tails as a baseline. (Note: I’ve been out of the game since 2016, so take this with salt.)

If you’re literally dodging the NSA, you need to put on a full face mask in winter, plan a route to a store you’ve recon’d, buy clothes with cash from goodwill, carry them in a trash bag as you walk out of your neighborhood, sneak in between two houses in the dead of night and put the outfit on + mask, walk to a taxi, have it take you near (but not to) the electronics store, buy yourself a burner phone + a few USB wifi dongles + anything else you want completely unlinkable to you (you’re on cameras), pay for all of it while getting some strange and worried looks that you’re going to rob something, then do the entire process in reverse until you’re back at your house with your untraceable electronics.

I did all that, and even then I was likely making some small mistake that would’ve blown everything.

Yet the city wide surveillance drones (god eye) will still have a nice little record of you that they can ID you with. And you sneaking around in the middle of the night putting on masks will probably get you in serious trouble. It never really occurs to you when you’re doing this sort of thing to stop and consider whether you’re just doing crazy things. (It’s tempting to believe the answer is “no,” especially the more you want to believe it.)

Suffice to say, threat modeling is key, and it’s worth thinking carefully about what exactly you want to accomplish.

> If you’re literally dodging the NSA, you need to...

Or just make friends with an developing-world advance-fee scammer, and then pay them to have one of their cash mules buy and send you (that is, an empty house somewhere in your city) a laptop.

That's an interesting idea I hadn't considered. But it involves a lot of the same problems: you need to get from where you are to where the laptop is, and back, without popping up on any sensors.

There are a lot of sensors. Gait detection + god eye is what convinced me this is probably impossible.

In my case, I was using NSA as a threat model for added security against the actual threat (cartels), so I wasn't as paranoid as I needed to be for NSA dodging. But in your case, you have quite a chicken-and-egg problem of getting that laptop to your doorstep in an untraceable way.

One optional step that I took, which is probably useless, is to live close to a wifi source that you can tap into from long range. I used a directional wifi antenna to a local restaurant. That way, if you do screw up and blow your opsec, it's traced to somewhere close but not equal to you.

(It's probably useless because once your physical location is traced, you're basically doomed – all they'd have to do is realize that someone's using the restaurant as a proxy. It's also quite unethical, since you're illegally using someone's equipment in a way that could very well land them in prison, depending on what you're doing. "Reasons not to fight the cartels" could fill up several notebooks, which is what ultimately persuaded me to stop trying.)

> you need to get from where you are to where the laptop is, and back, without popping up on any sensors.

Why? As far as They can tell, you're going to a house you've never been to before with no precedent for why, picking up an unlabelled brown box, and returning home.

The NSA would know you did that — but they wouldn't be able to connect it to a laptop in order to intercept/MITM it into being an insecure device (or to note down its MAC address for when you go online with it), since the "logistics chain" would be one entirely disconnected from you right until the moment you showed up at the house. To bug the laptop, they'd have to literally rip it out of your hands. Until the moment you pull into that house's driveway to pick up the parcel, they don't know it's your laptop (or what it is at all, really) so they don't know they should be trying to intercept it.

(And yes, They would likely have footage showing some other person dropping the unlabelled brown box off in the house's parking lot — but that would be a person who is not flagged as a Person of Interest in any NSA system, but rather some bright-eyed innocent college kid who had started a "new job" to "earn money fast" by "delivering parcels" just the day before. Parcels they pick up and re-box at AirBnB single-day rentals, rented just for the purpose of receiving that one parcel by the money-launderer.)

Replace "laptop" with "box full of dirty money" and this exact thing is done hundreds of times every day, with the NSA being able to do roughly zilch about it. "Cash mule" wouldn't exist as a profession if the transactions they facilitate could just be deanonymized+disintermediated in real time.

Or just buy a used computer at a pawn shop that doesn't keep track of the MAC addresses or serial numbers of its items, and pay in cash.

most distributions provide signatures/checksums to verify the download eg. https://ubuntu.com/tutorials/how-to-verify-ubuntu#1-overview

You might enjoy reading the "Cypherpunk Desert Bus" story by Peter Todd.

Is tor traffic that easy to detect?

Yes. It goes to a known tor node.

Not necessarily true. Tor bridges exist precisely for this reason: https://tb-manual.torproject.org/bridges/

Relay and exit node IPs aren't private, so admins will often collect them and just block them en masse. This causes problems, because a lot of that same IP space will often be shared with things like pool.ntp.org nodes.

By default Tor doesn't make any attempt to disguise the fact it's Tor traffic. Bridges are unlisted relays, which should work for IP blocking, while Plugable Transports are made to evade censorship-focused traffic analysis (i.e., they will prevent the adversary from detecting and blocking Tor connections as a whole, but not prevent a dedicated adversary from figuring out "has this client been using Tor?").

> you'd get an unpleasant phone call or visit to your office and be warned

sometimes I wonder why IT departments and security in general get a bad wrap, then I see things like this.

When someone just does whatever they feel like and violates policy, what do you think should happen?

Should someone send them a sternly worded email for them to ignore?

Or maybe they should be allowed to do whatever they want regardless of what risk it poses to the organization?

Why do people break rules? In that situation, I'd argue that education and understanding is the appropriate response -- for people on both sides of the table.

The meek pluggable transport together with Azure's domain fronting service explicitly makes it look like it's connecting to an Azure instance over https. [1]

[1] https://gitlab.torproject.org/legacy/trac/-/wikis/doc/meek

I can confirm as someone who works in netsec that this exactly how it would have gone at my previous employer.

There is a tone of “I know what’s best and will do what I want” in this thread.

If you think that the way to get the IT department to implement something for you is to sidestep around policy instead of working with them, you will just piss them off.

You will like this one as well “SSL/SSH Multiplexer” http://www.rutschle.net/tech/sslh/

Fixed link: https://www.rutschle.net/tech/sslh/README.html

Note that while this is a handy tool, its use is apparent to anyone observing the connection.

If your hard-to-reach server can connect to the internet (via a bunch of NATs and whatnot), you can just make it access your box of choice by e.g. Wireguard, or plain SSH with port-forwaring, or attach it as a node to your ZeroTier private network.

You only need a bunch of jump hosts if your target server has no Internet connectivity, and should not, in which case all these levels of bastions do make sense.

That requires having another publicly accessible box, or trusting ZeroTier though, doesn't it? The onion approach does not.

ZeroTier, Tailscale and such are OSS and have been independently security & crypto audited. I don't know if tailscale has been audited, but since they are a more popular tool I bet they probably are too. They're actually really good tools and would probably be more reliable than tor tbh, I would recommend looking into them.

> ZeroTier, Tailscale and such are OSS and have been independently security & crypto audited.

Both rely on their centralized coordinator servers which can mess with your routes (and thus your traffic) however they please.

ZeroTier has a published (but not OSS) coordinator, but their documentation pushes you towards their SaaS. Tailscale's coordinator is SaaS-only, unless something has changed very recently.

This is fair.

Their client node software is audited though, and the contents of your packets are not accessible to the router. This is why the amount of the possible meddling is limited to a DoS, AFAICT.

Who audits the Tor nodes that do onion routing is anyone's guess; I suppose ZeroTier is no worse than them.

> Their client node software is audited though, and the contents of your packets are not accessible to the router. This is why the amount of the possible meddling is limited to a DoS, AFAICT.

Normally the coordinator just forwards the keys from your peers, and so doesn't see the contents (the traffic doesn't pass through it, and even if it did it didn't have the key).

However, that assumes that the coordinator is being truthful with the network topology that it sends you. It could send you any topology that it wants to! This means that it could start MITMing whenever it wants to by telling you that $SERVER_IP's peer is now actually $COORDINATOR_KEY at $COORDINATOR_IP.

Theoretically you could defend against this by, say, running a cronjob that validates that the Wireguard keys are unchanged. But at that point you're not really gaining much compared to just using wg-quick.

Tor is different, because the .onion domain name inherently encodes the public key of the site you're connecting to. There is no way to change the key without also changing the URLs that people connect to!

zerotier adhoc networks are controllerless, though ipv6 only.

The client can be set to not allow routes/addresses from a controller.

The client and controller are licensed BSL.

Does this require addresses of nodes to be globally routable? (With such addresses you can as well connect directly.)

Ad-hoc networks don't seem particularly useful here. From their documentation:

> Keep in mind that these networks are public and anyone in the entire world can join them. Care must be taken to avoid exposing vulnerable services or sharing unwanted files or other resources.

you _could_ use your other device (the one you're connecting from) as the controller. whomst amongst us doesn't have a 3rd machine or VPS?

Your other device doesn't have a public IP address either.

Doesn't need one!

For that use case why not just use Wireguard?

Agree. Thats pretty interesting.

I use an SSH session and SOCKS5 proxy on a VPS provider for almost all of those other circumstances. Including checking external access etc.

But the last one is a solid use case.

Reply on news.ycombinator.com