How Asus and a Microsoft Bug Almost Broke Remote Work

Why our industry typically fails to agree to formal standards is that it takes time and effort to agree upon standards. The W3C is a good example: browser vendors move faster than the W3C could keep up with, so they decided to bypass them.

I don't fully share your negativism towards this: the ability to innovate quickly is important, and even when things get fully standardized, there is no guarantee that every vendor implements them correctly (again, see HTML and just how different things can behave with different browser vendors that all use the same standard).

I've frequently worked on such issues during my career and the time it takes to identify root cause / effective mitigations can be draining. I've found the key is to focus on creating reproducible test cases and a way to capture environment information in a manner that ensures you are comparing apples to apples. Some systems are so brittle that even adding additional load to capture observability data causes nuances that obscure the root cause. :(

Windows.

i remember when registry tweaks wouldnt randomly reset and random bloatware would not appear/reappear with each OS update. those were the "good" old days of XP/2000.

anyways, i finally got fed up with Windows 10 Updates + Defender shenanigans and perma-switched to Linux (EndeavourOS/KDE). Oh, the joy of your machine only ever doing what you told it, and blazing fast filesystem access with low resource usage! Unthinkable!

    type "Disable WPAD.reg"
    Windows Registry Editor Version 5.00

    [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Wpad]
    "WpadOverride"=dword:00000001

(and 'impossible using normal GUI/CLI tooling' seems unlikely, for one as regedit is still 'normal tooling' as well, and secondly because I've yet to come across a service that can't be disabled with the 'sc' command line utility, discounting Windows Defender though that is rather a special case due to the chance of malware itself disabling it, so it has a very specific series of steps to first disable the tamper protection in a trusted way and only then disable the service)

In this case it seems like an environment-specific issue, as if everyone would have this specific thing be broken, it'd probably long be noticed and fixed. In fact, one could probably easily find out what is wrong using a profiling session with WPR/WPA as the main symptom here is 'uses 100% of a CPU core', a methodology recently most famously spread by Google's Bruce Dawson.

(similarly, I wonder if these same people would 'complain' a Linux distro upgrade would affect their edits to /lib/systemd/system/*.service files - I would somewhat presume that to not be the case, in fact)

I am yet to find a title that doesn't work. This was discussed many times on HN.

As an opposite example, here's a Wine bug report of a Windows application failing to work in some network environments, but only if it in any way touches the Linux networking stack (such as using VirtualBox's soft NAT, or if running under Wine): https://bugs.winehq.org/show_bug.cgi?id=53346

I guess, in general, software (and networking) is 'bonkers'.

Shift+Shutdown to get an actual shutdown, or disable Fast Startup in powercfg.cpl.

And that Fast Startup is not to be confused with Fast Boot, which skips the POST.

And you can't add features to browsers if there isn't a formal specification, right? Right?

I would venture to say perhaps AzureAD?

When a OS receives an instruction from the router/DHCP server that no proxy should be used on this network, and the OS adheres to that…

And that alone just breaks AzureAD, that’s clearly a point where AzureAD needs to be made more resilient.

It's also a terrible idea. For example now anyone running a evil DHCP server in a WLAN you joincan get your browser to follow a malicious PAC script which lets them MITM even HTTPS traffic... see eg https://www.pcworld.com/article/415991/disable-wpad-now-or-h...

(This was of course back when Windows users were getting regularly pwned by a windows worm of the week so wasn't anything out of the ordinary)

I got to the point where I was pretty sure that WPAD wasn't working right, and was in the midst of capturing all sorts of data and escalating that, when a side conversation prompted someone to ask "was it you I worked with on X?". Then it all came together.

I think this is one of those things that ends up being so esoteric most folks haven't worked on it, then the chain of events leading to it... But the moment I heard the trigger I was able to repro it... That was quite a relief.

On the other hand, attacker could probably always send messages out via DNS tunnel

Please someone more knowledgeable do enlighten us

This isn't quite my space, but from what I recall there's a couple wrinkles with transparent proxying. It's overall a good idea, but has some edge cases:

- Authentication can go sideways in weird ways. A 407 from what looks like the correct site can cause odd things. IIRC there isn't great support from vendors for authenticating transparent proxies, too. Sure, you could auth off the machine instead (X device is on the network, therefore it's allowed), but what about shared machines... Proxy auth as user is better, because it allows requests to be tracked to a user ID and login session, not just a device.

- Client getting load balanced between proxies during a session can trigger reauth (because auth sessions not shared between proxies), this can confuse a client.

- HTTPS sites can get weird.

- Routing in very large private environments can complicate default routing to the public internet. Although this can be handled by doing transparent auth + optional manual config.

https://www.virustotal.com/gui/url/54e060dd220272974a9a5bffa...

Years ago Yandex was flagging on some sample perl code that had a .txt extension (some something like udpscanner.pl.txt or so, IIRC) that I had sitting in a directory. There's no perl CGI, no way for it to execute on the server; just sample code. IIRC it was even served up as text/plain for easy reading in browsers. It was something that would be run directly on an OS, to do some fast scanning of open ports. For an end user to run it they would have to download it and get it executed by their perl interpreter. Definitely NOT an exploit in a browser.

As I recall it was something super basic that I found on a compromised server years ago and referenced in an old writeup.

For some reason it was flagged by Yandex as a browser exploit, they reported up to other places, and Malwarebytes flagged the whole site as malicious.

Since fighting the technical reasons why their scan is flawed is Sisyphus, I ended up just removing it from the site and getting Yandex to rescan. They now list the site as clean, but some old tools still say something untoward is going on.

It's frustrating.

This was something new and interesting with COVID WFH. Whereas previously WFH was an optional thing, a bit of a perk, and usually not long-term, we could fault someone's home internet and let them contact the ISP/whatever to figure it out, 2020 changed things.

Since we were telling people they needed to work from home, and if they couldn't they'd need to contact HR, it really was best if we helped them out a bit more.

Not full-on support, but talking through the problem and making a recommendation of how to make things better basically became the norm.

Though for some time there is a push to just disable WPAD discovery, because it can be used for redirecting the traffic for nefarious purposes and most people don't use a classic web-proxies anyway.

And if you did use it, it's exceptionally rare because, AFAIK, it is only supported by Windows and then things which in turn use WinHTTP.

Of course, because the RFC was never adopted, you could have used option 252 for something else because, technically it's reserved for private use. And your (private) use may be different from someone else's WPAD use.

Because it wouldn’t work. It’s completely infeasible for an moderate sized non-tech business because the “IT environment”, such as it were, of employees homes is absolutely non-homogenous.

You can, as an end-user IT support department, rely on basically only two things about any user’s home kit and ability to interface with it: they have power (usually), and they can probably get internet access via wifi somehow. Any other assumption about their home environment will prove incorrect in some wild way.

I’ve run into users that don’t have Ethernet, or WPA2, or any kind of public IP, or next hop latency under 200ms, or more than 1mbps bandwidth. We have users that don’t have home Internet, and for the duration of WFH orders used their company provided phone as a hotspot. I’ve run into users who can’t promise that they’ll have mains power day to day.

Even if the user does have Ethernet coming out whatever their NTD might be (and it might be exotic!) you can’t guarantee that they can follow any kind of instruction to set up a router: they are about as likely to end up without a functional internet connection at all as they are to get whatever box it is you give them to work.

You also can’t even be sure your instructions will work for whatever it is you’re trying to do. Are you taking into account double NATing? Triple? Do you know that none of your users home ISPs have some kind of MAC whitelist on their NTD? What if their current router is built into some kind of HFC modem?

This is not a problem you can solve the mass use case for and address the edge cases: it’s all edge cases, in my experience… and that’s if you’re operating in a single, highly developed first world country.

Like sure, you could ship a router, firewall, IPS, WAF, et all. But if you're not stopping outbound, controlling clients ability to talk to each other, and blocking things like UPNP, then you're still basically on an untrusted network.

Your options are basically make the network unuseable or mark it untrusted.

At that point you might as well just force all traffic through a VPN.

Honestly, in my experience here in the US, /most/ people's home networks are just fine. There's just edge cases, and then there's weirdness like this.

I think COVID WFH / School-From-Home was actually a boon for internet access in the US as it forced a lot of people to think about their home internet connection as more than just a binary works/doesn't.

Something like this, for a large company, would be another massive thing to maintain. It'd be a whole additional cost to buy, platform to manage and support and patch and upgrade. And people would still just plug them into whatever network jack they see.

Then there's the whole management/update infrastructure that'd have to be built and secured and maintained... It'd basically be like reimplementing what Cisco offers with Meraki, a small remote VPN box, but for everyone. Not cheap at all, and for most users, just not necessary.

So there's not even the cost of reflashing them.

Unfortunately a lot of random utilities will add them, even old stuff that might not work right anymore.

Any why? Because lots of app developers seem to think that you'll want to quickly open anything with their app via a right-click popout...