eBPF on Windows
github.com
github.com
https://vbpf.github.io/assets/prevail-paper.pdf
The verifier in this paper also has some biting limitations; for instance, you can't resize a packet in it, because they don't account for pointer invalidation. I wonder whether they've since implemented these verifier features, since they'd be problematic for compatibility otherwise.
Additionally, the PREVAIL paper explicitly doesn't verify program termination, which is kind of a dealbreaker for kernel BPF.
From the counter examples directory, this implementation seems very promising.
https://www.delphix.com/blog/delphix-engineering/zfs-channel...
So it’s been done before. I think traditional BPF programs were really short and ran in performance critical contexts. Disallowing backwards branches and limiting size was preferred to slowing down these code paths with instruction counting.
I'd be more alarmed if someone had solved the Halting Problem and I hadn't heard about it, to be honest.
Termination proofs for systems code (PLDI 2006): https://dl.acm.org/doi/10.1145/1133255.1134029
Assuming certain constraints (what you can define bounded analysis rules for) you can prove that things terminate. These rules things are usually far more advanced than we suspect initially, but they're not still not fully unbounded.(And we can often come up with more rules to make things practical even if unwieldy)
Deciding termination for an arbitrarily large program in a turing complete system CAN require no less than execution of such a program thus we CANNOT guarantee to be able to verify termination for EVERY program. (This is the halting problem)
I spent a lot of time on my thesis learning about type inference systems and as many such systems for uncooperative languages(JS,Python,etc) need to do abstract interpretation they fall into the same kind of issues and this is why practical systems use JIT compilation instead of AOT even if there are exceptions (Shed Skin Python is quite impressive for example).
So there is no way to know, for EVERY possible program, whether that program will terminate or not. But there are still some programs that you can know for certain that those programs will terminate.
I knew this day would come. Be back in like a week. Gotta get my neurons warmed up and translating Theoretic CompSci/discrete math notation again.
If I don't come back send a search party. I'll probably be stuck somewhere around pumping lemmas screaming "This is arbitrary bullshit!"
No unbounded loops for one. I think no backward jumps was previously a rule? I think the ability to tail call another bpf program was added, so not sure if you can accidentally loop forever between individually terminating programs.
Being able to run an unbounded loop in eBPF would be a big deal (doesn't mean it can't be done; in fact: it almost surely can be) --- a serious vulnerability.
Is their assessment correct? If so, how comes that we got DTrace in 2019 and now eBPF ported to Windows? Are they trying to consolidate all tooling into one platform?
Most people here have probably seen Bruce Dawson's performance analysis and debugging posts. WPA/WPR are probably the only user-friendly (ish) ETW applications and even those are not that easy to use.
So yeah, Windows has had powerful tracing and inspection tools for a while, but few people know how to use them well.
As far as why they are doing DTrace and eBPF, I think both of those fill some holes that ETW doesn't. Mainly dynamic tracing and instrumentation, but I am sure there are other advantages I am not thinking of as well.
They could have come up with another hopelessly complicated Microsoftie framework to do the same things, but it's probably a good thing that they didn't.
Visual Studio's own Performance Profiler (Alt+F2, Ctrl+Alt+F2) also uses ETW - just more streamlined, better UI - but once it collects a bit too much data, it just can't handle it.
Yet, in a typical Microsoft way, if you want to use that Message Analyzer tool now:
>Microsoft Message Analyzer (MMA) is being retired and its download packages removed from microsoft.com sites on November 25 2019. There is currently no Microsoft replacement for Microsoft Message Analyzer in development at this time.
This could be a game-changer for the infosec community in particular - now, if you want to get into internals, such as tracing file system and registry calls, you've got to write drivers. And drivers are very tricky to write, and it's very easy to miss corner cases - which can result in the dreaded BSOD. Plus, drivers need to go through a verification and signing process by Microsoft.
Having access to that capability from user-mode, without having to write drivers... that would be amazing.
eBPF looks cool since it is a VM, that could have access to more kernel structures (network?), but solving the issues you described is largely a solved problem on the platform.
And RegNotifyChangeKeyValue is only useful for watching a single, specific value - if you monitor a tree, it doesn't even tell you what changed, only that something matching the filter did. And as with file system changes, if you want to know which thread/process/user made the change, you need to use a driver.
There is also System Monitor which logs the events to EventLog: https://docs.microsoft.com/en-us/sysinternals/downloads/sysm...
Are those not enough?
SysMon is a great tool, but the license prevents distribution (such as bundling with an installer) - users must download it from Microsoft.
Does anyone know if it can profile file/disk io type activities?
Importantly: eBPF has at this point not much to do with packets; it's a generic kernel and userland instrumentation layer. Most new BPF code is written to monitor local program runtimes, not to look at packets.
If Windows adopts XDP, we might get to a point where
Also, even in an XDP program, you're still likely to use a bunch of perf stuff, which is again pretty Linux-specific.
[0]: https://github.com/iPower/KasperskyHook
[1]: https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
Comments like this one take threads in predictable, uninteresting directions, and—what's worse—they often get upvoted, accumulating mass at the top of the thread and choking out the specific interesting stuff.
"Windows v Linux" is a classic example of a generic black hole, sucking passing spaceships into a state from which no light can emerge: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....
To be fair, this is a co-creation between the comment and the upvotes, and the upvotes are more to blame. But the idea of that guideline is to refrain from introducing black holes in the first place, so the spaceship can poke around something interesting in the vicinity without collapsing, screaming, into an irreversible fate.
Related explanation here if anyone wants more: https://news.ycombinator.com/item?id=26894739. Note the point about diffs—that's key. Diffs are what's interesting!
Intention was to highlight how MS, that fought with *nix tooth and nail, now they are adopting more and more ideas from *nix, basically moving towards being best enterprise Linux.
SSH is not even enabled by default on ec2. You have to Remote Desktop in, run a bunch of commands to enable it, and even then I could never figure out how to get authorized_keys to work.
Once you have SSH running, you can run commands via command prompt or powershell. Powershell is apparently pretty modern and well designed, but it's quite confusing if you're used to Unix.
A few applications I tried to install came as MSI's and they could not be installed via the command line, requiring the graphical interface.
It felt like a bad way to run a server. You want fast and lightweight remote access on a server, and you want scriptability. Windows was simply worse in this area.
Even if Windows was 100% compatible with Linux apps (including the modern CUDA deep learning stack...), I don't know why one would ever want to run Windows on a server. It lost the mindshare war, and I really see why. It really is that bad of an experience.
The main reason I've heard people use Windows server is because they are forced to because of app compatibility (no tech startup would find themselves in this position), or the preference of maintaining a server with clicking in the GUI (again, not a place most of us here find ourselves).
Anyways, I'd love to hear from someone, anyone, who likes Windows server more than Linux.
The point of this comment is also to state the opinion that Windows is currently a very bad Linux distro.
Some clarifications are in order.
MSIs CAN be installed on servers with no gui required. Or much more commonly via powershell automation. Completely remotely and I'm not referring to remote desktop either.
I know its fun to complain about Windows, but really, if you're not using powershell and the right tools such as a Windows Domain, you're kind of like some kind of debian linux admin that refuses to use bash scripts / debian apt / ssh etc plus chef or similar management. It really is that kind of comparison you're making. What would you say about a linux admin that refused to use SSH or any scripting? You'd wonder why they are complaining.
Most of Windows Server comes alive when you join the machine to a domain and it gets access to actual infrastructure rather than some lone isolated server. This is to the point where you don't even need to log into the machine via remote desktop or ssh at all. In fact you manage the server from a completely different Windows machine with an admin account within the domain. You can then run the powershell script on your management computer and things happen on the target computer(s). I haven't even mentioned the multiple other tools you get as part of the machines being joined to a domain. Or you can deploy the script to the machine and execute it there. Still no remote desktop required.
I keep meaning, year after year, to take at Samba and see how its Active Directory emulation functionality has progressed. I never end up getting around to it. Presumably, since so much of Active Directory is really just leveraging LDAP and a funky proprietary schema, it has progressed significantly.
I've had a hard time figuring out how to position Desired State Configuration. It feels like an appeal to Linux sysadmins used to Puppet, Chef, et al. I can already do everything it can do with Group Policy, and I'd want an Active Directory domain for single-sign-on anyway. I have a hard time coming up with a use case for learning all new tooling when I've already got AD and Group Policy.
I've been using Group Policy heavily since 1999 (during the Windows 2000 beta), so I'm used to knowing what applies when. Being familiar with which client-side extensions (CSEs) do which processing helps. The documentation for Microsoft's own functionality is reasonably good. Third-party software is a mixed bag.
Remotely rebooting machines is easy. There has been a remote shutdown API in Windows NT since the at least Windows NT 4.0 (and probably going all the way back to the beginning). Since Windows 2003 there has been a command line tool included to actually use the API. Likewise, there are command line methods to remotely logoff users, refresh Group Policy, etc.
Full agreement. We were remotely powering on/off hundreds of machines with a single command line. Then just a web page because we needed non-IT people doing it on regular basis.
We used to change wallpapers of certain groups of machines on a daily or even hourly basis due to special events. Do people really think we remotely logged into each one? Or manually remoted each one? Hmmm.
Next someone will be saying its impossible to get the event logs from each machine... truly bizarre. (Yes you can, and we did, even from client machines if a machine had developed "quirks")
Again, these complaints are just strange. Its as if they weren't using their infrastructure.
None of this is any kind of wizardry. I'd expect any linux or windows admin to know the tools required.
Also, encryption and authentication for log forwaring is bound to AD Kerberos credentials that frequently expire, leaving you without logs again.
Any old syslog client and server is far superior in all the aspects mentioned.
Also, event forwarding isn’t the only way.
Also, for a lot of things there is just no group policy you can set somewhere, so you are still in need of some "execute me that script that does the needful" in a lot of cases.
I'm not sure what to say re: "execute me that script that does the needful". That's going to hold true in any environment, Windows or Linux or whatever. If you've got third-party software that uses unique configuration persistence mechanisms, or doesn't work-and-play with the OS-provided management interfaces (Service Control Manager on Windows, systemd on Linux, etc) then you're going to have to script one-offs. Most real sysadmin work, in my opinion, involves leveraging or creating code and infrastructure to handle automating assemblages of software. Scripting is often how that's done. Configuration management tools add some formalism to the process, but it's still "glue and tape"-- albeit fancy.
You need a script that is idempotent, so you can execute it on all your machines and maybe do a retry or two if some are unreachable or an error happens. So the script should not only change things it is supposed to change (e.g. append a configuration line somewhere) but also check if that action is necessary or already done (e.g. that line has already been appended). Then you maybe need to restart the respective service to reread it's configuration. But of course you wouldn't want to unnecessarily restart it, since that might be disruptive or expensive, so you have to check if that configuration has been changed or not before restarting. Then handle automatic application for machines that are currently offline, have just been installed or need a refresh. Add in a some templateing and parameters, different groups, logging success/failure and you are at quite a complex script.
Or rather, a script that no-one should write, because it is exactly what configuration management is supposed to do. If you are writing all that by hand for each one-off, you are doing it wrong because you are reinventing the wheel, and usually badly. If you aren't doing all that, you are missing important parts. So imho configuration management like puppet, salt, chef, … is table stakes for any kind of sysadmin work. I'm not familiar with Windows DSC, but I find it strange that you dismiss it as pandering to the Linux crowd.
Everything you complain about Windows Server not supporting is actually supported, but it was your unfamiliarity that was ultimately the issue. "Linux guy who is unfamiliar with Windows Server prefers Linux" isn't particularly shocking news.
I don't complain that Linux is hard to administer with PowerShell. I learned SSH and Bash.
The reverse does not seem to be true, Linux admins generally expect everything to work exactly the same in Windows as in Linux.
A negative effect of this is that Windows has many recent hires working on some of their teams with mostly Linux experience, and they're copying Linuxisms into Windows bug-for-bug. Even if it makes no sense in Windows.
A recent example that boiled my blood is that the new Windows Terminal emulates the incorrect "Clear-Host" behaviour of ancient Linux terminals. It doesn't clear the virtual terminal any more, just the current viewport. Before, it was a convenient way to reset the terminal's display state without resetting variables. NOW, it just deletes some of the display state, so if you scroll back you get scrambled garbage.
This is not a small thing: it's significantly impacting my workflow. I've seen garbage output, overwritten output, scrolling back accidentally corrupts the output completely, you name it. This is not a coding error, this is correctly copying a limitation of some terrible terminal from the 1960s nobody should give a shit about in 2021.
Worse: If you look back at the history of the debate around fixing this, as far back as the early 1990s there were Linux people advocating for fixing this! It's obviously broken.
Then: When the same debate came up in the Windows Terminal issue tracker, people making the same rational, logical arguments were shot down with: "We have to copy the standard."
Hint: There is no "standard". This is not an RFC. It's just what Linux does! Linux does this just because this is what some random MIT or Berkley student wrote in a hurry in the 1960s! If you make the argument that the majority of systems work some way or another, and that's a defacto standard... then Windows should win because it was used for the the vast majority of computing for many decades. Linux has taken over only recently, and only in some areas (Android and web servers).
Hi! I run the team who made this decision, and I flatly reject the characterization that these changes were made by "recent hires" who are "copying Linuxisms into Windows bug-for-bug."
The discussion you're referring to spans multiple threads, so I'm not certain the exact one you're referring to. I can, however, give some of my input:
1. We worked pretty hard to make sure that PowerShell's Clear-Host and CMD's "CLS", which operate on the entire scrollback buffer in the traditional console, are properly translated into a full buffer clear over SSH/in Terminal/etc.
2. Terminal fully supports CSI 3 J, a non-standard extension to the Erase in Display control sequence that clears the scrollback buffer. The handful of tools I am using that support "clear" actually seem to emit both "clear viewport" and "clear buffer." If you're using a version of clear that has been configured to NOT emit ED 3 (clear scrollback history) by default, that's simply not my team's fault. I've heard that you can use "clear -x" to affect your desired behavior.
It turns out that there being no true standard here plays to our advantage: we can support requests for all manner of clear, and trust that an application can be made to do the right thing. If you're using an application that can't be made to do the right thing per at least thirty other terminal emulators' designs--spanning more than just Windows and Linux, mind you--I apologize.
If you'd like to reopen that discussion here instead of on GitHub, you are more than welcome to.
Hi! I see Microsoft people on "docs.microsoft.com" use forward slashes now in examples that only work on Windows. So, there's that.
If I open the latest version of Windows Terminal, running PowerShell Core (not some Bash thing with some VTY codes or whatever) and I run "Clear-Host", I can scroll back and see content that wasn't cleared.
THIS BREAKS MY WORKFLOW. I used Clear-Host so I can run a command with pages out of output, and then I know that scrolling back will go back to that output only, not something that happened three hours ago but superficially looks identical.
If I do this with PowerShell 5.1 with the old Terminal, it works as expected. It works like it has for the last 40 years of DOS and Windows history.
PS: It's broken even worse in Visual Studio Code.
You guys outright abandoned your user base to pander to Linux users. On Windows. Linux users on Windows. That enormous, huge majority of your paying user base that is apparently more important that the hundreds of millions of people like me.
So yeah, you should reopen the issue and rethink your priorities...
IMO they're going after developers using Macs. Linux (and Docker) runs better on Windows than it does on Mac OS. They're doing _something_ right.
Not Windows developers on Windows?
The Windows Terminal guys are similar. "Do you have any plans at all to make this work for users of Windows?" "No."
With luck you need to install VSCode and there is an extension to call said CLI.
Who needs VS designers or Blend for Windows UI development? Just recompile and run WinUI applications.
I really don't get it.
It is starting to become tiresome to get some CLI stuff instead of proper VS wizards.
All the tooling for headless / remotely-manageable Windows is "in there" in 2008 and later. Prior versions required more jumping thru hoops to manage via command line, but Resource Kit tools helped.
The tedious "re-apply the service pack and reboot" or "you thought about changing a setting, so therefore reboot" scenarios were nearly removed with Windows 2000. Having plug 'n play and hot-plug hardware work well was great, too.
For those unaware of it, IIS used to execute fully on kernel level, so ISAPI extensions were comparable to device drivers, which meant any programming error would just kill the kernel, thus requiring a reboot.
So tracking down memory corruption issues on ISAPI extensions was bound to have a countless amount of reboots during the work day.
I assumed that kernel-mode HTTP service was an answer by Microsoft to the "Tux" web server: https://en.m.wikipedia.org/wiki/TUX_web_server
Funny, I've seen people saying exactly that when newbies bounce off some aspect of Linux, and in both cases it's a usability failure.
Discoverability of this stuff on Windows is terrible. Things like powershell remoting exist but getting them working between two random machines is opaque. Yes, some of it gets easier in a domain, and that's one of the big Windows wins (since the NIS/YP era), but domains are an extremely premium feature.
Most of the stuff I learned about UNIX required actually buying books, not much different from Windows.
Drop someone on a proper UNIX just with man to see how far they go joining a NFS server or logging in via yellow pages.
If on workgroup, you just have to have the same user name on different machine and it works like in domain.
> Discoverability of this stuff on Windows is terrible
Meh... matter of taste and adequate Windows social skills, not a fact.
... and password. ... and no it doesn't really because an AD has a few other extras, such as Kerberos and an LDAP database with some odd ideas to chat to. You can of course spin up a Samba AD DC or two for a cheaper alternative.
And really, if you want to talk about discoverability between Linux in a server role and a Windows Server... Windows Server wins by an absolute landslide. But I don't think discoverability is all that important when you're talking about server administration.
msiexec /qn /i msi-name.msi
Edit (now that I have more time):
All the tooling to make Windows command-line manageable is there "out of the box".
In a former "life" I staged a Windows Server 2008 R2 install that was remotely manageable with SSH (using a third-party SSH server) "out of the box". Similar to a "kickstart" w/ the Anaconda installer, you're talking about chaining scripts to run after the installation to enable desired functionality. It wasn't much different than automating a Linux install.
If you enable the serial console you can do some nice command-line management of Windows VMs through your hypervisor's virtual serial port functionality too. I enjoy having the serial console open because I've been able to diagnose "hung" Windows machines (that is, not responding to GUI logon attempts) thru the console. It's very handy.
My main pain with OpenSSH on Windows was that its an older 7.7 Windows port that gets installed. However 8.1 is finally bundled as part of the recent KB5001391 and fixes some annoying bugs.
And ssh needs a restart.
That said, windows remote management is traditionally via WMI, not powershell. But PS has come a long way, and ssh is a sane transport. And it can function as a tunnel for WMI, RDP and PS (ssh is easier to use for key based auth, disabling password auth).
https://docs.microsoft.com/en-us/windows-server/administrati...
My gripe with installing things on Linux is that every project assumes install means “build from source”, and it’s often annoyingly hard to find the exact yum/apt-get name for something (do I need a PPA? is the name thing, libthing, libthing-dev, libthing2, or something else?) But this is a much smaller issue than anything I run into with Windows Server.
bash is pretty confusing eve if you're used to unix
-- Greg's
PowerShell was made by Unix people after 2+ decades of bash experience. It solved hundreds of things.
Yet people complain it didn't solve few or it takes a bit more to load (no, verbosity = RTFM). You can't ever please I guess.
I don't mind bash warts, history is what it is, I don't sell powershell .. but someone saying powershell is confusing has serious jaws. powershell is really nice and only if you fancy sed/grep everything then you'll consider bash* superior
https://www.networkworld.com/article/3110744/linuxcon-qa-wit...
thanks for the link
I'm not sure what the ideal shell would look like though. Maybe an actual (simple and expressive) programming language with a way to 'lift' statements ?
On some places tools are not there (docker image for example).
Some people complain about size of the dotNet runtime, but if you install all those tools bash "needs" for comfy usage, you can equally easy install dotNet.
unix give you partial stuff for that like comm but it's just brittle
I think the big issue for Windows is that most "admins" aren't in the habit of automating things, or at most they'll write a basic bat file. They still expect to click around their GUIs, so they'll be fairly reticent to install the Windows Server Core version [0]. "To be able to intervene in case something happens". The main issue with this is that they often aren't aware of possibilities offered only via PowerShell[1]. There's also the issue that when they look things over in the GUI, they won't see the configuration that's only visible through PowerShell.
---
[0] Windows Server Core comes without a GUI, but it can be managed remotely with the usual tools. However, not all server roles work on it. Remote Desktop Gateway is one such example, even though it doesn't have any "desktop" functionality.
[1] For example setting up split-view DNS. This is possible since Windows 2016, but only via PowerShell, and it's impossible to know from the GUI that it's activated. Also, this configuration doesn't replicate through ActiveDirectory.
It was maybe easier to find people to support Windows servers... Just click here here and here to install.
.Net Core programs run much faster on Linux.
Does that sound like something they would let happend if they wasn't serious about their Linux efforts?
That said: all is not good. For years there seems to be a fight going on in the wheelhouse.
One month it is: Microsoft, the reliable, reasonable vendor in a world full of Oracle and Google.
Next month it is: let's increase the cost for this small company by $10000 just because we can.
Next month: something admirable.
Next month: try to push Edge using some sleazy tactic learned from Googles Chrome push combined with resetting the defaults.
It must be frustrating for the guys who try so hard to drag the rest of Microsoft kicking and screaming into the future everytime some old guys gets up from the wheelchair and turn the weel all the way to the port side while the helmsman wasn't paying attention :-]
Hhhm I think the most remarkable parr is that it runs at all, really! .Net Core on Linux by itself shows plenty commitment.
The performance of it? Considering NT was only developped by a single company, has to maintains a lot of stable/legacy driver APIs, and seems to have been on the backburner for a few years, as a Linux user I'd find it very humbling if our kernel still came out behind :)
It's interesting to see where this is all converging. It'll be easier to run Linux tech on Windows, and all the MS legacy still runs.
But at the same time maybe they're familiarising their loyal userbase with the outside ecosystem a little more than they should. If new Windows users are encouraged to learn Linux ways, at some point you're making it easier for people to transition
With WSL you can setup some docker containers or do some orchestration with Kubernetes and push it off to the cloud with minimal effort.
If you really want a full Linux environment you can spin up a VM in Hyper-V. WSL just makes it easier to do things where a VM is a hassle.
zed@ZED-PC:~$ sudo service docker start
* Starting Docker: docker [ OK ]
zed@ZED-PC:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
sudo systemctl start docker
[sudo] password for u3332:
System has not been booted with systemd as init system (PID
1). Can't operate.
Failed to connect to bus: Host is down
> Nowadays you can try:
> sudo service docker start
> when using WSL2, if you are running on windows version 2004 or higher (I assume).
Which is what I did in the listing above.
sudo service docker start
[sudo] password for u3323:
docker: unrecognized servicee:
ver
Microsoft Windows [Version 10.0.19042.928]
e:
Everything except the service start, which I had to find in that StackOverflow thread. We've hit the max comment depth, so I couldn't response to you directly.
sudo service docker startIt's not a problem with Docker on WSL 2, but a problem with the way WSL uses its own init system instead of systemd, while some Ubuntu packages are packaged for a systemd system.
define terrible. Compared to what ?
> No one is seriously developing in WSL.
A very strong statement. Do you have any proof perhaps ?
Terrible = significantly worse than a native Linux distro that doesn't use a file system integration layer.
> A very strong statement. Do you have any proof perhaps ?
Just experience working in the industry.
What filesystem integration layer?
> We recommend against working across operating systems with your files
https://nelsonslog.wordpress.com/2019/06/01/wsl-access-to-li...
Just run everything on the Linux kernel and compilation will be the same as on Linux. You do have a choice of where you put your files.
https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...