Dave Cutler’s five-decade-long quest for quality (2016) | Hacker News Reader

Dave Cutler’s five-decade-long quest for quality (2016)

news.microsoft.com

101 pointstosh70 comments

For anyone interested in the history behind the NT kernel I recommend a book called "Show Stopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft" by G. Pascal Zachary - really enjoyed it.

I think the Windows NT kernel is a very underrated piece of technology - the reason it was unstable in the beginning is due to a corporate top-down decision: the hardware of the time was limited so they included GDI stuff in the kernel (crappy code not written by Cutler's team).

It's, in a way, the continuation of VMS and inherited a lot of the concepts.

Which is not really surprising , considering that Dave was one of the main architects of VMS.

On a personal note: Best OS I ever worked with. Rock solid stable and a few conecpts and tools, which are still unmatched. At to that the arguably best documentation I've ever had the pleasure to work with in any environment.

>At to that the arguably best documentation I've ever had the pleasure to work with in any environment.

Documentation for everything was a lot better in the 70s.

You're right, of course.

DEC, however, was in some respects really above the pack. Documentation being one of them.

I'd throw in DECnet and VAX Clustering as other examples.

NT was pretty damn good, in hindsight. 2K was alright too but bang-for-seniority NT4 was great.

NT 3.5 was rock solid, believe it was NT4 where they put graphics in the kernel to speed it up. One wonders why they don't put it back now that computers are 1000x faster.

I was under the impression that GPU drivers run in userspace since ... Vista?

At least I have seen the GPU driver crash on Windows 7 machines - the screen went blank for a second or two, then it showed the same desktop as before, with a popup that rougly said, "The GPU driver crashed, so I restarted it."

They did, it's why you no longer have to reboot when upgrading video drives. You just get a couple of black screens as the modules are reloaded.

They actually finally did this in OneCore AFAIK.

Awesome, NT finally shedding some baggage.

One particular flaw was CMD.EXE. In the few pages where it is mentioned in the Showstoppers book, it appears that this code came from IBM.

The scripting in CMD.EXE has been described as a "serial killer," and they really would have been better off implementing a Bourne shell for Windows.

http://blog.nullspace.io/batch.html

>I think the Windows NT kernel is a very underrated piece of technology - the reason it was unstable in the beginning is due to a corporate top-down decision: the hardware of the time was limited so they included GDI stuff in the kernel (crappy code not written by Cutler's team).

Err, no. The reason it was unstable in the beginning were choices like having a "MUST SUCCEED" kernel memory pool which caused a bugcheck if the system could not allocate from that pool. Random kernel components allocated from this pool and if you opened too many handles (sockets for example), you'd get a bluescreen because an unrelated driver was trying to allocate memory for some other activity.

There were probably other similar choices that I don't remember now. It has been 25+ years.

And let's talk about the NT Driver I/O model. The most highly complex, over-designed POS I have ever seen. And this design was "polluted" for performance out of the gate (Fast I/O path, for example). It took Microsoft years to reliably and safely cancel pending I/O in the kernel. Remember the days when a random peripheral driver could hold on to an IRP and cause bluescreens all over the place because the process it was doing I/O for exited ? It was hard for internal Microsoft people to write good drivers, let alone OEMs in the wild with a fraction of the learning resources at their disposal.

NT was based on the microkernels that were fashionable in those days (e.g, Mach from Rick Rashid et. al). As with most academic ideas, when microkernels met the real world they had to be adapted.

Win32K came about because GDI performance in NT was crap. There were other things that were similarly pushed into the kernel already. I believe a classic microkernel would also have networking modules in user mode, for example. DirectX in the kernel (hated by the purists back in the day) is inarguably responsible for NT being able to supplant Win9x in the consumer PC market.

The NT kernel evolved over the years to be something that actually met the needs of Microsoft's customers. I would argue that's the real achievement, and not all of that can be laid at the feet of "Cutler's team".

I second the recommendation for "Show Stopper" :-)

I can't really comment about mentions on the I/O model or the early instability.

We should give credit to the early architectural decisions made by Cutler which allow for niceties like the Windows Subsystem for Linux (WSL) that came about recently.

I third the recommendation for "Show Stopper" as well. :-)

Just ordered "Show Stopper" now and started reading it, looks like a great read so far.

>We should give credit to the early architectural decisions made by Cutler which allow for niceties like the Windows Subsystem for Linux (WSL) that came about recently.

i think most - 99.999% - credit here goes to Linus :) Given the modern power of Linux the WSL would have happened anyway - independently of whether early NT architectural decisions help it or preclude it.

For the Linux subsystem, sure.

For the POSIX, Win32, and OS/2 subsystems that existed at the birth of NT, the kernel capability was all Cutler, and Linux was in no position to contribute.

Showstopers explains the NT kernel personalities.

I really enjoyed ShowStopper when I read the original release in 1994.

In particular, I really admire the people who worked on that project, because they had a lot of field-tested reasons for the things they were doing.

But the whole subsystems concept basically cleaved the logical domains in a way that was obviously, even at the time when I read it, not going to extend well in the future. While the POSIX and OS/2 stuff can be seen as a kind of hubristic base-covering (the one kernel to end all kernels), it should not have made the cut in a disciplined engineering organization.

One advantage of the NT IO model the IO completion support for asynchronous operations. The same model works with network and file io. Linux does not have anything as nice as this.

I second the recommendation for Showstopper!, though I wonder if MS's culture of burning out developers for months contributed to its reputation for buggyness in the early 00s.

Whoever borrowed that book from me, please return it. Thanks.

Show Stopper is probably my second favorite product development story book. (My favorite is Soul of a New Machine but I'm biased there because I worked at the company a bit later and knew many of the people involved.)

If you liked Show Stopper, you would probably really like the various MIT Platform Series books. Each of them is excellent and while they are not really the story of new product development, there is plenty of product development in them.

That was a great book. In college, I read it on the plane on the way to an interview with an operating systems team. During my interviews, people thought I had a ton of experience with operating systems coding, which I didn't, but the concepts, terminology, and design debates from the book were fresh in my mind. ;-)

When I used to work at Microsoft, my manager (pretty high level person, not a middle-manager) used to tell me: Dave Cutler is one of the biggest assholes in the company and most people don’t like him but the company has to put up with him because the cost of losing him to another competitor (like Google) would be too much from competition standpoint.

Given the fact that Dave Cutler would basically be peers with Jeff Dean and Sanjay Ghemawat who had key contributions to multiple fields in the computer science, this makes sense.

Instead of hearing a story by a random Microsoft press corp reporter, or his management (like Steve Ballmer), I actually would like to hear stories about Dave Cutler from his subordinates.

What could possible be the purpose for you to tell us that your former boss thinks Dave Cutler is "asshole". There is zero people in the world that need that information.

It was a discussion about how sometimes upper-level engineers can be demeaning to others. I don’t remember the details, it's been many years. This was within the context of the company culture mostly prior to the Satya Nadella era.

I think we were discussing that how assholes should not be tolerated, but there are sometimes assholes up in the food chain that are far too risky to lose.

Please stop labeling people "assholes". It doesn't contain any useful information. That's the same as saying "I hate that person".

No, it says "that person behaves badly".

Whether that's true or false is another matter, but it's not the same as saying "I hate that person".

I think "asshole" tries to say much, much more. It's such a loaded word, you should never use it if you really try to explain how a person behaves.

It's very hard to understand why anyone would label Cutler as "biggest asshole". He clearly has a very, very good track record in working in teams.

Cutler is famous in his derision of early UNIX.

https://en.wikipedia.org/wiki/Dave_Cutler#Attitude_towards_U...

The quote most often heard is that "UNIX is a junk OS designed by a committee of Ph.D.s."

Showstoppers goes into some detail about his career goal "to get UNIX."

Whether this is is purposeful or not is likely subjective. It is, in my opinion, informative.

Are you saying that his attitude towards unix is the same as "being asshole"?

And yet, here we are reading about Dave Cutler.

If it's true about Dave Cutler, then it reminds me of a saying I heard recently. Life is as simple as being better than your problems.

Dave Cutler is famously reluctant to talk to the press.

Any subordinate giving details about him would certainly earn the ire of the entire corporation.

We've had anonymous Microsoft employees talk here before, so it would not be a surprise. I guess he’s not malicious to the extent that he’s driving people nuts and causing a bunch of other people to leave, otherwise I feel like we would’ve heard it by now.

Because there are no retired subordinates after his 40+ years career?

I was struck by some of the descriptions in the article…

“It was probably the scariest meeting of my life,” Multerer said.

The team was “scared to death,” Multerer recalls.

So I found this comment interesting, because it gave me more information about questions prompted by the article.

In contrast, Jeff and Sanjay seem much nicer.

the book "ShowStopper" describes his behavior quite a bit. very difficult to work with.

Here’s a memory of Cutler from the DEC days https://minnie.tuhs.org/pipermail/tuhs/2019-January/017435.h...

Three hours of interviews with Cutler for the Computer History Museum:

Part 1: https://youtu.be/29RkHH-psrY

Part 2: https://youtu.be/SVgSLud50ss

Great article about a very significant accomplishment. Few of us get a chance to write an operating system--Dave wrote three. VMS was about as solid as you can imagine.

At the time, I wondered if there weren't too many features in VMS and now NT. But that is always the tension, isn't it?

One interesting fact that the article doesn't cover is that MS had to pay Compaq, interitor of DEC, something like $60 million as the result of hiring Cutler. There are very significant similarities in NT with VMS. All in all, it was certainly worth it.

As in, WNT = VMS + 1

As in VMS = RSTS + 1

You sure you got the joke? V + 1 = W, M + 1 = N, S + 1 = T

Ach! I did not.

How much would be his salary at MS? Not looking for an exact number :-)

As an aside, since it's getting on time for me to move on from my current employer, are there any companies doing these kinds of hardcore engineering projects any more outside of the defense industry?

I have spent most of my life (until recently) doing systems work (sometimes but not always big-iron embedded) and I kind of wish companies were still investing in ~30 person teams for big engineering projects. I've always wanted to work on a Windows NT ... but no more routers.

I find the persimitic locking model of NT handles to a fundamental design flaw in NT. is this something that came from VMS.

Sure the unix model of ‘user must handle errors’ is a pain, but it leads to more flexibility in the long run.

Care to elaborate on this pessimistic locking model of NT handles?

Basic you cannot write a file in use.

Pretty sure it came from DOS. This behavior existed in Windows 95 and earlier.

It didn't. I was admin on a VAX 11/780 back around 1980. One day I noticed two directories contained the same files and assumed they were duplicate do I deleted one copy to open up some disk space (we were using RM80 disks). Then I noticed that the files had disappeared from both folders, because they were hard linked.

I was like "oh shit" because they were system files. I called support and they helped restore the files from tape and then I rebooted. So I can honestly say that VMS is very similar to Unix in that respect.

Having hard links in the file system has nothing to do with whether handles can be shared.

Nitpick, reading this published in 2016:

> At the time, some questioned why Microsoft developed a 64-bit system; today most computers are 64-bit systems and even our phones will soon have a 64-bit operating system.

I've had to refresh my memory: the first 64-bit iPhone was released on September 20, 2013. It was "we have 64-bit since almost 3 years ago" for Apple in 2016.

(2016)

Thanks—added above.

Discussed at the time: https://news.ycombinator.com/item?id=11504102.

Shame NT is so buried under 12 layers of crap that people write it off and demand that we just go entirely for a Linux monoculture.

Dave Cutler is a better OS engineer than Linus Torvalds. Fact.

Is it necessary to compare them? I'm puzzled why we seem to do this everywhere. When some people see Messi score a goal, the immediate reaction isn't "what a great goal", it's "he's better than Ronaldo". Why can't we just appreciate greatness without comparing?

I hate it too. The tribalism around tech things and politics and nutrition and cars and just about anything feels so damned toxic.

While I agree that Dave Cutler was a very good OS engineer, it absolutely ridiculous to compare the work of an experienced OS engineer who already learned a lot by designing other operating systems and who also was free to design a new incompatible OS, with the work of an inexperienced senior student, whose purpose was just to reimplement the UNIX API's.

Even if the Linux kernel has evolved internally a lot since then, it has remained constrained by the userland POSIX interfaces, some of which are obsolete.

You cannot predict whether Linus, at his current age and experience (which only now are similar with those of Dave Cutler when designing Windows NT), would do a better or a worse OS than Dave Cutler, if he would be free to abandon everything else and to design a new incompatible OS, without legacy baggage.

Even if it had a nice clean design, I do not remember anything really revolutionary in the initial Windows NT, at least in the features exposed to the users.

Almost everything had a correspondence either in UNIX or in OS/2, but because NT could afford to be incompatible, most NT interfaces had various nice improvements, especially in asynchronous IO, event handling and security. Also because they were designed together instead of being accreted over a long time, the NT interfaces and types were much more consistent in the beginning, compared to UNIX. After other 25 years, Windows has accumulated again countless API additions and replacements, so the initial consistency is long gone.

I don't think it's true anymore than Linux is constrained by POSIX. POSIX is dead; Linux has been adding new non-POSIX APIs for probably decades.

Apples to oranges. Cutler designs new OSs from first principles, Torvalds implemented a version of an existing OS.

You might compare Cutler to Tanenbaum if you wish.

Could you please try to increase the quality of your comments? We're here to learn something substantive, and one-liners don't deliver.

https://news.ycombinator.com/newsguidelines.html

Eh? It was a worthwhile comment written in a tongue in cheek style. Just look at the conversations it spawned.

NT was a huge disappointment. To wit, they've never had a shell that even approximates Unix sh from years ago.

Are you kidding? Check out WSL - they've got the full bash working just like it does on Linux.

With lots of caveats and gotchas.

See comment from Microsoft guy linked here. https://news.ycombinator.com/item?id=18783525

Dave Cutler was the kernel architect and a shell has almost nothing to do with the quality of the kernel it's running on. Does WSL running on Windows magically make its kernel better? Would the old CMD shell running on Linux make its kernel worse? No.

It's a kernel, not an OS.

Reply on news.ycombinator.com