Bringing the power of AI to Windows 11 | Hacker News Reader

Bringing the power of AI to Windows 11

blogs.windows.com

137 pointsmakepanic115 comments

> We have added native support for additional archive formats, including tar, 7-zip, rar, gz and many others using the libarchive open-source project. You now can get improved performance of archive functionality during compression on Windows.

This is easily the most exciting part of the whole announcement.

Yes, but why did it take so many years?

probably because they were spending all those years putting annoying ads in the OS and start menu.

MS has enough developers to build two features at the same time.

But not to fix annoying and abundantly obvious bugs, apparently

1. Ads in the Start Menu.

2. Ads in the rest of the OS.

You would think, yet WinUI 3.0 is years away to achieve parity with WinUI 2.0/UWP tooling.

Hopefully it reaches parity with WPF before the OS gets EOL'd

Or with the pleothora of AI announcements at BUILD, they will get CoPilot to do it for us. /s

So what you're suggesting is that they won't put any more annoying ads in the OS? ;)

The Big WinRAR lobby /s

What about zstd?

Update: looks like no: https://github.com/libarchive/libarchive/wiki/LibarchiveForm... :(

The wiki page is just out of date. It's been supported in libarchive for several years, though it's optional at compile time, so this new Windows build might not have it (I didn't check).

Interesting, thx! Fingers crossed, as zstd is awesome.

Wonder if it includes support for encrypted 7zip/rar archives, including support to add/modify. From libarchive notes it looks like zip format encrypted archives are supported.

Or you could just install 7zip.

Where is zstandard or lz4 on ntfs? Meta and data checksumming and autorecovery? New and improved filesystems?

You mean ReFS?

ReFS doesn't have compression at all and didn't have boot though it looks like win11 is getting it now.

Make sure to watch the embedded video.

A new permanent taskbar button opens an OpenAI LLM trained to use Windows 11.

You ask the LLM to do what you want -- "summarize this email and send the summary to my boss."

It works with all applications. It sees what you're seeing.

As a longtime Linux user, it pains me to say this:

It's... beautiful.

...

But it's also scary, because it's not under your control. Ultimately, it obeys only the bureaucracy at Microsoft.

In that sense, it's like "MOTHER" in Ridely Scott's Alien and "HAL" in Kubrick's 2001: A Space Odyssey.

Yes, I really want OpenAI observing all my actions closely and 'taking notes' of everything I do.

I would love this in Linux, but only if it works offline. Not a fan of sending everything I do to a remote machine.

> It works with all applications. It sees what you're seeing.

Sounds like it will also see prompt-injection malware hidden in plaintext documents.

Oh yes. This is going to be glorious to watch. Before, prompt injections were mostly a joke. With this, they will truly become a proper, new type of attack vector.

It's also under the control of anything that it finds in its current "context window" that it decides to act upon, not just Microsoft...

https://greshake.github.io

https://kai-greshake.de/posts/in-escalating-order-of-stupidi...

Having spent most of my ChatGPT time trying to get it to correctly write powershell/batch scripts to do different things in Windows, I will be fucking amazed if this button actually does half the things you tell it to.

The problem is you're a programmer. It will work right if a non-programmer does the prompting. How else do you think you're gonna be replaced?

> But it's also scary, because it's not under your control. Ultimately, it obeys only the bureaucracy at Microsoft.

perfect summary of modern Windows

This sounds like a massive security hole to me, especially if this runs in the cloud. Is this thing sending _all_ context of the programs on your windows computer to MS?

I'm also a bit curious how they will actually pay to have millions of these LLMs running and how much co2 it will output. This doesn't exactly fit their narrative about the new power saving features in their consoles.

You ask the LLM to do what you want -- "summarize this email and send the summary to my boss."

Follow it up with "And send out my resume, because I can apparently be replaced by a computer."

> It works with all applications. It sees what you're seeing.

Does it also snitch on you to your boss? "Here is my assessment of the work habits of cs702 for this week."

> It sees what you're seeing.

In the video the user drag&drops a file into the copilot window. How do you know that it can see your screen? I believe "works with all apps" only refers to the third party plugins.

You: Clippy, Open the CMD window please. Clippy: I'm sorry, I'm afraid I can't do that. This integration is too important for me to allow you to jeopardize it.

As a MacOS user, this looks really cool. I just hope Apple figures out how to do it, which I don't have a ton of hope for since they've flubbed Siri for so many years.

I mean seriously, the fact that Microsoft is this far ahead with AI (because of their partnership with OpenAI), comes off as a massive failure on Apple's part. Not only that, but Microsoft's continued positioning as dev central (Github, Github Copilot, VScode, and more) while Apple dwindles with docs that have been lacking in quality for years and only providing the bulky Xcode as a code editor is frustrating me as someone who loves coding on Mac. I love using this unix-like machine but not having to sacrifice UX by using linux. But as time goes on it makes me wonder if Apple will ever start focusing on devs again, and now AI!

I'm hoping Apple is working on something that run locally or potentially in some privacy-focused cloud approach but honestly I can't see any way to feed data into an LLM that doesn't result in leaking a ton of data about you if someone peaked at the input/output.

The good news is I've been impressed with running various AI-type things (Stable Diffusion and llamda.cpp) locally on my M1 Max Macbook so if anyone can pull this off locally I think it's Apple.

I do agree Apple's hardware is still top-notch. Their home-made chips have been amazing to use and I have no worry they'll continue to push the envelope there. And even iOS I feel is a solid piece of evolving software (other than Siri). It's MacOS and their lack of focus on devs that worries me.

> It's MacOS and their lack of focus on devs that worries me

You and me both, System Settings alone is a disgrace that I'm embarrassed for them that they shipped and it's only one of many recent failures related to macOS.

> I just hope Apple figures out how to do it,

This was done with bash mimicking similar functionality. I am guessing it will be really easy for the Apple developers to make a really nice version since it's this easy to do a hack job [1].

[1] https://github.com/realrasengan/blind.sh

They'd have to fight with legal to distribute something newer than Bash 3 though.

I believe they'll come to terms with the challenges soon. Siri faced limitations because it didn't have access to the extensive 'internet of knowledge' that Google could utilize. Apple tried to close this gap by partnering with Yelp and Wolfram Alpha, but it didn't lead to a comprehensive solution.

Nowadays, the 'internet of knowledge' can be seen as a repository that's readily accessible. It's challenging, but not impossible, to create an AI system that outperforms Google Assistant. Developing something on par with GPT-4 in the immediate future? That's unlikely. We've moved from an era akin to sailboats to one more like steamships, where the pace of development is much quicker. OpenAI currently has a significant lead, but in this rapidly advancing field, the playing field could change faster than we might expect.

AI is great but how about fixing start menu search? Search that was perfectly working on Windows 7 but was lobotomized in subsequent versions.

The context menu too!

How about the settings menu?

(edit: tbf while I don't have Windows 11, this feels like a safe presumption)

It's easy enough to restore:

https://answers.microsoft.com/en-us/windows/forum/all/restor...

Even in Win7 it wasn't particularly fast. I don't get how that one random person working on that "Everything" search utility was able to build instantaneous search on Windows, but Microsoft themselves couldn't figure it out.

One assumes that it's loaded with telemetry and bing-first searching, so it's design for data exploitation of customers rather than to be a menu

In fact, if you disable web search via a registry value or group policy (don't remember which), it becomes a lot faster. Not Everything fast, but significantly faster nonetheless.

It’s the next great thing about surfacing semantically relevant bing ads in your start menu and sending Microsoft summarized “diagnostic data” about everything you do and read to “improve their product experience.” Enjoy!

Maybe the AI will be able to delete files and directories faster than me doing it myself with a magnetised needle and a steady hand.

They’re still working on perfecting the system that sends all your data to the cloud so you can search it again.

More privacy setting changes that will default to spying on you and sending everything you do and all you data to Microsoft. I wish I could look at this kind of announcement and see something positive, but it's all just thinly veiled espionage.

I can't agree more, totally opaque data will almost certainly be sent to an ever growing number of locations. If I so much as ask about it I'll be labelled a nutter.

I'm 100% in for this purely for the fact that it looks like it will actually tell you / provide a button to perform the action. I am a Windows fan and use it as my primary operating system, but I cannot tell you how irrationally angry I get when I end up in the new control panel and there are links like "find out how to do X" which just dump you into Bing search. Microsoft already knows the answer, that answer is potentially relevant to the specific version of windows I'm using, but they need to pump up Bing's numbers so they send me there and honestly the results aren't always that helpful.

I don't think your anger at that is irrational at all.

My issue with this is that it feels less like a new toy to play with and more like a new worm that is meant to hook me, my workflow, and my data into their silo. I'd like to see better demarcation in windows tooling when it comes to how they're managing user data. Similar to how developers tend to colour code DEV and PROD environments so that they don't accidentally alter PROD. I'd like to see colour coding on microsoft tooling that delineates "local data that is processed locally, and is 100% private" processes from "local data that will be processed in the cloud and possibly stored for training later without you being able to know" processes. The moment data leaves my local network, I assume it is compromised, so having clearer delineation will help me know what tools to avoid.

I'm pretty sure I won't get this offered since I am not signed into any Microsoft account on my Windows 11 installation. Or, I get it offered, but requested to sign in.

I have zero issues with signing GitHub Copilot into my GitHub account to use it with Android Studio or VS Code, I just don't want the OS to be logged into a cloud account which I don't really own and where I don't know what's getting synced to it.

Embrace the future or use linux.

What do you mean or? the future is linux and it is here

I wish I had that option, but unfortunately I'm forced to use Windows at work.

I used linux for many years. Ultimately, I like my laptop's touchpad and trackpoint to remain functional. Believe me, I have tried.

Embrace the future -> Extend the future -> Extinguish the future

Thanks but no thanks, it is a long time ago I took a different path which leads to a different future, one which is not likely to be extinguished.

Why does it matter if you are signed in? I mean, it's not like not singing in into microsoft account somehow prevents them from collecting telemetry. It's might be /slightly/ harder for them to match your profile, but we are talking about data collection scale that this is arguably irrelevant anyway

There are two things to note here:

1. A cross-application connectivity layer that pipes data and actions between apps

2. A natural language interface to control #1

Thinking about them separately is useful, because although chat is the new UI hotness, #1 is valuable on its own and the two can potentially be deployed separately.

As presented here, I suspect the natural language interface will be faster and easier than buttons for operating the cross-app layer for complex queries, but potentially slower than operating buttons for simple things (like "start dark mode").

But personally, I believe #1 combined with some AI context awareness is more powerful of the features.

...

And btw, I left Apple last year to build a local-first and developer-extensible assistants for the Mac that's pretty similar. If this interests you, would love to chat (email in profile, as well as a waitlist).

I was also thinking about how they could've integrated this into every app, but I figured that simple tasks would be more reliable across all apps if they did AI-recognition on screenshots of the desktop + injecting data like "Installed Apps", then by the end of it output virtual keyboard event or a virtual click at an X/Y coordinate.

Just some afternoon postulating haha

#1 has existed in Windows and other OSes for ages: accessibility APIs and UI test tooling (called UI Automation in Windows).

It looks like this is a high level interface more like Intents on Android, e.g. in the video it opens the Logo Templates page of Adobe Express probably without automated UI interaction.

This looks like it could be great. I use GPT-4 all the time, so a system-wide integration and especially assistant capabilities sound nice to me. And yet I cannot make myself excited for this.

I can't get past the feeling that Windows Copilot will be used in slimy anti-privacy ways. I am somehow very sceptical that it will only use the screen's context as context for the user's queries. Probably because many new Windows 11 features have been anti-consumer, and Microsoft have been consistent at building this kind of reputation for years. It has numbed all of my excitement for anything Microsoft does.

Continuing the justified, if cynical take, in the near-term I'd worry less about privacy, and more about ads. Microsoft has a track record of putting overt and covert advertising directly into the OS. With deep system-wide integration and awareness of commercial applications, how long it'll take before Windows Copilot turns into a sneaky salesman, preferentially suggesting commercial tools and solutions of the highest bidders, while Microsoft gulps up a firehose of money coming from approximately every company of the planet paying to slot themselves into model fine-tuning?

Soo instead of finishing the migration of the old Control Panel to the new Settings app they slapped some ML onto it and called it a day?

We need something like this on Linux, maybe powered by Vicuna. I’m not sure if the current batch of LLaMA variants is coherent enough to work as a digital assistant, but my gut feeling is that a little fine tuning on tool use might be all thats needed.

The hard part I think will be integrating the LLM closely enough with the other programs on your machine that it's actually useful in that context, and not just a glorified chat window or a text interface to do things you can already do more easily with a KB+Mouse.

In a perfect world you give the LLM a python interpreter and it does the rest.

Realistically, with the LLMs we have today, the right approach is probably to curate a set of APIs it's allowed to interact with. Some basic file system access and FFmpeg support would be extremely useful on it's own.

Linux is fundamentally not monolithic like Windows, but maybe some DEs could expose hooks for LLMs to use.

There is also the performance issue. Right now the task energy/memory usage of llama implementations is very high, and it takes some time to load into RAM and/or VRAM. It seems Microsoft is getting around this with cloud inference, and eats the hosting cost (for now).

> little fine tuning on tool use might be all thats needed.

Maybe I am interpreting this wrong, but LORA finetuning is extremely resource intense right now. There are practical alternatives though, like embedding databases people are just now setting up.

Plugging a llm into dbus may take you surprisingly far.

Fine tuning is moderately intense. Going from base LLaMA to Vicuna 13B costs about $300.

The compute requirements are reasonable, but the memory requirements are extremely high, even with some upcoming breakthroughs like 4 bit bitsandbytes.

How does accessibility work on Linux?

I don't know if I'm quite willing to go there yet, but LLMs are barrelling towards a level where, if the pace continues, they will easily be looked upon as the most import advancement of computing.

The fact that training a statistical model on terabytes of text can lead quite naturally and obviously to what is shown in the video in the webpage is shocking. The shocking part is how natural it feels, and how I know this is something that can work.

Congratulations to the teams that have been going down this path, this is wildly impressive stuff.

Can the AI fix my taskbar and ungroup task buttons or will i still have to use a 3rd party program?

There's a section in the linked article about improvements to the taskbar, including the following:

> You can now quickly identify and access any instance of each app housed in the taskbar with just one click. All instances of the app are ungrouped with labels on the taskbar.

It makes it sound like you need to click a group first to ungroup it, but maybe there's an option to keep them ungrouped by default. I guess we'll see. You can catch a glimpse of what the labeled windows in the taskbar looks like in the Dev Home video embedded in the article.

I can't wait for somebody to figure out a webpage that tricks this LLM into deleting C:\Windows.

Edit: A less harmful proof of concept could be, say, turning volume up to 100% and opening fifty Edge windows with the Baby Shark video.

> AI-Generated review summary: We are making it faster and easier for customers to scan reviews for apps by using the power of AI to compile thousands of reviews into a simple summary, enabling customers to discover new content with ease.

Would be cool to have an option to actually read those reviews. If I open Firefox page in the Store app it says 420 ratings but I can only read a single review. If I press 'See all' it just shows an empty page. If I open Firefox store page[0] in the browser it says 'No one has reviewed this application yet. Be the first to add a review.' Reviews seem to be split by country or by some other criteria for no good reason.

[0] https://apps.microsoft.com/store/detail/mozilla-firefox/9NZV...

This is great, but I wish they'd focus as much on fundamentals. The bloat is tangible and measurable.

A feature designed by the marketing department. Hopefully, we will be able to turn it off like Cortana.

Microsoft and Windows has lot of neat stuff going on. But something happens in the productization process and the end result is a trainwreck. Large part of that is the distinct lack of trust that we have, both as consumers and technologists. Lot of fancy stuff gets built, then remains unused and unutilized, which inevitably leads the thing to wither and die. If MS wants to stop FAANG etc from eating their lunch they need to figure out a way to reverse the trend. And that doesn't happen with gimmicks like this, no matter how cool they are.

I'm not sure Google is eating Microsoft's lunch right now... Microsoft seems to be coming to the lunch table and starting fights for Google's turkey sandwich (search).

Office is the bread and butter of MS, and Gsuite is very much eating into that.

edit: I wouldn't be surprised at all if Amazon released an office(/groupware) style suite also soon. It definitely would be an match, they already have lot of user-facing enterpricy software

This might actually be a better UI for finding settings than their real settings app cause I swear I can never find anything in it and they move stuff around and rename things every other update

Would be great if it can help you find how to do things in Office apps too, because it's the same problem. Those apps have had so many redesigns over the years, Googling for anything more complex than the most basic of usecases is impossible.

I'm not the target audience, but reading the comments made me curious so I watched the videos and the examples were disappointing. They all require more steps than my current workflow and it has to go through some third-party server to achieve it.

I was expecting more than suggestions for playlists and slack hooks.

Cortana is back: this time with much less personality and yet more ability to hallucinate facts than ever.

I honestly don't see any benefits to the Dev Home app as is besides Dev Drive, which could be cool if it is that simple to just create a drive that is sandboxed.

Perhaps the community will whip up (most likely not), extensions for stuff like Sentry or CI/CD. Could be a neat thing then.

After their move to train AI off of Github repos I am deeply skeptical of wrapping up any code with a neat little bow for MS to further analyze.

Is this limited to Win 11 Pro edition only or also available for Win 11 Home users ?

Everywhere they are mentioning coming soon to Windows 11, but considering they are projecting it as a developer focussed instead of for normal people i get the vibe it is not for Home edition.

I was already uneasy about the day when my employer makes us switch to Win 11. This makes me even more uneasy.

I hope it's possible to remove the AI controls (especially the copilot button) from the UI.

Meanwhile, Apple users still fighting SIRI that can only provide answers to 1% of questions at best, given you are lucky and it was able to understand your accent.

Maybe bring the power of in-house QA to Windows 11, please?

If I'm reading this correctly, it's going to upload anything I copy/paste to their own LLM.

We have confidentiality agreements that I'm not sure would pass this.

Err. F$#k?

This is a significant announcement.

Note namechecking every non-Apple CPU vendor and NVIDIA - but not mentioning AMD/Intel GPUs. Another note: Lots of small improvements, but not allowing users to move the Taskbar (to Microsoft, some things are important). Oh, and they haven't abandoned ReFS - so will it be available without Windows Professional for Enterprise etc.? I can't see many devs paying for that alone.

It can see what I see? So it can see that unlabeled button on the screen and click it for me? It can see the game area I'm in, and can send controller input events to get me where I need to be to do the next part of the game I can play? I can describe this barely accessible Winget GUI app to me and check update all for me?

Now that would be pretty freaking cool!

Will the AI allow me to disable the big blue "bing chat" button that shows up every time I try to search for something?

Here goes me never using Windows again for anything serious. This is a ticking time bomb of privacy and security issues.

It's a great set of updates, and it's especially telling that "privacy" wasn't mentioned even once in the whole article.

I'm impressed with the progress.

I'm also scared about privacy of the data. How will that work with installing windows in business environments with sensitive IP?

Dev Home dashboard seems neat. Although not sure how worth the rest of it is. ReFS is nice I suppose.

We are so very close to 1) AI escape and 2) AI reproduction. I don't see how broad swaths of humanity stand a chance against ambitious capital holders in combination with AI.

Great. Pretty soon I'll be able to say "Computer, enhance!" and it'll hallucinate whatever details I ask for in my JPEG.

Reminds me of that spoof commercial, about Microsoft designing the artwork for an Apple mouse.

am i the only one who is curious about how they are going to charge for this?

Ads. Ads everywhere.

customers will pay with their data

Everything is an on-ramp to Azure, Github, M365, etc.

“Bing, fire 5,000 of our workers with the lowest productivity scores”

You’ll need an AI tool to summarize that press release!

Thanks, I hate it.

The videos in the article are absolutely slick!

>horrible OS designed for consumer cattle

>now with AI

Now you can actually talk to Big Brother!

But the Big Brother wouldn't listen.

Reply on news.ycombinator.com