Hacker News Reader

Comment by throw310822 | Hacker News Reader

The other day I've clicked on one of Outlook calendar's copilot prefilled questions: "who are the main attendees of this meeting". It started a long winding speech that went nowhere, so I typed in "but WHO are the attendees" and finally it admitted "I don't know, I can't see that".

View on HN

Absolutely! There are so many scenarios where they could actually add some value, and they're fulfilling, like, exactly none of those?

Even in Visual Studio Enterprise, their flagship developer product, the GPT integration mostly just destroys code regardless of model output. I truly cannot fathom how any of that made it past even a cursory review. Or how that situation would last for over 6 months, but, yet, here we are.

And, again, it's fine with me: I'll just use Claude Code, but if I were a Microsoft VP-or-above, the lack of execution would sort-of, well concern me? But maybe I'm just focused on the wrong things. I mean, Cloudflare brought down, like, half the Internet twice in the past two weeks, and they're still a tech darling, so possibly incompetence is the new hotness now?

Imagine a circumstance where Windows Search was as good as Apple's Spotlight, and could integrate with cloud services to index documents, browser bookmarks, web history, maybe podcasts, etc.)

Hey Copilot, where is that document I was reading about the new network diagramming software Jacob is testing out?

Or Hey Copilot, my disk is getting pretty full. What software is taking up a lot of space that I haven't used for a while? Or are there any files I can move to cloud storage to free up space?

But no, instead it's just 'we're going to take screenshots of all your windows, OCR it, and index it, so that when someone infects your machine they can see your credit card numbers and pornography habits.'

It's so broken. I have a suspicion the AI teams are in some kind of an internal standoff with privacy/compliance teams, and Copilot simply never gets access to any tools or data.

Because if not that, then I don't know what. I can half-ass a better product with ChatGPT API and a PowerShell script, and I could've since GPT-4 was released; in fact the product can actually write itself, it's that simple to do a better job.

Being Microsoft, I'd guess it's less that and more Product Team A doesn't want to spend time building enabling features for Product Team B to look good.

The org chart joke, etc: https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...

> the product can actually write itself,

Used to be, that was just something people said. Turns out, we live in a future where that's literally true!

I have Copilot at work, it feels so useless sometimes. As an example, I had a report which I needed to make some batch edits to. I figured why not let the robot take a crack at it, so I clicked the Copilot button and spent a couple minutes describing what I needed changed.

Copilot tells me it can't edit my current document, but it can create a new one. I figured okay, Microsoft doesn't want to set it loose on the original, guess it makes sense that it requires a copy. So I said yes.

Nope. Instead of creating a copy of my document and editing it, it created an entirely new document which excised basically everything in the original report and replaced it with a very short summary - I'm talking 5000 words down to 500. All my tables and figures were gone, as was the standard report template my employer uses.

What utter garbage. Office productivity is a major use case for LLMs, and here the largest vendor of productivity software on the planet is happy to fuck it up.

The chatbot doesn't see the document structure; it sees tokens. When it generates a response, it's not editing; it's rewriting from scratch. Without a mechanism capable of precise diff/patch operations (like Cursor attempts with code), an LLM will always be a destructive tool for complex documents. Microsoft marketed this as an editor, but in reality, it's a summary generator

I’ve found it fairly useful in Excel. The suggestions to clean up data are pretty good and it’s spat out some quite gnarly formula on request

Compare this with Cursor or Supermaven. There AI acts as an extension of the cursor, understanding the code's AST. VS Copilot often feels like smart copy-paste that doesn't understand where it is Microsoft, owning VS Code and LSP, should have done this better than anyone, but they lost to startups on UX

> I truly cannot fathom how any of that made it past even a cursory review.

Maybe it's a fifth column group working to destroy Microsoft.

LOL

Since DECADES Outlook is _not_ capable of putting the first(!) just(!) useful datetime-stamp found in an email which says "calendar" or something in the text when pressing CTRL+R: This function creates a calendar entry from the current email with current receiver list, but default-date/time is alwsys set to the current even if recipients last ansewer was simply "lets do: 11am" - why does even this simple shit not work today? You dont need AI for this simple idea even, a robus / probability parser will do fine in 99% cases

Agree 100%. Don't train a 100-billion parameter neural transformer if you can solve it with a bash one-liner.

Use your human intelligence and save our planet!

I would just be happy to never have a ICS file every saved to my drive again. Whoever created the concept of a calendar invite saving a ICS file has created me hours of frustration.

It's so easy to ship completely broken AI features because you can't really unit test them and unit tests have been the main standard for whether code is working for a long time now.

The most successful AI companies (OpenAI, Anthropic, Cursor) are all dogfooding their products as far as I can tell, and I don't really see any other reliable way to make sure the AI feature you ship actually works.

Tests are called "evals" (evaluations) in the AI product development world. Basically you let humans review LLM output or feed it to another LLM with instructions how to evaluate it.

https://www.lennysnewsletter.com/p/beyond-vibe-checks-a-pms-...

Interesting, never really thought of it outside of this comment chain but I'm guessing approaches like this hurt the typical automated testing devs would do but seeing how this is MSFT (who already stopped having dedicated testing roles for a good while now, rip SDET roles) I can only imagine the quality culture is even worse for "AI" teams.

Yes. Because why would there ever be a problem with a devqaops team objectively assessing their own work's effectiveness?

Traditional Microsoft devs are used to deterministic tests: assert result == expected, whereas AI requires probabilistic evals and quality monitoring in prod. I think Microsoft simply lacks the LLM Ops culture right now to build a quality evaluation pipeline before release; they are testing everything on users

Microsoft: What? You want us to eat this slop? Are you crazy?!

50% of our code is being written by AI! Or at least, autocompleted by AI. And then our developers have to fix 50% of THAT code so that it does what they actually wanted to do in the first place. But boy, it sure produces a lot of words!

I've fooled around with some vibe coding on several LLM's like Claude, Gemini, and ChatGPT with some pretty decent results.

Since I have a full Copilot license at my corporate day gig, I figured I would try using Copilot for a basic static site. Nothing too hard, and something that's been handled easily with the other LLM's.

The prompt was pretty basic just to get something to start working with. "Build a four page template. With a home or index page, two pages of content and a contact page with a responsive slide out menu from the left hand side of the page."

It ran and put everything in a folder. I open the home page and everything was broken. I opened the files in VS Code and saw this:

    <ul class="drawer__list">
      <li>index.htmlHome</a></li>
      <li>services.htmlServices</a></li>
      <li><a class="nav-linkeduling</a></li>
      <li>contact.htmlContact</a></li>
    </ul>

And then this:

    <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Home · Acme Web</title>
    <meta name="description" content="Accessible, responsive starter template with a slide-out menu."/>
    <linkts/css/styles.css
    /assets/css/styles.css
    </head>

I mean, if you can't even this right, I don't have much hope it can do anything more complicated. To say this was pretty sad is an understatement and clarified how far Microsoft is behind other LLM's.

I guess this is code people working at MS would commit to a useless file in order to simulate work done with AI support.

There are cheap 1b open coding models that do a far better job.

Not a problem. We can call it HTML6 and make browsers interpret it correctly.

Kudos to you. I never use new buttons out of fear of something irreversible happening, like sending a random email or deleting something. I still feel uncomfortable with the Gmail UX, I would _never_ use a ”hello iz magic ai”-button.

Me: Can you access my inbox and Teams messages?

Copilot: Yep!

Me: Please find any items in my inbox or sent items indicating (a) that I have agreed to take on a task or (b) identifying me as the person responsible for a task, removing duplicates and any items that I have unambiguously replied to via email or Teams. Time window is preceding 7 days.

Copilot: Prints a list with, at best, 5% accuracy

I know some folks have the peculiar idea that search is dead in favor of AI, but if AI can't accurately find information, it is useless. As near as I can tell, Copilot finds 3-4 items (but rarely the SAME 3-4 items across runs) and calls it a day. It just seems like nobody is actually testing any of this stuff. Microsoft is actively destroying its credibility because it's offering a tool with a party trick but is utterly unreliable. I will, therefore, not rely on it.

It's a generalization problem. We can train LLMs that 'know' a lot of stuff in the global sense but the tasks that are interesting to people require the LLM to know a lot about you and your world in a very specific sense. The technical problem is that it's all corner cases and that's impossible to scale right now. No amount of context window is going to get you there either.

I respectfully disagree. The scenario posted by the GP is very clearly an "easy" task of basically sentiment analysis. The only thing it requires is a way to query your message history, then it just goes through them 1-by-1. I would expect any of the current SOTA agentic models to handle this with ease.

Sounds like Siri - unable to control much of anything on the iPhone outside of reading/sending text messages and setting alarms.

It also can initiate phone calls and play music.

And those are really the core use cases for AirPods, HomePods, and CarPlay, the contexts where a hands-free eyes-free interface is most useful.

And that’s ok. Those are core features of the phone that absolutely must work reliably and consistently. Far better to do a few important things really well than a hundred things execrably.

However the other day I asked the Gemini assistant on my phone to check the birthdays in my calendar, get all their dates, then make a graph of how many fall in each period with a 15-day moving average. It did everything as instructed including writing a python script to generate the graph, then discussed the results with me :)

I would expect Siri to be able to do anything on the iPhone that I can - change settings, report stats, kill/launch apps, etc.

It would be nice if it could control 3rd party apps too, like GMail, but being able to control the stuff that Apple themselves have built doesn't seem a lot to ask.

Apple is somewhat privacy focused, allegedly. I would imagine that letting an unpredictable LLM read whatever system data it wants is not something they want to partake in.

Maybe there are different concerns now that Apple are apparently going to be using Google LLMs for Siri, although I'd have thought that any access to system settings or other iOS functionality would be via tool calls with user opt-in.

However, the original Siri, obtained from SRI ("Siri" = SRI) was pre-LLMs, and there should have been no more concern over accessing system settings or controlling apps than things like reading/sending messages. For some reason Apply completely dropped the ball with Siri; initially it was expected that they would expand it into revenue generating areas like restaurant reservations etc, but then nothing. I'm not sure what documentation exists, but even today it's not clear what Siri is actually capable of.

We all know they Apple lack the capability, not a choice.

I asked Microsoft 365 Copilot to create a new word document for me (since they have hidden the link on office.com) and... it refused to do that.

Edit: Just tried again. It refused to do it. I mean WTF.

Reply on news.ycombinator.com