Hacker News Reader

Hacker News

Comment by ryanjshaw | Hacker News Reader

Full thread

ryanjshaw

I recently saw a (pre-recorded) demo of M365 copilot. Here's some stuff it covered that the article doesn't attempt:

* Asking Power BI to analyze some employee data and generate a Power BI report page

* Asking Power BI to re-theme the page to match a different Power BI report

* Asking Power BI to refine some of the page content to answer a particular question about retention

* Asking Word to generate draft minutes from a Teams meeting transcript

* Asking Power Apps to create an app and iteratively build out features

* Asking Power Automate to map out an automation flow

This article seems to be focused on a very narrow understanding of Copilot's capabilities. I'm waiting to see if the above use cases are fact or fiction.

View on HN

The article is based on the author's own use case and understanding of Copilot's marketing. I don't think he used Copilot inappropriately, he just had an uncommon need that the tool badly failed to meet.

I will add that Robichaux is far more diligent and skeptical of Copilot than most people who use it, including Microsoft. MSFT's public Bing demo had a ton of awful mistakes, which is shocking: I assumed MSFT would have faked the demo! The fact that they didn't suggests they have a magical and undeserved belief in OpenAI's technology.

I suspect the demo you watched also had a ton of mistakes that went unnoticed - if Power BI makes a pretty graph with a plausible trendline, how many people are really going to go back and check the underlying code, making sure it's doing the right thing to the right data sources? If it generates draft minutes from a meeting transcript, but screws up 5% of the items, those errors likely won't be noticed in a quick spot-check, and will end up misleading people who didn't attend the meeting. Robichaux's post brings up a ton of issues that affect any use case - the tool just is not reliable.

The article is about Microsoft 365 copilot. Copilot for power automate or power bi are separate products. This is probably poor marketing by MS. Additionally, they've rebranded their chatbots as copilots, too. If you're in IT, and your users are asking for "copilot", good luck.

In my experience with copilot for power automate, the demos you see are mostly smoke and mirrors. I tried to recreate the demo as exactly as I could, I couldn't. Maybe they can help absolute beginners, but if you know the first thing about power automate, you're better off with a web search than asking the copilot for help. This is still v1 though (if it's not still technically a "preview"), but it really should be marketed so hard because it is currently not delivering.

People will get fired due to hallucinated bad reports, won't they.

Microsoft specifically point out it's named "copilot" because it's there to assist you, not replace you. You still need to check the work and decide whether to put your name behind it or not.

This is emphasized in every "What is Copilot?" article or video I've seen.

The not so fine print is Copilot may return garbage and it's up to you to confirm the results.

Well, I wouldn't really expect it.

Its customer data, and it can technically be anything. Imagine making a system that is expected to be reliably excellent at generating things within that domain. It is already a very hard problem that I can confidently say is unsolved, and probably not even as good as a human can sort things. Even in a good dataset that is unstructured and unlabelled you will face issues in just a single domain (e.g. vision tasks trying to classify a variety of vehicles), imagine that being extended to a wide variety of subjects that can mix and match.

I imagine at most, it'll be a reputation hit.

Machine-generated or people-generated hallucinations? You would be surprised at how bad most existing reports in business are - and people don't get fired for those.

This tedious, snarky response is neither clever nor insightful.

Like, we get it - you resent your coworkers & consider them beneath you.

Hopefully, your unique, elite output keeps you employable for a few quarters longer than them.

No, it’s a legit gripe.

I’m routinely asked to make reports, but things stall when I ask, “so what do you actually need to see on the report, and where should the data come from”. A lot of people out there are kind of cargo cutting their way through things.

> cargo cutting

Cargo culting. Though cargo cutting would be interesting to watch :-)

ooops. Missed that typo.

I didn't read the comment as especially snarky.

Your response was both snarky and personal which I think comes across much worse.

I don't think parent was trying to be clever. In my neck of the woods various reports have to be massaged hard to account for some of stupid data decisions in the old times.

You would be amazed how many issued we identified so far and our mgmt uncharacteristically wants us to bring up issues. Some companies kill their messengers.

So yeah.. I absolutely buy that a portion of business reports are of low quality.

I have not gotten fired for a bad report, but I certainly made mistakes in reports/Excel sheets that were intended as customer-facing deliverables. The good thing is that my coworkers checked my work and often noticed mistakes - not always, once an angry customer noticed my mistake, and my boss chewed out my coworkers for being lazy. Obviously humans are not foolproof.

But if I was making mistakes, say, in 5% of my facts and figures...I probably would have been reassigned to other work. 5% is way too high for an Excel sheet: it makes the model completely useless. It often takes more effort to audit and fix 5% of busted Excel cells than it does to just rebuild the model from scratch. GPT-4/Copilot/Gemini/etc almost certainly make more than 5% errors for any real use case.

I am sure there are many bad organizations with lower standards than the organizations I worked at. My concern is that good organizations will adopt technology like GPT, inadvertently lowering their standards because they trusted a machine they didn't understand. It's very worrying that good people are rationalizing this choice by saying "well humans are also unreliable" without putting specific numbers to it. 5% errors in document summarization is not good enough for human workers, and it certainly shouldn't be good enough for software.

People are already killed and jailed due to garbage machine-generated data, so why would firing be off the dump?

I didn't test any of these use cases directly. In the sandbox I have access to, I don't have any Power BI data (or Teams transcripts for that matter), and I'm not willing to move any of my real production data into it.

Out of those use cases, though, four of them are just using Copilot as a natural-language front end (e.g. re-theming in Power BI). I would expect that to work well and, frankly, don't think it's that interesting as a use case. Sort of like Copilot for Windows… why do I want a natural-language system to tell me where setting X lives?

You may have seen a video that was more of a "vision" rather than actual current capability, e.g. I know that PowerBI Copilot does not support re-theme. I also know that Power BI Copilot does not look at your actual data within your dataset, but does make use of tables, relationships, measures, and column names

It's possible, that's why I want to see the real thing or a review that tests these claims.

Its a fact. You can open Power Automate Cloud which is basically Scratch for Grown Ups and type in “every day at 3 pm please check if i have an email from Jane Doe and if not then save a file to my cloud drive called “noemail.txt” and it basically does a halfway decent job , not chatgpt level but its getting there

Reply on news.ycombinator.com