ESPN AI recap of Alex Morgan’s final professional match fails to mention her
awfulannouncing.com
awfulannouncing.com
It is and was mostly done for search ranking. The more seemingly applicable content, the better your SEO.
This situation has probably happened several times before AI but has gone unnoticed or noticed to little fanfare. It's more indicative of ESPN not having its finger to the pulse and ensuring that one of the copywriters manually updated this particular article. It's not too surprising to me. They've always been known to favour quantity over quality.
source: I worked as a developer on the tech side of a company with this kind of content
To be fair, I don't think it's gone unnoticed at all
Is it though? My takeaway was that they lament the terrible "journalistic" standards that ESPN embraces.
Clearly it was never "there" yet though previously, and obviously still isn't when this article is what's generated. You can tell that a lot of sports articles are essentially "fill in the blank", which is why they get the AP stories up right away, and then have their actual beat reporters come out with something later that night, or early next morning.
The fact that it is profitable to make this generic sports-lingo-laced content is, on its own, pretty depressing.
You're missing the whole middle part about users and revenue. SEO alone does not a Wall Street valuation make.
Maybe the indictment here should instead be about discovery and how we share ideas and utility with one another.
Are they? Or are you over-estimating that part?
Do you have an example of a company whose valuation is driven principally by eyeballs? Not metrics tied to eyeballs, e.g. ad revenue. Just eyeballs.
Do you have an example where the metric is eyeballs?
This was a thing once! But to my knowledge, it isn't the sole basis of any company's valuation today.
So people will see content ESPN is actually paying meaningful amounts of money to create ie videos.
If you are faster and have the best SEO at that point in time then that means everyone trying to read about Lebron James retiring is going to search it in Google and is going to read your article first, and you're getting the ad revenue. It's a 90/10 situation. The "top" website in the rankings is going to get 90% of the clicks.
You can only be the "top" if your SEO to that point is also the "best". And to have that you need to have all this generated content.
Unless anything extraordinary happens, every post game interview features the same questions with the same answers and every article about the game looks the same with only the names of the participating players and their stats changing.
Doesn't sound so.
>web-based sports content companies have been automatically generating content articles for at least a decade
There are basically two types of content when it comes to these sports recap articles. I'm also excluding opinions/editorials because those are completely different.
1. low profile, not-so-popular content. eg. A Canadian Football League preseason game between the Toronto Argonauts and BC Lions
This was still generated using a template. No human intervention except maybe double-checking it before publishing it. This was before AI, too.
Often it was there for SEO or just updating people via a headline
2. High profile content, eg. Jeremy Lin puts up 38 pts in a game
This is often one of:
- pre-written, especially if we expected to happen that day - withheld from publishing if there was auto-generated content but something crazy happened that game and then quickly re-written and released. Usually this would still at least be pre-written before the game ended and details that needed to wait for the game to end were filled in seconds before/after the game ended.
There's just as much labour put into it now as before. AI is now generating the low profile template instead of the madlibs we did before. The value add is that it's probably a better read, so you actually get user engagement, and better SEO than before.
And this same issue highlighted in the OP article would also happen without AI.
Odd question but is that why the banal reporting on so many websites is such utter shit? Like in all dimensions, it gets facts wrong, it's usually rife with spelling and grammatical errors... when I spend too much time online I start feeling like the sole person on the planet who gives a shit about writing correctly.
Good news take time and journalists often don't have that luxury.
Broadcast media outlets fail at that. I don't know much about sports but even I can tell that people on TV are speaking in vague generalities so they can never be proven wrong. I get told that the team that lost "didn't want to win" as if people getting paid millions a year to win at sports aren't motivated.
When people are afraid of being proven wrong above all else, they avoid making any substantive claims. If someone isn't afraid of that, they'll make more interesting predictions I can't get elsewhere, and eventually it'll be a more enjoyable product because I can make fun of that person for being wrong.
If I was into online sports betting, I would want all the info I could get. Since I'm not, what I want is video replay of what I missed, that's where ESPN lost me.
Note that I have no knowledge whatsoever of how ESPN work, I'm inferring from what I've seen elsewhere.
On the other hand, the rather simple task of "here's a set of goals, their times, who made them, who assisted... turn that into prose" could even be done without LLMs with a deterministic algorithm, and may very well have been in this case. Some of the grammar issues in the OP feel very pre-LLM in nature, like a combination of substitution rules gone awry.
Now, could you create a system that repeatedly interrogates the statements made by a first pass of an LLM on summarizing a long transcript, and comparing those results against structured data you know for accuracy? Would this lead to richer content and accessible error rates relative to the simpler approach? Would this be the type of thing that the best machine learning engineers in the world could probably prototype over a hackathon? The answer is very possibly yes to all three of these. But it's far from low-hanging fruit for any sizable, risk-averse organization. It's very difficult to fight against "the thing we have is imperfect, but at least it never gets the facts wrong."
Here is a transcript of a soccer match. In the style of an experienced professional sports reporter, please write a 200 word article about the match.
Its summary starts, "In her final professional match, Alex Morgan delivered a performance filled with emotion and resilience, though her San Diego Wave fell short in a 3-1 loss to North Carolina Courage. The game at Snapdragon Stadium in San Diego was more than just a contest; it was a tribute to one of soccer’s most iconic figures."
It did get the score wrong. Here's the rest:
1. https://chatgpt.com/share/de8c60d1-69ab-4291-99dc-d4d95af3d3...
One thing to try is only using the post-match commentary, which starts after the line containing 'final whistle.' When I did this, the result was more factually accurate, while still focusing on Morgan.
https://chatgpt.com/share/9c122702-b46f-4e5e-bd05-e063a74126...
If some teenage intern was given a table with the goals scored (player and minute mark), they would have written a similar article... but that's definitely not a good excuse for news orgs and sports sites to just use generative AI for everything, so I can see why people are annoyed with ESPN.
It couldn't, which is the problem. One of the big selling points of generative AI is to cut people out of the process of writing. If someone actually has to watch the game and describe what happened in the prompt, what's the point of the technology at all?
This is what an application of generative AI looks like: using low-quality input to generate something that looks like an article. This is our glorious future, brought to you by OpenAI.
And even then, it's unclear that the AI would be smart enough to identify "retirement" as something that should be called out in the new article without specific prompting.
It seems like it highlights a clear avenue for improvment: rather than just feeding the events of a game to the LLM and asking it to summarize them, it seems important for the prompt to include summaries of e.g. the last 10 news stories involving participants in the game (players, coaches, etc.) and maybe the 10 top all-time news stories as well.
Then the prompt can be asked to summarize the game (not the other information), but to draw from the other information where it might make the article better.
Seems like exactly the kind of things LLM's can do, right?
I don't believe they'll have a human editor ensure quality and accuracy. The whole point of having AI write your stories is to minimize the number of people they have to pay, so paying a trained professional to thoroughly review every story is probably off the table. They may compromise on the thoroughness of the reviews, or the expertise of the editor, or they may just not have a human in the loop for every story, but what they will not do is pay a professional editor to do their job the right way, this I can guarantee.
I don't think that holds up. They can save a lot of money by using generated articles, but they lose customer (advertiser) confidence if the content isn't accurate. One editor per ten replaced writers is still a significant cost savings.
A few iterations from now we might see editors getting replaced too, but I don't think we're there yet.
The content isn't what it should have been, which is why the previous commenter rightly assumes that no editor looked at it.
Even a low percent of all published articles containing such problems doesn't in any way prove there's no editor involved.
Today, with the field of journalism in freefall, we actually have the worst of both worlds: not enough editors, not enough time to report, not enough original reporting being done, and too many AI or computationally generated articles. But, I don't see how getting rid of the humans and doubling down on the AI actually solves that problem in the medium and long terms.
In their announcement of the service, ESPN made a point to note that “each AI-generated recap will be reviewed by a human editor to ensure quality and accuracy.” It’s unclear if the human editor failed to notice Morgan’s absence or also decided it was not worth mentioning.
"Blame the intern" has been the great scapegoat for the last hundred years.
The AI highlights cut off during a point, skip entire points or even sets. Not to mention they don't account for important events that happen between points, they might cut off exciting commentary/celebrations that happened after a point, and even the handshake at the end of the match.
This data exists for most live games at this point via various web services. I'm sure espn has significant resources internally to source that info
But I wonder if there are licensing issues with using the audio/transcript to generate your summary. I know that the raw stats are public domain but I wouldn't be surprised if they can't use the transcripts or audio.
Now, I imagine they take that raw API call and just use a prompt like, "write a summary article for a game using this data" and it spits it out. And I assume the prompt is more thought out than that (or not? It is ESPN after all).
I don't ever remember "retiring_players" being part of an API response, though, ;P
edit: Oh and yes, the play by play recap is documented EXTREMELY well. You would be surprised. The more popular sports like Gridiron Football and Basketball would literally have player locations by the second. This data all comes from feeds like SportsRadar.
They probably wouldn't pipe the fine tuned stuff like that in to a prompt, but you still have a decent summary like how many 3-pointers someone had and where they shot them from.
Can we all just admit this AI phase is just a bubble?
> It’s unclear if the human editor failed to notice Morgan’s absence or also decided it was not worth mentioning.
However, they're almost always so general, or even elementary that anyone who would bother to read the article would already know that stuff would wonder why a sports writer would write it. There's zero need for them for the audience they will attract.
A few even have these hilarious "this report was is not intended to indicate X, but .." type paragraphs at the end.
You can almost imagine the prompts that brought them about.
Commentators share a lot of random facts and insights during games, and that would certainly enrich the AI summary.
but most likely the result of opposite-end backlash from the past where LLM's would name people in a way that could compromise their privacy
upon which they were made "safer" by AI companies making stop dropping names as much as possible
So... there was a human-written article to cover the human interest side of the story and an AI-written technical recap of the game. What's the issue?
And regarding the complaint about the link in the sidebar, it's easy to miss only if you came in directly to the AI-generated article. If you go to the main site, they prominently feature the human-generated content [1].
These are the emotional sprinkles, that AI often misses.(Which is to be expected from an emotionless AI)
It's hard to watch anything by ESPN or NBC when you're just interested in the game and all they want to do is feed you everyone's backstory, or they're only interested in big name players.
For instance, the year after the Mavs beat the Heat to win the NBA championship, they face off in the first game of the year. Nevermind that the Mavs are the defending charmpions, the Heat had LeBron James and Dwayne Wade. They showed 8 highlights from the game, all from the Heat. The Mavs won the game.
An impartial AI recap does not have a high bar to get over.
Only if you view the game as a cold emotionless process where the important thing is simply the data coming out, rather than the entirely human construct it is.
If you want the drama, that permeates any soccer game - you watch the game, read a 10 page review or watch a long review video.
It's a software bug, and there's plenty of resources available to the AI's owner to try and address it. It doesn't need a human advocating for it being OK.
Why is it a software bug? Just because you want to see those emotional sprinkles, doesn't mean that the person planning how to create the article decided that.
You personally and emotionally decided, without evidence, that this is a bug. No one else is saying that.
Imagine if you asked an AI to describe the discovery of radiation, and the AI deiced for you to talk about the personal life of Marie Curie for 2/3 of the whole text.
If you look into the happenings of the game, you'll find that there was a special ceremony held at 13 minutes for the retiring player. It's like summarizing the discovery of radiation and not mentioning Marie Curie at all. It's not emotional sprinkles (which is a pretty messed up way to refer to human interest articles), it's just omitting a notable deviation from the normal game flow is a bug.
We as a community are not used to calling AI failures bugs, but we probably should - it's the most accurate term we have. As is an non-requested backwards joint, or a fan of fingers, on a photorealistic generated image.
It doesn’t matter if the humans in the loop are at fault, technically, for the omission, the fact that AI replaced a human who could have been tasked with going to the actual events and writing about them was the reason the events were misreported!
We’re the technologists who are supposed to think hard about how to safely implement technology like this and instead of pointing out flaws and carefully testing things, we’re just cheering on tech we don’t understand how to use or get working properly!
AI has been promoted as the best thing since best thing references were created. Of course people that understand the tech will know it's not really that great, but the people being sold the tech just accept the brochure as gospel and think it is amazing.
Do you think that anyone making the decisions on how many human journalists/editors to employee know how AI works, or that they hear the promise of being able to use even fewer humans and run with it?
The machines simpley obey their instructions, which was presumably to fluff out some words about who scored points, who defended from those who wished to score points, etc...
Ignoring the halftime events seems like a plausibly sane thing for a sports stats fanatic to have happen, and that's exactly what happened here.
Was this an AI/ML geneerated thing? I would question your definition of AI, unless a serries of IF/ELSE statements satisfies your concept of AI... It's just following the rules it was given.
what a time to be alive!
It isn't computers and digits and statistics. It's people and personalities and how they interact. There's a reason sports is called a reflection of "the human drama."
Except for the Olympics, I don't watch or follow any sports. But I know enough to know that sports is about people.
The only people who think it's nothing more than numbers are people with gambling addictions.
But to answer the OP's original question: no, she didn't do anything worth mentioning in the game per the newly updated ESPN article. Her team was dominated, losing by 3.
"It was the final game in the nearly 14-year career of USWNT star Alex Morgan. The two-time World Cup winner and Olympic gold medalist played 15 minutes, exiting in the first half. Her shot on goal in the 10th minute was saved by Courage goalie Casey Murphy."
So should she have been mentioned in the article? Yes, but not for her performance and not in the context of her play in the match.
The commenter, intentionally or not, is suggesting that nothing is relevant about any particular game except the statistics of the match and the outcome.
So yes, they were suggesting the game is only about numbers.
"You could certainly make the case that everything in the recap is accurate from a factual perspective. However, the fact that it doesn’t include any information about Morgan and how important this night was for her, the NWSL, and U.S. women’s soccer speaks to how these kinds of services can’t replicate human writers who can see an event from a 360-degree perspective."
The reviews that come after are typically the ones that cover broader impact. Reviews also include third party commentary and more insights from specialists.
The benefit of having a factual recap generated minutes after the game far supersede the value added by a few words from a low paid recap writer hours after the game.
I guess thats subjective and depends on what the individual finds interesting/important in sport. However including something potentially unecessary is far easier to do and covers more bases than leaving something potentially necessary out.
But there are many different types of people who might disagree with you or prefer something different. You seem to be just stating that what you want is what everyone should think is best.
No one, including you, is interested in every single detail. And no one, including you, will read a novel for every single game.(and every single soccer game can easily become a novel)
And I've made a lot of money on disregarding maximalist user requests, thank you very much.
I mean, hell, even if it were just another game I’d probably find it relevant to include that Alex subbed off in the 14th, because that’s really eye-raising. This is a total miss by AI that would have never gotten past a real editor.
Specifically to this game: it also literally affected the game (maybe not in points, but it did have an impact), and that it wasn't mentioned means the information provided for the game was inaccurate.
If AI is going to summarize a game, it should do so accurately. It did not do that in this case, and that's not debatable. It was wrong. It did not accurately report the game.
From the article. Or is the claim that the AI model should expect there's nothing noteworthy about this series of events?