Hacker News Reader

Comment by gnat | Hacker News Reader

Interesting to see The Economist doing modeling. Curious that they see no problem in it being utterly unauditable. Unlike a research paper that would cite specific datasets and model parameters, this is just "here are some numbers we're taking as true because model". They don't even appear to have done any validation on their model (holding back a few years from the training set to see how well it predicted them).

> We trained our model to predict the daily average gas demand per person for 26 countries (Britain and all EU member states except Malta and Cyprus, which use little or no gas), for each winter month between January 2013 and February 2022. After the model had learned the relationship between temperatures and gas demand, we gave it real temperatures from this winter and asked it to predict how gas usage might have played out had Russia not sparked an energy crisis.

View on HN

They'll push it on Github shortly. They have a fairly active and open Github - https://github.com/orgs/TheEconomist/repositories

Welcome to the majority of "journalism". All it takes is a look one or two levels down into the "facts" presented to realize they're wildly and poorly extrapolated to fit a narrative by whoever is paying the bills.

>Welcome to the majority of "journalism".

welcome to the majority of "modeling".

tuning a model to a statistical sample of the past doesn't give as much assurance about it's predictive power as people think.

then, only in the future do we find that the model failed to predict, by which time they tell us, "yeah, but we have a new model", to which the only appropriate answer is "yeah, but you had the same certainty about your old model that did not work"

I'm not saying that there's no point in modelling: you learn a lot about the dynamics of the system when you are working on the model; but that's not what the resultant model conveys.

> welcome to the majority of "modeling".

This is less insightful then you might think, as it's not limited to modeling. We just have a tendency to use previous observations as truth for future endeavors. Mainly because it's usually fine and works fine.

You can see it in every second discussion about any topic that's currently being researched:

I.e. there is nothing guaranteeing we won't have a super viral virus with 90%+ chance of death. It could happen...

There is nothing guaranteeing that LLMs ability to output relevant data will get better, even if it's been tremendously improved within the last year alone.

You can basically see this in action whenever someone is making a prediction. The likelihood of it coming true might be good enough to work with it, but you can always have something go amiss, or a meteor out of currently unknown materials hits the sun causing a chain reaction which causes it to go supernova...

>Mainly because it's usually fine

It usually isn't. John Kay and Mervyn King wrote a good book on this issue. They talk about events in three categories. Deterministic predictions, say where is the earth in five years in the solar system, probabilistic predictions based on past data, i.e. how likely is it that the volcano will explode in the next five years, and the most important one which they call 'radically uncertain' events.

Complex human events are almost always in the last category. Using the language from the first or second class of events to make predictions about completely dynamic systems that are entirely dependent on human intervention that have no clear relationship to the past makes no sense. Using quantitative language in that case is actively misleading because it creates the impression you have any notion of the total space of possible events at all. "X has a 80% of going to war in Y years" is an example. What they really mean to say is "I believe it is likely that..", but that 80% number is completely made up.

Or as Frank Knight put it: “A measurable uncertainty, or ‘risk’ proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all.”

But modeling is always a thing of probabilities, with a low confidence rating for getting it entirely right.

What you seems to have an issue with is how it's often reported on or used in politics... But the people actually creating the models are in my experience always aware that the results are never proving anything by themselves... And blaming the act of modeling for the actions of politicians and reporters is misguided, as they're just using whatever is convenient. If it wasn't a specific model, they'd find something else to validate whatever asinine bullshit they're peddling.

Sounds like a reasonable point at which to defer to the internet's pithiest meditation on communicative subjectivity. https://youtu.be/QsogswrH6ck?t=3

All models are wrong, some are useful.

https://github.com/orgs/TheEconomist/repositories

I can’t tell if this is facetious or not.

Yeah, they've had a few swings & misses already with their modeling.

They released some model around estimated covid deaths that was just completely wrong around China for the time.

They also failed to grasp that outside temperature might not be the only factor affecting power consumed for heating, assuming we want to keep constant inside temperature. The most obvious is sunlight - when it's 0C outside, my home requires much less energy when the sun is shining through the windows and on the roof.

Are you saying that EU saw more than usual amount of sunlight this past winter? Where can one find such data?

https://www.knmi.nl/nederland-nu/klimatologie/geografische-o...

Which, if I read that right, indicates that for Netherlands, winter months in 2022-2023 had more sunlight than in 2021-2022?

EDIT: Another data point would be total solar energy production in winter divided by installed power, per month, in comparison to previous winter.

Interesting indeed. Isn’t a machine learning model kind of an overkill for this kind of data analysis? Wouldn’t linear regressions suffice in this instance? Don’t you risk introducing unknown biases or even overfitting by using such a powerful technique when a simpler one is available?

This kind of reminds me of when some journalist on twitter used machine learning to find hate-speech against British politician. Of course the journalist found exactly what they were looking for. There was more hate speech against conservatives... Of course, the journalist was only an amateur in statistics, and used a flawed model, that could have yielded anything they wanted

disclaimer: I didn’t read the article because paywall, and waiting for an archive to be posted.

You can’t write an article about just scrolling through twitter, but if you pile up enough linear algebra it becomes an Officially Interesting Thing to report on.

Do you have a link to the research that found conservatives were targeted with hate speech more frequently?

It took me a while, but I found it

News source: https://www.bbc.com/news/uk-63330885

Twitter thread: https://twitter.com/petesherlock79/status/159025317484515328...

And a response thread explaining the flaws in the methodology:

As news source: https://novaramedia.com/2022/11/09/the-bbcs-investigation-in...

And on twitter: https://twitter.com/AyoCaesar/status/1590403667974582272

Basically the model was developed at google to find “toxic speech”, but the researchers had a very biased definition on what constituted as “toxic”. Racist speech didn’t count, while angry shouts did.

I would expect there is more racist speech against white people than anyone else. Regardless, race doesn’t put someone in a particular political group so I don’t see how it debunks anything. Did the critics measure racist speech?

Short answer: Yes, they tested the model with racist speech, and it failed.

> I entered a selection of racist tweets I’d received in the past year into Perspective’s API, of varying lengths and sentence complexity. Just to make it easy for the machine, I deliberately chose one which included a common racial slur against South Asians.

> None of these were registered as potentially toxic at all by the AI – but, “You’re a fucking G”, a compliment, popped up with a 90.29% likelihood of being toxic.

Your hypotheses that white MPs experience more racist speech than non-white MPs, is testable. I have a pretty strong feeling that you are wrong, but I haven’t seen any data to support either (only anecdotes; and this deeply flawed analysis). What you can do to test your hypotheses is find all British MPs on twitter, scrape all tweets directed at them, and measure the proportion of racist speech against them. However, I advice you to do a more traditional cluster analysis, rather than a training model, the latter is very likely to yield a flawed model just like the journalist’s “8 month labor of love”.

At least they were explicit about it.

Looks like they are effusively talking about a linear regression "model". We did those in high school. You can basically look at the graph or points and fit line to audit it.

Rentech also does linear regressions. They have phds picking and preparing dimensions though.

Reply on news.ycombinator.com