With a valid measure I would expect a roughly even distribution over time between underestimates and overestimates. For a valid measure worth considering I'd also expect the stat to be released later when revisions are less likely because more actual data has been collected
This is a valid hypothesis. It’s wrong, and I’ll explain why. (It’s a bad and invalid thing to conclude.)
If measurement errors were iid, you’d be correct. But they’re not. They’re well documented for not being so. Earlier survey results are biased by directional response bias inasmuch as the employers with the lease changes respond first. So the earliest releases tend to match whatever was going on before. Then the employers who had to do paperwork respond. And then, finally, someone gets around to calling the folks who never got back. Some of them aren’t around anymore.
So yeah, the directional tendency in revisions is well documented. And for a long time, the early releases were appreciated. But maybe American statistical and media literacy is such that only final releases should be released, which would mean we’d always be working with data 6 months to a year out of date.
[1] https://www.apmresearchlab.org/blog/how-abnormal-are-the-rev...
Naturally all of this metadata about the BLS surveys is available for free from the BLS, so you can just go look at it.
Yes. The reasons for this are well documented. Changing methodology for the preview estimates is rigorous. That means our published estimates lag best estimates, something the primary sources note in every release if one gets past the headlines.
Also, if you have one year of massive job gains and four years of flat and falling, you’ll spend most of your epoch biased one way. Again, not a sign of methodological problems. Just a predictable methodological artifact that folks are supposed to be able to incorporate before using, much less emotionally reacting to, the data.
Related, I wouldn't expect past data to bias a current estimate. If 6 or 12 months of positive growth biases the next prediction it falls into the hot hands fallacy. It isn't predicting based on current predictions, its predicting based on recent past behavior and extrapolating forward. This only makes sense to do if the data is not yet available, and even then the extrapolation isn't a useful estimate of current conditions.
It’s a sample of a sample. The full sample is the final release. The early results are the preliminary releases. When firms change things they take longer to respond. So whichever way the economy is moving, there will be bias in that direction. If the economy is turning, you won’t know direction. If it’s accelerating or slowing down you don’t know magnitude. Sometimes context clues can help. Sometimes they can’t. There is no known statistical treatment for intuiting the missing data before one has it11
You’re still concluding from ignorance. They are not. A better question would be ask to whom they’re useful and how.
Like, if a fire is burning in a neighborhood, every sighting is valuable. You don’t always need to wait for a comprehensive picture before being able to do anything.
> I judge them as early and either inaccurate and useless or politically motivated to push markets while there's no meaningful data to contradict them
That’s wrong. But it seems to be a common error.
Maybe the solution is to make these numbers available only to gatekeep these numbers. Policymakers, academics, enterprises and banks can get a rarefied sheet for a fee. But the public doesn’t get PDFs, much less public reporting.
> while there's no meaningful data to contradict them
There are bajillions of them. ADP. State reports. Private surveys. Fed studies. That said, I’m leaning towards your view—maybe these data aren’t best made broadly public.
That assumes there are a meaningful number of reliable reports. If I regularly am told there is a fire only to have authorities come back a week later to adjust reports down I wouldn't trust them. If they over estimated the number of fires based on the last 6-12 months of fire data, with little recent data to go on otherwise, I would ignore the reports.
> Maybe the solution is to make these numbers available only to gatekeep these numbers.
This seems more reasonable at least, though I don't see much use in the data still when its released so early that its based primarily on recent historic trends and few survey responses.
> There are bajillions of them.
Those are all anecdotal in this case. If said sources were applicable and reliable the official data would consider those and have more accurate reporting. My point was that said reports depend on survey results, and when they report early results so early that few responses are in yet them there is no official data to contradict the early reporting.
At some point a lack of decision to take compensating action becomes faking the numbers.
There is no more conservative. The data will bias in the direction of trend. The point of the data are, in part, to measure that trend. Fucking with it to make it politically correct to the statistically illiterate is precisely the sort of degradation of data we’re worried about.
(They’re also useless as a time series if the methodology changes quarter to quarter. That’s the job of analysis. Not the data.)
Reporting biased data as the default because the bias compensation is already built into the audience seems like a weak argument for not improving.
They can provide for the continuation of data visibility/granularity by releasing the prior numbers as previously calculated and at the same time changing the calculation of the headline number to be better compensated.
The simpler argument is that changing it at all will result in a negative step change in the reporting that no one wants to take accountability for.
Ex post facto. Before the fact, we don’t know.
Imagine you know the weather will be a strong gust regardless of direction. Averaging the models will produce a central estimate. But you know it will be biased away from the center. You just don’t know, until it happens, in which direction.
> They can provide for the continuation of data visibility/granularity by releasing the prior numbers as previously calculated and at the same time changing the calculation of the headline number to be better compensated
They do. These data are all recalculated with each methodological change. They’re just deprecated indices the media don’t report on because they’re of academic, not broad, concern.
> simpler argument is that changing it at all will result in a negative step change in the reporting
Simpler but wrong. Those data would be useless for the same reason we don’t let CEOs smooth revenues.
It also seems like you said they shouldn’t revise the numbers but now you are saying they already do.
What am I misunderstanding?
There have been revisions since the forever, and this is because they depend in part of surveys, and if companies (and the people with-in them) don't bother responding in a timely or accurate manner then that's going to throw the sampling off.
> CES estimates are considered preliminary when first published each month because not all respondents report their payroll data by the initial release of employment, hours, and earnings. BLS continues to collect payroll data and revises estimates twice before the annual benchmark update (see benchmark revisions section below).
* https://www.bls.gov/opub/hom/ces/presentation.htm#revisions
Post-COVID surveying seems to have become more difficult (and BLS budget stagnation/cuts haven't helped). This has been a known issue for a while; see Odd Lots episode "Some of America's Most Important Economic Data Is Decaying":
> Gathering official economic data is a huge process in the best of times. But a bunch of different things have now combined to make that process even harder. People aren't responding to surveys like they used to. Survey responses have also become a lot more divided along political lines. And at the same time, the Trump administration wants to cut back on government spending, and the worry is that fewer official resources will make tracking the US economy even harder for statistical departments that were already stretched. Bill Beach was commissioner of labor statistics and head of the US Bureau of Labor Statistics during Trump's first presidency and also during President Biden's. On this episode, we talk to him about the importance of official data and why the rails for economic data are deteriorating so quickly.
If response rates are down or something else is making surveys more difficult, its reasonable that confidence windows would weaken and size of revisions would increase. Its unreasonable that difficulty in surveying would lead to a consistent bias in results though, that's a methodological issue at best.
It's been to too many moons since I took a prob/stats course to comment accurately on population sampling, but how valid is the assumption that errors 'should' skew both positive and negative?
That's always possible, though again I question the validity of the measure and results if its getting consistently skewed results. Either the methodology is faulty or the results simply can't be trusted because they can't reliably get good data.
Honestly, your comment made me mildly angry. That said, can you say why you believe parent's comment is misleading?
Admittedly, if I could find a single instance of someone willing to vouch or share insight on having filled out a BLS survey, that'd cure a healthy chunk of skepticism. There's still be the other distortions in the data to account for, but I'd at least have an instance proving that yeah, there is somebody filling out these surveys and it isn't just something they say they do to make their magic unemployment number sound legit.
Note, I'm in a massive sceptical shit phase at the moment. Last decade has burned my optimism hard. So when it comes to my ability to assume benevolent intent right now, there's a heavy bias against doing it, and a heavier bias in the direction of "what would be the easiest way to keep the System limping along?" The answer to that is "say you do one thing, in reality do another, and as long as no one comes lookin', it's gold." The finance industry runs on Trust moreso than anything else, and there ain't much to be said for Trusting anything you can't verify these days. Not from other humans.
I’ve never met a single chicken farmer. Does that mean I should be sceptical about them existing? Like, what sort of metric is this for truth finding?
> to assume benevolent intent
No need. Markets move on these data. The rich and powerful bet their money on what they say.
I’ve never != nobody has.
It comes from the Census bureau, a letter like this: https://old.reddit.com/r/frederickmd/comments/1p1j1my/did_an...
https://www.google.com/search?udm=2&q=%22current+population+...
They only reach ~100k unique households per year, so you'd need to survey a few hundred people to find a respondent: https://en.wikipedia.org/wiki/Current_Population_Survey#Meth...
> Note, I'm in a massive sceptical shit phase at the moment
How might one distinguish such "scepticism" from ignorance?
>How might one distinguish such "scepticism" from ignorance?
Ignorance doesn't seek to invalidate itself. Scepticism does. It does not enrich my life knowing there's a methodology to collect a "high value statistic", but not finding any on the ground proof of people who actually have primary exposure to the methodology. One can't reason around what the system is actually measuring without a sampling of that. I can find screenshots of UI at times. I find papers around low response rates. I never find an actual person who says "Yeah, I get dinged to do those every few months, once every few years..." I sure as hell know when I'm gathering statistical data, that sampling bias evaluation requires foot work, and if you do that footwork, if your methodology works, it shouldn't take you that long to run into someone you've surveyed if you're doing it right. If you're not, and only hitting "the usual suspects", you're not getting a representative sample/measuring what you think you are. So I look for payroll people or people who have done payroll in the U.S. and ask if they've actually ever been directed to provide input. I've been doing it the last few years. Nobody seems to recall ever having been asked to participate in what amounts to billion dollar money movement at stake jury duty.
So yeah. This is kind of a weird tic of mine at the moment. Ranks right up there with the time I felt the inexplicable urge to figure out what the deal with zoning as applied to city planning was and how it worked. Something just doesn't add up. I hate that. Mental equivalent of a thumb detection via hammer.
Unless you have introduced yourself with this question to thousands of people, this is a totally meaningless statement. It says more about your social circle, your grasp of descriptive statistics, and the weird online stew you are soaking your brain in than it says about the CPS.
So my main argument ( and frankly the only argument that should matter ) is that is a bad fit for the goal of estimating values ( even though we do know its failure modes ). Is that not enough?
<< That's what people are asking you to do.
No. What I am being asked to do is: "Show me a better way, but I only accept a better way that is already utilized by someone else". Not a recipe for a thoughtful exchange of ideas.
<< do other countries
No, it doesn't mean I am wrong.