Why can't you get the data used to create the Consumer Price Index (CPI)? | Hacker News Reader

Why can't you get the data used to create the Consumer Price Index (CPI)?

70 pointsErikAugust51 comments

The Consumer Price Index (CPI) measures inflation and, in the words of the US Bureau of Labor Statistics, "directly affects hundreds of millions of Americans".

I would have guessed the data used to construct the CPI would be open to everyone.

But no, it appears to be unavailable and it is instead a black box due to rules of confidentiality. What is strange is, it appears much of the data is publicly available data. For example, 8% comes from e-commerce sites, the USPS public website, etc. If you browse through the categories, you can also see that almost all the data could be collected from public sources - APIs, websites, etc.

So, does anyone know why this data cannot be obtained?

Reference: https://www.bls.gov/opub/hom/cpi/pdf/cpi.pdf

View on HN

You might be interested in The Billion Prices Project, which calculates price changes by scraping online product prices. They make the data available.

http://www.thebillionpricesproject.com/

Spoiler: Billion Prices Project has mostly tracked CPI within 1-2%

1-2 pc every year or lifetime? Even a small difference in the yearly rate will add up to a huge difference over time.

Lifetime: http://www.thebillionpricesproject.com/wp-content/uploads/20...

Squinting at the graph, for 2008=100, in 2016 CPI was ~108 and BPP was ~109. (Couldn't find a newer graph with a quick look.)

That’s because the inflation is mostly hidden in hedonic adjustments and improper basket allocation. You need only to look at the basket fraction that is housing and healthcare to see that inflation is being covered up as it’s too small a piece of the pie. Look at the price of a truck if you want to see hedonic adjustment in action covering up price increases.

CPI does not cover either house prices (it covers rents, and imputed rends for home owners) or employer-paid/government-paid healthcare (it covers out-of-pocket expenses and add-on insurance only).

If you are interested in house prices, look for the Case/Shiller phouse price index, if you are interested in health care costs, look for the PPI components of drugs, hospital services, medical devices etc.

To counter your implied assertion - no, CPI is not being manipulated, but house price/CPI ratio has risen because mortgage rates are low, and medical care costs as a whole have risen mainly because of (a) proportionately more fat and old people, (b) agency problems in the health insurance model (overprescription/overtreatment out of defensiveness even if it is not cost effective).

So CPI is not being manipulated just cost of living has risen and it’s not reflected in CPI. That’s silly. The CPI index basically picks and chooses which values to adjust and which to fix in order to get a desired outcome. It’s not a robust process there is a ton room to nudge numbers one way or another depending on if you want to show inflation or not. Think of it more like an accountant working towards whatever profit number the company wants to report this year.

Generally the government can’t Show high inflation because doing so would have all sorts of second and third order consequences like increased SSI payout and higher government salaries as well as affect contracts and treasury bond yields.

The entire market would crash if you got a 3-5% hot inflation number. That basically can’t be allowed

+8% vs +9% is a 12% divergence is just 8 years, and the numbers where even further apart in 2014. So, it looks like they are very different models.

The dollar has lost more than 90% of its value in just 50 years.

The point is the BPP is proof there is no nefarious conspiracy to manipulate CPI to hide “true” inflation. Amongst the gold-bug hyperinflationist crowd there is a commonly held belief that inflation is much higher than CPI or that CPI is manipulated to hide inflation. The fact that a simple model scraping a few online retailers results in nearly the same annual percentage change demonstrates that this conspiracy theory is a myth.

Essentially inflation is a political measurement that drives policy decisions and that is cause for a lot of concern.

The current inflationary model is mostly driven by salaries. Thus in the 70s unions drove up inflation due to oil shocks.

The concern for me is that models do not include asset inflation for goods (housing) that most people would like to own and where there is little productivity improvement. Using an equivalent rent model ignores this preference.

CPI measures annual change in the price of goods consumed. A house is not consumed within anywhere near a year, and also has expenses. The parties in a housing sale may not be consuming the house at all, but treating it as an investment and renting it out. Stocks, investments, etc. aren’t consumables so they aren’t included either.

Rent is the price of housing consumption.

They do include price increase of Housing in owner equivalent rent but there are other issues with this adjustment. Also once again basket allocation is lower than it should be for rent.

> once again basket allocation is lower than it should be for rent

No, rent and owner's equivalent rent make up a quarter of the inflation basket - just in exact proportion to their share of aggregate consumption expenditure in the US.

The San Francisco tech bro who spends 70% of his take-home on rent is not representative of the US population at large...

But it fails to account for property taxes which in many areas have increased dramatically.

If only! Property taxes are close to free of deadweight losses.

(Only the proportion of property taxes that falls on the buildings etc causes deadweight losses. The proportion that falls on the value of the land is free of deadweight losses.

See Land Value Taxes.)

CPI is probably near CPI the problem is that CPI does not actually measure inflation felt by the average American. The basket of goods is not reflective of actual allocation.

Because of that the CPI value is significant lower than actual inflation

I think they’d say that inflation is in assets, not consumer goods, which to me rings a lot more true. It’s also true that for the average person on average wage housing has become greatly less affordable in the last 50 years but that’s because income hasn’t kept up.

I don’t think the price of home sales is a good measure of housing affordability.

Here’s a graph from Zillow which does not show a huge increase in last 30 years: http://www.doctorhousingbubble.com/wp-content/uploads/2016/0...

According to the American Housing Survey, housing as a percentage of income only increased from 19 to 22% from 1985 to 2005.

Overall, wages have exceeded or kept pace with inflation since 1979: https://www.politifact.com/factchecks/2019/jan/30/kamala-har...

Averages ignore the growing income inequality.

From 1979 - 2018 incomes for 90% of the population is up 23.9%. 90-95% up 45.7%, 95-99% up 71.3%. Top 1% is up 157.8%. https://www.epi.org/blog/top-1-0-of-earners-see-wages-up-157...

So chop off the top 10% and numbers are going to look very different. From housing to CPI it’s tricky to adjust for such things to understand how most of the population is doing.

In that time homeownership has gone down - those BUYING houses spend less of their income on it, but that’s a smaller and smaller piece.

The difference does not need to be always in the same direction.

It would be if the thesis is that the CPI is systematically manipulated to suppress inflation.

That would be pretty hard to do in our data rich environment. It's pretty easy to look up historical prices of things like oil, wheat, and other commodities, as well as labor, automobiles, and other consumer goods. Even a small systematic manipulation would become really obvious after a decade or two.

Not necessarily, if the manipulations are reverted smoothly over time. There are certain moments in which inflation is a focal point (now for example) and other moments where nobody really cares.

Also, I think the inclusion of things like tuition and medical care kind of adulterates the data. I have read that people go to the hospital for non-emergency reasons less often than before the pandemic, and also that several schools slashed tuition due to the pandemic. That offsets price increases in other areas.

The problem with this type of argument is that the fed doesn't want to exceed more than 3% inflation. It has no reason to lie and pretty much every central bank is struggling with lack of inflation.

You can criticize the Fed for doing reckless policies that fail to increase inflation and only drive a bubble. Why would you even need an unproven conspiracy theory like CPI when the problems with the Fed are undeniable? You can even complain to the fed that inflation is too low and they should either calm down or do something else since the Fed is clearly not capable of driving inflation. Considering the mission statement of the Fed that's a huge blow by itself.

Your complaint about the Fed was right in the last decade.

But something seems to have changed for the better when they changed to average inflation targeting recently.

I don't think you understand what I am saying.

1. I'm not arguing that the fed is artificially inflating inflation. I'm saying that if they want to suppress the appearance of inflation in the short term so that they can keep rates low and prevent any panic about the dollar until they can react to it, they have some ability to do so because of the opaqueness of the data.

2. Bias in a signal doesn't have to be persistent to be problematic. The average reading does not have to be biased in the long term for there to be a problem with the spot reading. That's a major reason why more granular data is always better.

3. I'm not stating any conspiracy theory. The Fed is driven by data. Biases in data affect data-driven decisions. CPI includes various inputs, not all of which are experiencing the same demand. CPI is an agglomerate reference.

Thank you - but I ask on the Spoiler Alert - it looks like they only have data through 2020 from two online retailers [1]?

http://www.thebillionpricesproject.com/datasets/

The CPI is a weighted average of several categories. and some of those categories severely underestimate inflation. For instance, the CPI says new car prices haven't increased in the last 20 years but, that's obviously not true, if you calculate the real cpi for say a corrolla or a camaro or whatever model you choose, you can see it's gone up about 40 to 60% in that time which means for that category, they're undercounting CPI by almost 2 to 3 %.

I think they're trying to prevent people from calculating the real CPI. there's alot at stake: the bond market, social security, federal interest payments, perception of real wages, real productivity per capita number, etc.

They adjust car prices based on feature improvements:

https://www.bls.gov/cpi/quality-adjustment/new-vehicles.pdf

Certainly debatable since it is impossible to actually buy a new car that doesn't have a bunch of those features now.

Debatable is an understatement. their discount model is completely disproportionate to the benefits that those features bring. If an auto now costs 50% more, and CPI really is 0% increase, then I expect to see a monetary benefit of 50%, either through reduced maintenance cost, increased MPG, or maybe it makes me breakfast of the equivalent cost. but, I don't see new cars have anywhere near this benefit.

And it becomes even more preposterous when you look 50+ years back. CPI, says x3 increase in cost of car, but it's actually about x10. So, how does a car return +233% of it's value to a user when MPG hasn't even doubled over that time.

i mean sure, monetary rewards are not the only advancements but they should the primary basis for the inflation discount model. those auto-dimming mirrors and countless other features just don't mean very much.

A typical new car practically lasts way longer than a car built 30-50 years ago, early 2-3x the value there. Account for significant reduction in chance of death, MPG improvements, and these numbers make complete sense.

Yeah but compare it to a truck in the late 90s not really a huge difference in economic value. Bit more luxury but a 1/2 ton truck is still a 1/2 ton truck it can do about the same amount of work but it can go 0-60 a bit faster

Cars my family bought in 1990 (i.e. 31 years ago) drove happily for 15-20 years. I don't think the "lasts way longer" claim stands up to scrutiny.

That is how it is supposed to work, yes, but those numbers are questionable - I'd be curious to see if that data actually can be shown to pan out, esp as certain vehicles have systemic issues which cause them to not last as long as the claimed lifetime.

Others severely overestimate inflation. Healthcare, for one - there are lifesaving treatments that are available at some price today that weren't available at any price in the year 2000. Per the textbook definitions, this is infinite deflation. Consumer electronics also arguably fits into this category.

Fundamentally, though, the problem is that CPI does not track changes in the perceived minimal acceptable quality of the various baskets of goods. If the house you "need" to fulfill a specific lifestyle is twice as big and twice as expensive, this has zero impact on CPI. Same with healthcare quality - if your insulin works 10% better and costs 10% more, the CPI-centric view says that people are simply using more real dollars to buy a higher quality of life. Education has a similar problem, where job requirements shifts from high-school diplomas to undergrad degrees simply do not show up in any way in the CPI; we're just choosing to be better educated to end up working the same jobs.

I think this all boils down to the CPI measuring things as if all goods were absolute goods, while nominal GDP per capita would be the appropriate measure if all goods were positional. I don't know what fraction of spending is on each, though, but you can reasonably put the "real" inflation number somewhere between the two.

> Healthcare, for one - there are lifesaving treatments...

The correct metric would be life expectancy, as it sidesteps the zero priors problem. Using that, the healthcare you are talking about looks a lot more like the diminishing returns you get on higher priced modern cars. Before anyone says anything about "quality of life", I'd recommend looking at the way assisted living operations are run. Anyway, we really started to lose steam after dentistry... dunno what the analog would be for cars, power steering?

> If the house you "need" to fulfill a specific lifestyle...

This is another thing I've been dealing with recently. I've delayed entry into the housing market way past what I'd have liked because the pricing makes no sense to me, and I've found that usually if something doesn't make sense to me - it is because it is misrepresented. After watching the market for a few years in two very different parts of the country I noticed something interesting: sub-million dollar homes were quickly getting more expensive, and multi-million dollar mansions were relatively static. I wonder how much of that has to do with home-loan accessibility, similar to the cost of educating students magically increasing in lock step with student loan program scope and availability.

> sub-million dollar homes were quickly getting more expensive, and multi-million dollar mansions were relatively static

It's because the price of the average house is due to the cost of urban/suburban land - which is scarce and therefore tracks affordability (i.e. rises steeply with median incomes, population growth and especially falling mortgage rates).

Meanwhile the price of multi-million dream homes are (a) often dominated by building costs which rise slower than incomes (b) don't rise due to falling mortgage rates as the NFL player/movie star/exiting startup founder pays in cash rather than over a 30yr mortgage term.

> and I've found that usually if something doesn't make sense to me - it is because it is misrepresented

That is a statement full of hubris. You might find that if something doesn't make sense, you might just not be in possession of all the relevant information. (In this case, the trajectory of mortgage rates and their effect on affordability of homes, demographics and population growth in a metro area, income growth and wealth distribution.)

> It's because the price of the average house is due to the cost of urban/suburban land...

That maybe true, but it would be unrelated to my observations.

> ...rises steeply with median incomes...

Whoops! For that to be true there would have to be a non-linear relation that would itself be far more interesting than the housing prices issue.

> ...often dominated by building costs which rise slower than incomes...

Not a lot of new mansion construction going on. Oh, and those construction costs are pretty ridiculous right now due to a jump in lumber and appliance costs - income would have had to nearly double to match construction's rate of increase. I collected bids a few months ago for a grading and paving, jobs dominated by labor costs. The quotes were laughably overpriced.

> ...as the NFL player/movie star/exiting startup founder pays in cash...

lol, your conception of today's millionaire is way off - it is a distinction that has never been more easily attained.

> That is a statement full of hubris.

No, that is - and presumptuous :) Also, that final sentence agrees with the mortgage issue I mentioned, gets the direction of migration wrong, and finally overlooks the steadily growing income disparity - which is somehow resulting in mansions doing poorly on the real estate market. So you did a pretty good job of unintentionally proving my point with regard to the situation being misrepresented.

The BLS makes nearly all of their products publicly available for free. You can poke around the old-school way here: https://download.bls.gov/pub/time.series/

They have an API with documentation here: https://www.bls.gov/developers/

And a few other options here: https://www.bls.gov/data/

But I suspect that what you mean is that the raw data used to calculate these metrics isn’t available. Such as the “worker X on 25 Feb found that the price of a pound of flour at the Kroger store #357 is $1.” I suspect there are two reasons. (I can’t say for sure because I don’t work at the BLS, but I am a statistician, work for the government, and have been involved in decisions not to release data).

1. The BLS almost certainly contracts the leg work for the CPI. While the calculations are relatively simple (arguments about methodology excluded), the work to collect prices nationwide is nontrivial. Including permission to release the data publicly likely increased the cost of the contract. These data vendors also sell their price data to other “businesses intelligence” companies, and allowing the public release hurts their other business prospects, so they would charge a premium for it. And ironic as it may seem, government agencies/workers are interested being good stewards of the public’s tax dollars and had to make a value decision to not buy the rights to release the data.

2. For anything that isn’t legally/contractually prohibited from release, there may have been a conscious decision that the data at its most granular level either wasn’t useful to the public at all, or was likely to be misleading/misused because of critical gaps in what can be released. If the vendor for Alabama prices allowed release but the vendor for California prices did not, they may have just made the decision that releasing incomplete data set wasn’t a good idea.

From the text:

> Multiple confidentiality issues arise in the production of the CPI. One set of issues arises out of the need to prevent unauthorized access of data that are embargoed, or yet to be released to the public. because the CPI data can affect financial markets, it is essential to ensure that no one without authorization has access to the data before release. BLS personnel who do see the data ahead of time are restricted by law from engaging in certain financial transactions during the period where they have seen data that are not public. Pre-release data are encrypted and always kept on secure servers, with any hard copies locked in secured areas.

You can get access to some of the data through the University of Chicago, but it is super restricted (as required by their source):

https://www.chicagobooth.edu/research/kilts/datasets/nielsen...

Good luck, this type of data is really really guarded as it is ridiculously valuable.

Wait, so nobody who participates in CPI surveys chooses to publish the specific questions? Is it illegal for the respondents to do that?

I know somebody who got a CPI survey last year -- maybe I can ask her.

Disclaimer: Sharing links to my own content

I hear you. I tried recreating the CPI for India and couldn't get the data needed. (see https://arun.tanksali.com/topic/rpi/). I eventually thought I can come with a Personal Price Index which reflects how inflation is when seen from my personal perspective. I took a basket of goods derived from our household consumption and scraped prices for a year (till last month). See (2) for data. Obviously, CPI represents the average citizen and no individual is likely to fit that definition well. I am no economics expert but I expected some correlation between the published CPI and my own observations but there were times they were moving in opposite directions. 1: https://arun.tanksali.com/topic/rpi/ 2: https://realpi.tanksali.com/#/app/dashboard

What exactly is it that you want, but can't obtain?

Beyond the plethora of data that is published, you could submit a FOIA request for anything you feel is in the possession of the BLS, but not being published.

Isn't this it?

https://catalog.data.gov/dataset/consumer-price-index-cpi

Because if you could lay your hands on the data and calculate the CPI for yourself, you would find that the 'official' CPI is pure baloney.

The CPI is a propaganda tool. It is designed to keep the real inflation hidden to make it seem that inflation is not a real thing to worry about. Because it is formulated in secret, they can manipulate it to give a fairly rosy picture. The published CPI never panics.

the process is literally posted on the Bureau for Labor Statistics site [1], it's like the second result on Google

https://www.bls.gov/cpi/questions-and-answers.htm

The CPI absolutely crashed in April 2020

https://tradingeconomics.com/united-states/inflation-cpi

Do you have any actual evidence of this?

I wish I had a dollar for every irrational (extremely emotionally charged!) response to this same question when I have posed it.

Reply on news.ycombinator.com