Hacker News Reader

Comment by flagrant_taco | Hacker News Reader

The Consumer Price Index is a scam at this point, as far as I can tell.

They regularly modify the list of items in the cart, meaning the data isn't reliable over time.

They also include caveats where they can modify the Scritsmier price differences. For example, they attention to adjust the new price to account for what they consider to be added value. Say a product is 15% more expensive one year after a new model is released. When calculated the CPI, they can decide that the new model added 10% more value or benefit to the user, meaning the price only rose by 5%.

I wish I could find the study now, but around 2013 someone compared the CPI listed price increase of magazines to the actual change in the price listed on the cover over something like 20 years. The true value change was around 130% but the CPI had adjusted for value 30%.

Most people seem to be feeling this now when they notice that process have risen much higher than the claimed 4 or 5%.

View on HN

> The Consumer Price Index is a scam at this point, as far as I can tell.

The BLS goes into excruciating detail about their methodology, which is not something often find scams doing.

The Consumer Price Index is just that, an index. It seeks to distill the complexities of the consumer economy into a single number, and to do so reliably over time. It's impossible to do this without a whole bunch of compromises and trade offs. It's trivial to rip apart any such methodology because of these trade offs - just find a trade off you don't agree with and argue it should be the opposite.

If someone attempted to make a t-shirt using the weighted dimensions of every American, the result would be a t-shirt size that would not fit any individual. But, over time, that t-shirt could be used to understand changes to the American physiology over time.

> For example, they attention to adjust the new price to account for what they consider to be added value.

They cover this in their FAQ [1]. But again, this is a trade off. If laundry detergent becomes more concentrated, thus can do 20% more loads for 10% more price, then how should that be handled? If $1000 Nike shoes become popular, does that mean that the price of shoes increased 10x even if cheaper alternatives are still the same price?

[1] https://www.bls.gov/cpi/quality-adjustment/questions-and-ans...

The most successful scams actually do go into excruciating detail to appear methodical, that's how you get people to buy in.

While I do agree with many of the points you raised, for me those point to the uselessness of the CPI as a matric. If the dataset is so complex you can only boil it down so far before too much context is lost. The CPI is an attempt to create a single number of off the entire economy, a task that is impossible made only more clear by complaints over the obvious and egregious tradeoffs made.

Statistics can be useful, it can also be a pointless game of manipulating so much data that you can make it say whatever you want. I think you'd be hard pressed to find an average American that would feel like prices have only gone up by ~4-5% over the last few years.

> those point to the uselessness of the CPI as a matric

Unless you're setting federal policy, it probably is useless to you. For personal forecasting, you're better off using the CPI calculated for your state or metropolitan area. (If you're procuring at a national scale, you're looking at a producer index.)

CPI plays a huge role in our everyday lives though. From cost of living raises to entitlement programs, it all moves based on inflation numbers and the CPI. The Fed even considers these numbers when watching the economy and deciding on fiscal policy, setting the Fed Funds Rate and impacting interest rates across the board.

> The Consumer Price Index is just that, an index

Except it's not. It's how we calculate inflation, which in turns drives how we measure GDP growth. It's also how we measure productivity and purchasing power variation over time. This measure has a very big impact, including on our assessment of policies and the evolution of the economies of nations.

Off course there can be no absolute value for such a thing, but the problem is that economists have been obsessed with “real” prices and disregarding “nominal” values (you'd notice the irony of naming the arbitrarily constructed thing as “real”), and so almost every economic analysis includes CPI and now everything depends on it so its measurement become included everywhere indirectly. And it is in fact a big problem.

'excruciating detail [...], which is not something often find scams doing'

Except for e.g. the small print in an EULA, or the XXXX pages Congress spending bills. There is definitely a bureaucratic technique of 'obfuscate through sheer volume of irrelevant information'.

Which details of the CPI methodology do you think are irrelevant and should be omitted?

Scoping to food CPI to simplify the conversation, provided a link to the actual raw data, e.g. prices & relative weights over time (do you have such a link?), how do I sift through it to answer elementary questions about the cost trajectory of the daily caloric intake, with geographic and socioeconomic breakdown.

On top of my head:

* basic breakdown: grains, meat, veggies

* processed vs. unprocessed

* grocery sourced vs. eating out, largely fast food

* sourcing, binned by log distance travelled

* scenarios: Med diet? carnivore? vegetarian?

On reflection, the core objection is that there is a subjective hidden parameter: the relative weights of different components. See e.g. https://www.bls.gov/opub/hom/cpi/calculation.htm#item-replac... I'd rather use my own weights.

> provided a link to the actual raw data, e.g. prices & relative weights over time (do you have such a link?)

Yes [1][2].

[1] https://www.bls.gov/cpi/tables/

[2] https://www.bls.gov/cpi/tables/relative-importance/home.htm

Splendid, thank you!

Who hasn’t done this. And now gpt can do it for us, for greater obfuscation at lower cost.

Your examples cover one of the problems: When the cheaper same old shoe is rarely on the shelf, then what? And are they as enthusiastic at removing value as they are at adding value (i.e. when then "same old" shoe is not same old but faster wearing)

They actually do describe this "If the BLS data collector is forced to replace a short-sleeve cotton/poly shirt in the CPI sample with a long sleeve 100% cotton shirt..." but they provide no answer aside from [well the one on the shelf now is better].

So that the buyer of the "common" shoe, shirt or TV pays more.

It's easy to argue that it's possible to dig deep and find that "older" cheaper item but that doesn't change that the "basic basket" is now more expensive.

The second problem is one of perception - and perception really matters in this case. And CPI doesn't solve that problem.

The BLS methodology is certainly “elaborate.” But as is pointed out down-thread, the CPI uses completely bizarre measures and makes little attempt to source data from the kind of places that would typically be used to do these analysis were more rigor being demanded or actual money was riding on them being correct.

https://news.ycombinator.com/item?id=37895252

A lof of actual money rides on CPI measures. For example social security payments are indexed. Which is one of plenty of strong incentives to keep the "measurements" low.

Is there any actual evidence of BLS personnel being pressured to keep their measurements low?

BLS is part of a large organization. Specifically it's part of the United States Department of Labor (DOL), one of departments of the executive branch. I haven't yet heard of a government with an incentive to find a high inflation number.

(And if it were an "independent" agency like statistics are in some countries, that would provide more shielding but not necessarily perfect either.)

Asking for specific evidence of a specific BLS personel filing a grievance or similar because of pressure is placing the bar way into the "blind" territory. This is not how influence works. I mean sometimes it is and we get to see it laundered in public, but that's not the common way. In the case of BLS it may be there, I haven't searched.

Oh hey, there is someone who got caught with his hand in the cookie jar: https://www.washingtonpost.com/archive/politics/1988/09/11/b...

As usual what would help is ever more work from the producers to justify their work. BLS comes short. They are not the only ones but still.

So you are insinuating something but have no actual evidence. Just another baseless conspiracy theory.

> They regularly modify the list of items in the cart […]

Yes? Of course? They are based on spending surveys:

* https://en.wikipedia.org/wiki/Consumer_Expenditure_Survey

When people change their habits in life the CPI is changed to reflect what that life costs. Up here in Canada, the CPI is also adjusted from spending surveys:

* https://www.statcan.gc.ca/en/survey/household/3508

You can see the list of changes going back to 1913:

* https://www.statcan.gc.ca/en/statistical-programs/document/2...

Do you think coal and lard should be included, like they were in Canada pre-1956? Or do you think the CPI should try to model reality?

> […] meaning the data isn't reliable over time.

It is meaningful in that life now and life ten or twenty or more years ago was different in what people purchased. That should be reflected in measures that model consumer spending.

Spending habits are mitigated by inflation though. For example, I buy significantly less beef than I did five years ago because it is now too expensive. Putting less weight on beef prices because of the reduction in consumption would indicate that inflation is less than it actually is.

> For example, I buy significantly less beef than I did five years ago because it is now too expensive. Putting less weight on beef prices because of the reduction in consumption would indicate that inflation is less than it actually is.

If people spend on item X less or no more, why would it be included in the index at the old weight? For any given period of time being measured, the basket of goods and their weights should reflect what people are actually doing. Why would you run a model that samples the wrong things?

This is one reason why (e.g.) StatCan went from updating their basket every 2-3 years, to annually in the last little while: spending habits shifted drastically in over the course of the pandemic.

* https://www.statcan.gc.ca/en/statistical-programs/document/2...

Substitution is a known source of (possible) error and there are attempts to take it into account:

> 9.22 The other potential source of error with respect to the basket weights is referred to as upper-level substitution bias. This bias arises because of the use of the Lowe formula, which is an asymmetrically weighted fixed-basket price index. Because the weights are obtained from a year that precedes the price reference period, the expenditures are not likely to be fully representative of consumer spending patterns in the price observation periods. This is because consumers tend to adjust their spending habits in response to changes in relative prices, buying more of the products whose prices have fallen or risen less rapidly, while reducing their purchases of products whose prices have increased the most. In other words, they substitute towards relatively cheaper products from relatively more expensive ones. The asymmetrically weighted fixed-basket formula of the CPI does not account for these types of changes in consumer spending until a basket update is performed.

> 9.23 Unlike the Lowe formula, there are five known symmetrically weighted price index formulaeNote which are theoretically free from upper level substitution bias. These index formulae use expenditures from both the price reference period 0 and the price observation period t and therefore account for product substitutions that consumers may make. In this regard they are representative of consumer spending for the periods in which price change is being calculated.

* https://www150.statcan.gc.ca/n1/pub/62-553-x/2023001/chap-9-...

* https://www150.statcan.gc.ca/n1/en/catalogue/62-553-X

> If people spend on item X less or no more, why would it be included in the index at the old weight?

Say ten years ago, hypothetically, everyone ate steak and wine every day for $100/week and today everyone can only afford to survive on ramen and tap water and it costs $100/week.

If you change the basket to say people today only want ramen, you will conclude that there has been 0% inflation in a decade. Everyone still spends $100/week on food, no change at all!

That does not seem like a useful measurement.

If they didn't do those things, then wouldn't we just get a flagrant_taco from an alternate dimension who would instead argue about how CPI doesn't account for changes in consumer preference over time or how a cell phone in 2023 is different from one in 2003?

If they listed both that would make sense, that's commonplace when listing net vs gross revenue, nominal vs real GDP, etc

But then they would just have to make a list of each year's CPI basket and it's changes, and then have to calculate that going forward for forever.

Like, the differences in the 1991 basket to the 1992 basket are pretty small (please, someone, double check this). But the differences in the 1991 basket to the 2019 basket are larger, for example.

So it would seem to me to be a bit silly to have to update the prices of the 1991 basket all the way to today, especially when you're updating the 1992 basket as well, and every other year.

I'm not a CPI expert person, so if there is someone that really does understand these things in depth, please chime in!

Furthermore, some of the items in the 1991 basket are no longer even available at any price. What's the price today for a new 25 inch analog CRT TV?

But that IS part of the problem. Obviously without looking back 12 years for a day to day practical result (but still looking that far for long term economics concerns.)

A "normal TV" now is a smart (spit) TV which compromises privacy and viewing experience and durability - but which is sold massively cheaper than a plain TV as defined perhaps 6 years ago. You have to dig hard to find a plain TV and it's more expensive. Do you count "smart" as a positive hedonic feature? Or do you count the effort necessary to find a plain TV in the price increase of a plain TV? Do you compensate for much other price increases just because a "normal TV" is now much less expensive and (allegedly) more hedonic?

Not at all, they could simply report the real cost difference in the basket as well as the adjusted cost. Every year we don't see a matrix of GDP compared to both the real and nominal GDP of all previous years. We see those two numbers compared to last year and it's left to basic mathematics to do longer term comparisons.

Part of the challenge is with how frequently they decide to change the basket of goods. If the basket was largely stable over time we would have a pretty good estimate simply by aggregating annual data, especially if they focus on staple consumables that don't change drastically over time.

And they could exactly do that. That's what computers are for.

Not even close. The BLS hires workers to collect price data from retailers on-site for their CPI computations.

The question was about offering multiple versions of the resulting statistics summary. Based on various starting dates, theories and such - something like output that could be explored or dialed including dialing which CPI method to use. That would explode the amount of reporting. It's not about data collection.

Making a list of each year's CPI basket would require price data collection for each year's basket.

> If they listed both that would make sense

They provide their raw data [1]. You can reconstruct the index however you want. There is an entire industry of consultants who create custom benchmarks for planners.

For personal purposes, your metropolitan area's CPI is a better metric to watch than the national number.

[1] https://www.bls.gov/cpi/tables/

There should be two parts to the list. A constant long term index and a shorter term more variable one.

Would the long term index have a horse-drawn carriage, a butter churn, a typewriter, a 50# block of ice for the icebox, and a rotary dial phone in it?

At some point, even the long-term index can't be constant.

I think it is a bit of a scam but I do think it had good intentions. CPI is used to grade and since it is a scoring product it is manipulated. One thing you will notice with inflation is that goods that can be outsourced have been what is keeping CPI down. TVs, appliances, etc. Housing, Education, Healthcare are all things the US cannot outsource and there you will see a larger impact on CPI. Food is the on in this theory that I am not sure about. The US grows much of their own food so I wonder if it still fits even though you can outsource.

Food. The technique is to shift from 'locally sourced organic veggies' to 'dorritos made with mass produced GMO corn', because 'consumer preferences have shifted' and both have the same calorie payload. Note how 'preferences have shifted' also discreetly helps to insinuate a moral fault in the people themselves. Anything but 'healthy food has become too damn expensive for the majority of the population'.

That is not even remotely how the food components of the basket work. You're just making things up and obviously haven't read through the detailed list of items or the methodology for how those are updated.

Fair enough, BLS seems reasonable, though devil in (lots of) details. I was extrapolating the 'ketchup is a vegetable' debacle, https://www.foodpolitics.com/2011/11/ketchup-is-a-vegetable-..., where American ketchup is a mix of tomato paste and high fructose corn syrup, see e.g. https://www.nutritionvalue.org/Tomato_ketchup_by_HEINZ_54041...

I'm curious about how the housing crisis is measured here. As homelessness booms, people are "choosing" to not pay rent. Does that zero out their housing expenditures in a survey?

Although I'm sure that outsourcing made some difference, are we sure most of the benefits didn't just rise to the investor class?

TVs dropped in price due to the benefits of adopting moores law. Look into a modern TV and you see a symphony of integration and cost reduction thanks to moores law. I'd imagine the same applies to anything else that jumped on the moores law bandwagon. We need to measure appliances that benefited from this vs appliances that didn't.

The size of transistors has very little to do with TV prices. They are cheaper now because they are built with outsourced exploited labor and because they are full to the brim with ads and stalkerware.

It absolutely does make a difference. As I mentioned in my original comment, it enables consolidation of components and functions. Take apart an old CRT and a modern LCD screen and in the new TV you will see a single IC doing the job of multiple separate boards in the old TV. This is where a lot of the cost has come out of. All thanks to the capabilities of modern ICs...powered by Moores law.

The CPI is reliable over time. When the basket of goods is changed to reflect what consumers actually purchase the index is normalized to maintain continuity.

But on longer time scales it becomes a bit meaningless. Lifestyles were so different 50 years ago that very little is directly comparable beyond certain staple foods. People tend to interpret the CPI numbers based more on their preconceived notions of how things are going than on any objective reality.

> They regularly modify the list of items in the cart, meaning the data isn't reliable over time.

Please show me an economy where the list of goods stay stagnant and don't constantly change.

Unless your CPI is comprised of raw commodities like Iron Ore and Salt and Copper - then you're not going to be able to keep a stagnant list of items.

If x% of consumer food spending was on milk in 1920 and now it's 1/10th x - it doesn't make sense to keep milk with the same weight as it did in 1920.

It's pretty hard to tell if people are drinking less milk now because they're poorer or because they just prefer almond milk (which is more expensive) instead.

So, please, find a better & more cost-effective way to measure inflation. If you do, the government will use it.

Contrary to popular belief, we don't live in the USSR - where our government provides us with blatantly crap data. We just have shill politicians that spin everything into a lie.

> Unless your CPI is comprised of raw commodities like Iron Ore and Salt and Copper

In which case it won’t be a Consumer Price Index at all, as those things in their raw forms (well, salt is close, I guess) aren’t exactly consumer goods.

Of course, we already have industry-specific Producer Price Indexes which cover the raw materials and other inputs used in those industries.

Anything that is not basically a raw element can easily have the quality dropped substantially and make it difficult to measure the cost over time.

If you take steel as an input - you can make lower quality steel for a lower price.

Even something as seemingly mundane and simple as oil has a lot of variety.

> They regularly modify the list of items in the cart, meaning the data isn't reliable over time.

It may not be reliable over time, but it is reliable over shorter time periods. If the basket is not changed for a few years, then the month-by-month changes in CPI are generally judged as reliable, or as reliable as what is out there. It is when you try to look over a 10, 20, 30 year period where you begin to run into trouble.

Reply on news.ycombinator.com