Ask HN: Is it time to measure Inflation and CPI without the government? | Hacker News Reader

Ask HN: Is it time to measure Inflation and CPI without the government?

16 pointscyrusradfar6 comments

The Bureau of Labor Statistics (BLS) cancelled the October CPI report and delayed several other releases. Key series such as CPI, real earnings, state JOLTS, and county wage data now have gaps or shifted timelines.

Independent analysts note that missing a full month breaks continuity in the most widely used inflation signal in the United States. Forecasting models lose a key input. The Fed will make its December decision without two consecutive inflation prints.

My assumption is that we should not depend on a single centralized (potentially untrusted) source for something this important.

Private data sources already track groceries, rent, CPG pricing, energy, e-commerce, shipping, and wages in near real time.

---

What would it take to build a community-driven, open CPI alternative that is transparent, reproducible, and tamper-resistant?

Some starting points:

  * Multiple contributors could publish item-level price feeds (groceries, rent, utilities, services).
  * Weighted baskets could be open and versioned.
  * Methods could be fixed, auditable, and stable over time.
  * Aggregation could be public code using public data where available.
  * Licensed private datasets could be added for specific categories.
  * Output could be monthly, weekly, or daily depending on the data.

A decentralized CPI would not replace official CPI, but could offer a continuous, independent signal.

I'm looking for practical approaches: data sources, weighting schemes, methods, and any existing open projects that are doing this today.

The October CPI report was canceled and delayed because they couldn't retroactively collect data during the period of the government shutdown.

You may be interested to know of MIT's Billion Prices project, which scraped the Internet to calculate a CPI. It mostly aligned with BLS numbers, so I'm not sure if it's really worth the effort to repeat. https://en.wikipedia.org/wiki/MIT_Billion_Prices_project

I encourage you to get clear on your goals for a project like this. There is a political and social dimension to this work aside from the technical challenge.

For example, ordinary people already know whether they're being squeezed or not. They don't need a CPI report. If you want them to believe your CPI report over the government's, then you have to do a lot of marketing. On the opposite side, government decision makers are generally moving away from a "technocrat" data-driven approach to decisions, towards a "great man" theory of decision making: giving them better CPI data won't suddenly make them care about using data again. A lot of people are putin in work to dismantle American government capacity, and there are worse problem than inflation tracking.

My personal opinion is that it doesn't make sense to create a community-driven CPI as a direct alternative to the government CPI. But it does make sense to collect a lot of this data for other uses (ex: to highlight illegal collusion among merchants, archival).

From a technical perspective, here are some thoughts from a Canadian lens: * CPI calculations are already fairly transparent. You can read about the work that goes into them (click TOC on upper right: https://www150.statcan.gc.ca/n1/pub/62-553-x/62-553-x2023001...) * Government-collected raw data is not open to the public, because merchants wouldn't have give the govt' access to their POS data if it were otherwise. The Canadian government made a tradeoff: it gets very accurate data, in exchange for it being hidden from the public. * As WheelsAtLarge mentioned, data collection and analysis takes effort and money. Especially when you're doing it for the long term. You'll have to think of funding and keeping people motivated to do the work. * Scraping data might be a lot harder than you expect. I compile the largest open dataset of grocery prices in Canada, and have 20 months' worth of data. Its very difficult to average prices across merchants (because many don't reveal product UPC codes). And prices vary by geography. None of this is insurmountable, but it takes effort and money. Go ahead and play with the data to get a feel for how easy/difficult this is (https://jacobfilipp.com/hammer/)

I hear and accept all your points. I def. recognize the marketing difficulty.

Frankly, I was assuming there were folks doing this and I'd get educated on that.

Also, really good info on Canada and kudos on compiling that data. I followed your article to this one: https://jacobfilipp.com/the-story-of-canadas-bread-prices-in... which was a good read.

Thanks for your work.

Yes, it would be great to have a second source but no one has the resources the government has to gatherthe data. They do a pretty good job and they try to be as accurate as possible. I can't see any another organization gathering the data without needing to be greatly compensated.

As far as trust, there have always been people that have and will criticize the results but they only have a small view of the prices which doesn't give a true view of inflation. So their criticism is invalid.

The problem with a decentralized version is that people will need to be compensated. People might get together to do it a few times but for it to be useful it needs to get done for as long as there's an American economy. It won't happen without some kind of compensation.

For rent, scraping property rental websites would work. Same for grocery/etc.

>we should not depend on a single centralized (potentially untrusted) source

>Is it time

[1972]?

Reply on news.ycombinator.com