You're absolutely right, which is why that's exactly how poverty is measured. :) The World Bank actually uses a PPP measure of $1.25 per day in 2005 dollars for "extreme poverty". Let's break that down:

First, as I said, the World Bank uses a PPP measure of $1.25 per day. PPP stands for Purchasing Power Parity, which means that this is measure doesn't mean "has US $1.25 of local currency at current exchange rates", but instead means "has enough local currency to buy a basket of goods and services which could be purchased for $1.25 in the US". Or in more concrete terms, the World Bank is defining it as "can buy as much rice or flour as would cost $1.25 in the US", not "can buy $1.25 of rice or flour at local prices".

Second, the measure is in constant 2005 dollars, which means that it is indeed indexed for inflation.

And now you know. :) (Other poverty measures do focus directly on calories or food, but food isn't the only good or service the very poor need to purchase. That's why the World Bank uses a basket containing a mix of goods and services. Nothing's perfect, but the Bank's measure is pretty good at accurately tracking the condition of the poor both over time and between countries.)