>Price shouldn't be based on the cost of production, but rather the value derived by the buyer.
As to what price "should" or "shouldn't" be based on, I'm confused with the meaning here. Are you saying that we ought to change society such that this is true, or that when accounting we shouldn't consider the cost of production?
>As a consumer, I don't care at all what the manufacturer paid to make the thing.
This is exactly true, and actually this was recognised even by the classical economists and Marx: value (in labour hours) works behind the backs of both producers and consumers[0], because it is precisely socially determined by the market. Consider that if you spent hours making shoes nobody wanted, there are two explanations as to why you stopped eventually: (1) nobody "derived value" from them, (2) the labour embodied was not socially useful. These two claims are not actually in contradiction with one another.
A society in which price of a freely reproducible commodity (or some quantity transformable to price under the right conditions) is dependent on the cost of production is precisely one dependent on the value derived by all the buyers. You don't need to care about how long it took, because that's taken for you through the abstraction of labour. Nobody knows how long it took since to work out such a thing would be as futile as adding up the prices of the labour and tools required to make the good, the labour required to make those, the labour required to make those, to extract the raw materials to make those, to produce the electricity to power the machines etc. - all of this is taken care of and reflected in the final price.
Your argument is similar to the one put to Ricardo: pearls don't fetch high prices beacuse men dive for them, but rather, men dive for them because they fetch a high price. The fact that they fetch a high price is explained away along the lines of "people just find them desirable, it's just a matter of subjective preference/utility". Cockshott[1] asks us to think about it more: would the pearls fetch such a high price if they just washed up on the shore? And indeed, rich people wouldn't even want them any more - they want to be seen wearing something expensive that required a lot of time and effort to procure.
So Ricardo really says that the precondition for a commodity having "value" is that it is an object of demand. Marx roughly agreed, to which his theory of socially necessary labour time appears.
[0] "The different proportions in which different sorts of labour are reduced to unskilled labour as their standard, are established by a social process that goes on behind the backs of the producers, and, consequently, appear to be fixed by custom." - see also the theory of commodity fetishism which obscures the real concrete labour content in favour of its abstract representation in the Price-form.
[1] "Neither does it matter what people are willing to pay
for X. The market will be in equilibrium when demand equals supply at a price in line with labour content. If you’re not willing to pay that much, you don’t get any. If you’d be willing to pay a lot more you are in luck, you don’t have to. In the limiting case where nobody is willing to pay a price for X that corresponds to its labour content, X doesn’t get produced."