If an expert in say, economics is telling me the the climate crisis is exaggerated, and an expert in climate science is saying that it isnt - I will tend to believe the expert in the most relevant field.
[1] Calling it a single field isn't really fair. It's the nexus of many specialized fields, most of which are purely physical sciences.
Additionally, the processing of said historical data also seems to be statistical in nature, since there is a lot of missing data once we go past a few decades, data is from a disparate variety of sources, most are proxies for temperature and CO2, etc.
So, it isn't obvious to me they are 'purely physical' as in there is some kind of direct reading off of physical facts. Instead, it appears most are a lot of inferences derived by fitting models to bodies of data, which entails a certain amount of uncertainty and a statistical nature.