Solar intensity has increased slightly over the last few billion years, but previous changes in climate have been driven primarily by Milankovich cycles, volcanic emissions, and plate tectonics.
That the post-industrial rise in atmospheric CO2 is of anthropogenic origin is hopefully not a point of dispute, but it is demonstrable if necessary. Thus it remains to show that this must raise the equilibrium temperature. So, as you say, CO2 selectively absorbs outgoing IR. In the lower atmosphere, this actually does not have as much of an effect as you might think. Water vapor blocks quite a bit of the absorption spectrum, and the effect of CO2 is more-or-less saturated already.
The mean free path of an outgoing IR photon in the lower atmosphere is quite short. Absorbed photons are re-emitted in a random direction, but take an overall upward course, the mean free path rising with altitude. At the (radiative) top-of-atmosphere, the mean free path is infinite: the photon is more likely to leave Earth. At the edge of space, there is essentially no water vapor, so the action of CO2 is greater. The effect of increasing the amount of CO2 in the atmosphere is to push the CO2-dense region of the atmosphere further out into space. Photons must take a longer path out of the atmosphere, and this must raise the overall temperature of the Earth proportionally, specifically by 3.7 W/m^2 per doubling of CO2, which is commonly held to be equivalent to 1 degree C of global temperature. This must be the case unless our understanding of thermodynamics is very wrong (and if you have an issue with thermodynamics then you have some pretty serious issues).
So, one degree C ain't so bad, right? Well, it wouldn't be if that were it. However, there are several problematic feedbacks. One is that melting a lot of ice lowers the Earth's albedo, which causes it to absorb more heat. Another issue is that there is a lot of this "water" stuff around, which is very readily absorbed by the atmosphere, in a manner that increases very sharply with temperature. Water vapor is a much better greenhouse gas than CO2 by all accounts.
Climate science is not an extrapolation from the temperature record. There is a solid minimum bound on the temperature effects of doubling atmospheric CO2, and a variety of amplifying positive feedback effects. So far, in the last twelve decades, we have not managed to find anything which would reduce those effects to something manageable. At this point, the effect would need to be both very large, in order to offset the strong H2O feedbacks, and very small, to not have been noticed. The most plausible option would be "something poorly understood about the H2O feedbacks". I believe the most successful of such theories would be Dr. Richard Lindzen's Iris Hypothesis, which has generally failed to find support. At this point, there are no particularly plausible mechanisms which would transfer this extra energy to space, and if those did exist, then they would not necessarily be a non-issue: even if thermodynamics and optics are entirely wrong, the planet is warming, and we will have to deal with that even if it can't be prevented.
If you have any other questions, or would like citations for any of the above, do feel free to ask.