The models are trying to predict based on various factors, CO2 being one of them. If you were correct all you are really saying is that the models could be much simpler than still be precise. So on that alone it seems like you are missing the point.
Second of all, while most agree human activity have some correlation with temperature rise we have no actual idea how much. There is no scientifically demonstrated percentage of the temperature that can be attributed to humans.
With regards to the permafrost. This is pure speculation and in many ways relies on the accuracy of the models and doesn't take human ability to deal with it into account. It might be bad it might not, regardless we can't stop living the way we do just because there is some percentage risk of bad outcomes.
We should always take the climate seriously but so should we any other event like astroids hitting earth, super vulcanos, nearby super novas, pandemics etc. The world is always and have always been filled with dangers which is why 99.99999% of all species ever existed are extinct way before humans were around.