> non-religious people understand church to be a building. Whereas religious people understand that the people are the church
"Church" is equivocal [0]. It can mean "a church" as in a building used for liturgical worship (and specifically, the sacrifice of the mass). It can mean the institution founded by Christ, a divine and universal (hence "catholic" [1]) society, in which case we tend to capitalize it in English. In this latter case, theologically, we can speak of the human and divine elements, where the human (here on earth, the so-called Ecclesia militans or "Church Militant") is afflicted by the ravages of sin, while the divine dimension remains indefectible. The totality of the Church also includes the Ecclesia poenitens ("Church Penitent", "Church Expectant", or "Church Suffering") and the Ecclesia triumphans ("Church Triumphant").
(As a footnote, people in countries with a Christian heritage sometimes commit fallacies of equivocation with respect to these two meanings. This leads to some absurd inferences. For example, proponents of secularism have sometimes called for the separation of "mosque and state" in Muslim countries in an attempt to mirror the liberal exaggeration of the Christian distinction between Church and State, but this is nonsensical. "Mosque" has only one meaning, namely, the building. There is no institutional "Mosque". Islam does not make a distinction between religious and secular authority (which has its roots in Matthew 22:21 [2]). In the Islamic worldview, there is only Islam - "submission" [3] - and the unconquered world of the infidels. Liberalism is, genealogically and as a matter of substance, a Christian heresy - something it shares with Islam - but one that is alien to the Islamic worldview. The secular/Islamic divide in the traditionally Islamic world is rather a result of Western geopolitical influence, an arrangement merely tolerated, for the time being at least, by devout Muslims.)
> to the religious, the use of the building for non-religious purposes is "shrug".
Perhaps for some Protestants, but Catholic churches are not simply meeting houses, but the successors of the Temple of Jerusalem where the perfect sacrifice of the mass is offered (the liturgy itself has the structure of the liturgy used in the Temple). It is most certainly not a "shrug".
> Whereas to the non-religious the idea of turning a church building into something else is some kind of desecration.
I don't understand. To non-religious people who do not recognize a religion's truth claims, it appears that there is nothing to desecrate. Catholics/Orthodox most certainly would consider the illicit use of a church as desecration (example: the man who recently climbed onto the main altar in St. Peter's Basilica).
[0] https://www.newadvent.org/cathen/03744a.htm
[1] https://www.etymonline.com/word/catholic
[2] https://www.biblegateway.com/passage/?search=Matthew%2022%3A...
[3] https://www.etymonline.com/word/Islam