No, Classical music was popular
period. But the elites consumed most of the professionally performed music as it was expensive, whereas other people had to rely on friends and families who could play an instrument themselves, or go to a place where there was a free or cheap performance, for example Church. But just because the elites consumed most output of professional musicians did not mean that ordinary people did not consume the output of amateur musicians, especially in their own families, where it was not uncommon to perform music after a meal or during family gatherings.
Many families would have at least one member who could play the piano or the violin - more often the violin. Here is a noted British musicologist reporting on his travels through Europe:
"I crossed the whole Kingdom of Bohemia from south to north, and being very assiduous in my enquiries, how the common people learned music. I found out at length that not only in every large town, but in all villages, where there is a reading and writing school, children of both sexes are taught music”[1].
Yes, many of these violins made by local luthiers were cheap but by modern standards they were serviceable. In terms of people who did not own their own instruments, they heard organs and choral pieces in Church -- the Church was the primary provider of public music at the time, and sacred music was until recently the most commonly performed type of music, and indeed the development of Western music began with sacred music (the Byzantine empire was responsible for elaborating on the music of classical greece and giving us much of western music theory).
When public operas began to open they were wildly popular. Mozart wrote for one of these. This is much like the situation with restaurants. Originally the elites had private chefs that cooked for them. Over time, these chefs began to open their own business and cook for the public. Then over time, the food got cheaper with casual dining.
But this is not to say that people didn't eat traditional meals, they just cooked for themselves, as amateur cooks. So with music, they performed for themselves as amateur musicians. It is interesting to note that the first uses of copyright were not recorded music but sheet music, that people would pass around, copy, and perform for themselves.
But "free" public music -- e.g. apart from a wealthy benefactor paying for a public concert or something outside of a Church -- did not become widespread until the advent of the radio.
[1] The Present State of Music in Germany, the Netherlands and United Provinces. London 1775: newly edited by Percy A. Scholes as An 18th Century Musical Tour in Central Europe and Netherlands. London Oxford University Press, 1959) Cf. Chapter X, p. 131,32