Since we're talking about the Shannon limit, why does a 56 Kbps modem not violate it?

The common wisdom in the early '90s was that the fastest possible modem for a dial-up connection would be ~35 Kbps. I even remember reading this "fact" in a communications theory book from that era. Analog modems could never be faster. (It's assumed that US and Canadian analog telephones have a 3 kHz channel.)

Then why does a 56 Kbps modem not violate the Shannon limit for data transmission on a 3 kHz channel? How is it possible to break a fundamental principle of information theory?

This question has bugged me a lot. Based on some research on web, I tried to put together an intuitive answer to what happened. Here's my answer, but I welcome insight or corrections from people knowledgeable in telephony or information theory.

At the time the book was written--when people believed that dial-up modems could never be faster than 35 Kbps--the telephone network was perceived as being analog end to end (though lots of it was already becoming digital), like this:

    User<------------{ telco network }------------------------>ISP
           (analog)        (analog)      (analog)

A critical assumption in calculating the Shannon limit was the noise floor of the network, which was taken to be 35 dB, a figure based on an all-analog network. (A higher dB number is better for transmission quality.) The Shannon limit would be 35 Kbps based on a 3 kHz bandwidth and that particular signal to noise ratio of 35 dB. ( bps = BW log2 (1 + P/N) = 3000 log2 (1 + 10^(35/10)) = 34881 bps )

However, by the mid-'90s, most of the telco network became digital. Only the customer loop--the connection between the user and the telephone company--remained analog. So now, with respect to the Shannon limit, only the noise floor of the customer loop matters, since the rest is digital:

    User<------------{ telco network }------------------------>ISP
           (analog)        (digital)     (also digital)

It turns out that it is possible to achieve a much better S/N ratio when only the customer loop is analog. The Shannon limit would be based on a 3 kHz bandwidth (the same bandwidth as before) and but a much higher noise floor of 98 dB for the customer loop. The Shannon limit would now be ~97 Kbps. ( bps = BW log2 (1 + P/N) = 3000 log2 (1 + 10^(98/10)) = 97664 bps )

The Shannon limit still exists, but other factors related to the network would limit the data rate before this bigger Shannon limit takes effect.

So the key thing that the experts and the communications theory book got wrong was to neglect the hidden assumption about S/N ratio. The bandwidth matters, but the S/N ratio matters too.