This belief is unfortunately misguided; there is much suffering involved in industrial milk production [1]. It begins with the cow being separated from its calf, usually when it is just newly born. The cow generally responds by searching frantically for its lost calf, repeatedly crying out for it until its throat becomes sore. The cow might also become sick and lose weight due to the separation. It continues with the milking process, which is painful and terrifying for the cows, as evidenced by their response. Milk-producing cows have been bred to maximize milk production, which means they have oversized udders that lead to illness. These are just a few examples of many. Generally speaking, cows in the dairy industry lead a short and exceedingly miserable life.

If the goal is to minimize suffering without going vegan, I think that wild-caught fish provide the least offending animal-based food. But wild-caught fish are generally bad from an ecological perspective.

[1] See for example http://modernfarmer.com/2014/03/real-talk-milk/ , http://www.farmsanctuary.org/learn/factory-farming/dairy/ , and the documentary films Earthlings and Food Inc. A quick google search will reveal many more sources.