Having bought the idea up a couple of times, I believe the majority of academics (even in computing subjects) don't know they "should" be boycotting, and most have no intention of joining in when it is discussed with them.
I wonder if most of the noise you hear is from people attached to the 'free' journals, who obviously have an interest in getting people away from Elsevier.
Editors of Elsevier journals will send papers off for review, and get a growing number of "Sorry, I don't review papers for Elsevier journals." If you tell people that you are an editor for a nonprofit journal, everyone will agree that's a good thing, but telling people you edit for Elsevier will not have the same effect. Editorial jobs will become a drag. Morale will drop. People will quit. Journals will have to recruit lesser-known editors (and no one wants to submit good papers to journals with a bunch of B-list editors).
Meanwhile, people like Tim Gowers -- who is spearheading this movement, and who is also one of the most prolific mathematicians in the world -- will find other homes for their work. Wherever Gowers, Terry Tao, etc. publish, junior scientists would be very happy to follow. That is how the prestigious journals got that way in the first place.