The only graphics cards from the last years that don't support OpenGL >= 3.1 are from Intel before Sandy Brige. Since Sandy Brige OpenGL 3.1 (for which my remark is also true; the only difference is that in OpenGL 3.0/3.1 you probably want to use a forward-compatible context instead; this remark only makes sense with this specific OpenGL versions) is supported and since Ivy Brige OpenGL 4.0.

Thus you only drop support for computers with only Intel GPU and processor older than Sandy Brige. When I consider how many app developers drop support for older iPad/iPhone versions that are much more recent than pre-Sandy-Bridge CPUs and hardly anybody complains (the same is, of course, true on Android), I really have difficulties understanding where the problem is.