A CS degree prevent you from making a lot of obvious (if you have a CS degree) and costly mistakes. It sort of gives you a crystal ball. You can see that some code isn't going to work when a db table gets to 100,000 records, or that some code is making the wrong space/time tradeoff, or that some code is using the wrong data structure from the standard library.
When your performance monitoring tool is telling you something is slow or leaking memory you have the foundational knowledge to understand why and fix it, rather than spending money on 10 more dynos or whatever.
Computers are so fast and cheap today a lot of this doesn't matter most of the time. The naive solution does just fine. But when the core dump hits the fan you better have one or two CS grads on staff.