Yes, you can charge people without the address field. You still need the CVV code for fraud protection.

I read somewhere that credit card companies charge lower rates if the address is provided as well. If that's the case, it seems like Stripe normalizes the price for you.