It's not crazy cheap at all. The incumbent technology here is flooded lead-acid, a proven solution used everywhere from telco COs to forklifts to off-grid cabins for over a century. Industrial FLA batteries (the top grade, longest-lasting, etc.) retail for somewhere around $250 per kWh, and typically have warranties ranging from 7 to 10 years. We don't know exactly what the differences are between Tesla's two offerings, but even if we assume that the 10 kWh model is rated for continuous deep cycling for 10 years (their language suggests it may not be), we would still have to justify the $350 per kWh price tag. The other model comes in at $428. So this is somewhere around 50% more costly than existing solutions with a similar expected life span. The limitation of 2 kW continuous power is also a major drawback relative to FLA solutions, which can typically provide at least 200 A (around 10 kW at 48V, much more at the higher voltages the Tesla unit operates at). Since the Li-ion battery is maintenance-free, a better comparison might be AGM or other VRLA technologies, which are slightly more expensive than FLA, but still less expensive than the Tesla units.
Since no inverter, grid-tie transfer relay, or charge controllers are included in this unit, those will all be additional costs just as they would with any other batteries. And it appears you will be limited to a few inverters and probably a micro-inverter PV design; this is purely a grid-tie-only design. Existing battery technologies can do all of that, but they also give you much broader flexibility for other applications and system designs.
I'm not sure why I would want this. I'd be paying above the odds for some serious design constraints and limitations I wouldn't otherwise have. Tesla seem to be taking a very different approach from their launch of the Roadster, which was a super-premium offering at a super-premium price. This is a modest offering at a premium price in relative terms, but low enough in absolute terms to appeal to people who aren't well-versed in the field. Their success will probably depend on installers deciding it's easier to just take people's money and sell them the Tesla than to educate their customers and give them the best value available. Not a bad gamble, for Tesla.