San Francisco first needs to figure out what BART is - if its future is as a rapid transit municipal system (that's tied together with other municipalities), or if it should be strictly a commuter and inter-city transport system.
The current operating model is not a bad one assuming what you want is a regional commuter rail system, it fails utterly at being municipal transport.
I'd suggest that the "failures" of BART are really the extension of the massive, completely unmitigated clusterfuck that is MUNI. Looking at the recent history of MUNI is like watching a really bad slapstick act - it'd be funny if it weren't so damned sad. MUNI is broken, its system of governance is broken, and the propensity for California/Bay Area to engage in hyper-democracy is keeping it broken (ballot measure ALL the things!)
San Francisco holds 800K people in 47 square miles of land - the argument that this is insufficient density for a proper rapid transit and municipal transport system is IMO complete bunk.
Caltrain map: http://www.caltrain.com/stations/systemmap.html
Metra map: http://www.mrl.ucsb.edu/~yopopov/rrt/us/chicago/rr_chicago_m...
A comparable rail map for the San Francisco Bay Area: http://www.urbanrail.net/am/snfr/sf-bay-area-map.htm
That puts the Bay Area commuter rail at 472 track miles and 598,000 passengers daily. Rather closer to your Chicago numbers.
But we've got to add in Chicago's CTA 'L line (709,000 daily riders, 108 miles). Tips lead to Chicago again.
Chicago benefits greatly from having grown up around (and not torn down much of its) commuter rail lines, including significant urban and suburban growth from 1860 - 1910, pre-automobile. It's occasionally noted in both areas that commute times a century ago were comparable to, or faster than, today (dedicated rights of way, less traffic, arguably fewer safety restrictions). The Bay Area notably tore down its old Key System commute rail, and several counties (San Mateo, Marin) have fought BART extensions for a half century, and saw much of its growth in the car-friendly post-WWII era.
The Bay Area's layout is actually fairly rail-friendly, with a few major transportation corridors (Peninsula, East Bayshore, CA-24, I-580, I-80, US 101 north of SF), though overall densities are low. Chicago has experienced a great deal of sprawl in the past 30 years, as well as employer flight to the suburbs, greatly increasing suburb-to-suburb commuting, though both SF and Chicago have also seen an increased appeal of the inner urban core especially for younger professionals.