This is a woulda/shouda thing going back 60 years to the original construction of the system. Even if we gave BART billions in funding, there are certainly better uses for that money (Build a rail line into OAK, extend service on all the currently-approved-but-unfunded routes, extend to San Mateo, and deal with some expensive, lingering issues with the transbay tunnel.)
In a single-track system 24/7 operation is not possible: they need to be able to power-down the track for maintenance.
That's consistent with my recollection riding BART. Shutting down one of the two tracks for unexpected daytime maintenance ("single-tracking") is a common source of delays, since in single-track mode the system can't run nearly as many trains per hour due to the need to pass at specific points. But it can still run some trains, probably enough for a night service.
You can research this but I'm not going to do that for you. I'm comfortable that I'm correct, if you're not, well, meh. It's not personal, I'm sure you can understand.
And to your original point -- the engineer driving the train is a fraction of the cost of keeping the system running. Even if BART was dual-tracked, the salary of the engineer doesn't hardly move the needle on their costs, which include: Salaries of station agents and security, additional maintenance and wear-and-tear, janitorial (including dealing with higher janitorial costs between 2AM and 6AM than other 4-hour periods due to the nature of overnight passenger traffic), baseline power costs to keep the power rail hot at 1000+ volts over 100+ miles of track, etc.
In a single-track system 24/7 operation is not possible:
In double-tracked systems, there are crossovers (typically located near stations, but they can be anywhere) or switches to storage tracks that allow traffic to be routed along single tracks as necessary to allow track segments to be taken out of service.
That said, it's double-tracked in many (though possibly not all) sections, which might allow for running periodic service along one set of tracks, assuming switching and train control systems would allow for this. With a full circumnavigation of the Bay, it might also be possible to phase system downtime such that service might run on either eastern (Contra Costa) or western (Penninsula) lines, with service between SF and SJ. By phasing which specific track segments are down for maintenance at any one time, effective coverage of much of the system might be possible (at the cost of greater scheduling and maintenance complexity).
Triple-tracking the system could also provide for this, though at much greater infrastructure and land costs.
As it is, the 4 hour shutdown doesn't affect too many patrons, and service during operating hours is very reliable (much more so than other regional transit sywtems, or driving).
First, the city has buses running at night and there are 8xx buses that run along BART lines after BART is closed. While one's experience on BART is much better than on a bus, current options don't require that much more time.
Second, the last north-bound CalTrain leaves Palo Alto a little before 11 pm making it really difficult for people living in the city/East Bay without a car to do evening activities there. If one wants to get back, the best options are spending two hours on a SamTrans bus into the city or taking an expensive cab.
Caltrain map: http://www.caltrain.com/stations/systemmap.html
Metra map: http://www.mrl.ucsb.edu/~yopopov/rrt/us/chicago/rr_chicago_m...
A comparable rail map for the San Francisco Bay Area: http://www.urbanrail.net/am/snfr/sf-bay-area-map.htm
That puts the Bay Area commuter rail at 472 track miles and 598,000 passengers daily. Rather closer to your Chicago numbers.
But we've got to add in Chicago's CTA 'L line (709,000 daily riders, 108 miles). Tips lead to Chicago again.
Chicago benefits greatly from having grown up around (and not torn down much of its) commuter rail lines, including significant urban and suburban growth from 1860 - 1910, pre-automobile. It's occasionally noted in both areas that commute times a century ago were comparable to, or faster than, today (dedicated rights of way, less traffic, arguably fewer safety restrictions). The Bay Area notably tore down its old Key System commute rail, and several counties (San Mateo, Marin) have fought BART extensions for a half century, and saw much of its growth in the car-friendly post-WWII era.
The Bay Area's layout is actually fairly rail-friendly, with a few major transportation corridors (Peninsula, East Bayshore, CA-24, I-580, I-80, US 101 north of SF), though overall densities are low. Chicago has experienced a great deal of sprawl in the past 30 years, as well as employer flight to the suburbs, greatly increasing suburb-to-suburb commuting, though both SF and Chicago have also seen an increased appeal of the inner urban core especially for younger professionals.
San Francisco first needs to figure out what BART is - if its future is as a rapid transit municipal system (that's tied together with other municipalities), or if it should be strictly a commuter and inter-city transport system.
The current operating model is not a bad one assuming what you want is a regional commuter rail system, it fails utterly at being municipal transport.
I'd suggest that the "failures" of BART are really the extension of the massive, completely unmitigated clusterfuck that is MUNI. Looking at the recent history of MUNI is like watching a really bad slapstick act - it'd be funny if it weren't so damned sad. MUNI is broken, its system of governance is broken, and the propensity for California/Bay Area to engage in hyper-democracy is keeping it broken (ballot measure ALL the things!)
San Francisco holds 800K people in 47 square miles of land - the argument that this is insufficient density for a proper rapid transit and municipal transport system is IMO complete bunk.