> I do know some things about containers.
Great! :D
> With Sandstorm.io, the rule of thumb we landed on was that a container takes 100MB of RAM. Some apps used more, some used less. This is real, empirical data.
I hate to keep harping on this, but what do you mean by "used" here? Are you saying that the RSS was 100MB per container, or that the sum of private mappings used was 100MB (/proc/$pid/smaps)? Did you use a filesystem like overlayfs that facilitates page-cache sharing by allowing read-only inode sharing between different containers, or a driver like {btrfs,devicemapper,aufs} that didn't? (Sandstorm was kicking around a while ago, so this might've been before overlayfs was in mainstream use.)
I know I probably look like an asshole, but I actually don't think I understand what you mean by used -- because saying that a container uses 100MB (which I take to mean that each new container spawned costs 100MB of real mmeory) simply doesn't sound right. It implies you could run less than 80 containers on an 8GB machine, and I doubt that Sandstorm programs were this big. It's like someone telling you that they spent $1500 on lunch -- I'm actually confused what the word "spent" means here.
I'm sure that you'd see less RSS memory usage by only having one process, but it is very possible that this memory usage benefit is not actually real -- I could be completely wrong but I'm just having trouble understanding how putting the same code inside a single program could make such a difference (given that the page cache already shares the memory for the V8 process). If you were to run 10k V8 processes your RSS would increase by 10k but your real memory usage would only increase very slightly because of the minor kernel memory cost of "struct task_struct".
As for context switches, yeah okay that's a cost you pay by having more than one process on a machine. But you do still have context switch overhead (though obviously it's much smaller) if you're running more than one program's state in a single process -- you have to switch context to a different set of protected variables right?
> And no one wants to write C, they want to write JavaScript.
My point about page caches is that the code for V8 is in the page cache and thus running more V8 processes doesn't take up any more real memory than just running one. So that cost of running JavaScript on top should be similar (though probably slightly larger because V8 stores other program information in memory -- but definitely not 10x larger).