littledanehren

Note: This is my own opinion, and not that of my employer (Google) or based on trade secrets or other IP from my employer.

Keeping pre-erased blocks is useful, but it can only reduce the write latency to what the chip gives you. And the chip gives you a longer write latency than read latency, especially for MLC with smaller process sizes.

Note: This is my own opinion, and not that of my employer (Google) or based on trade secrets or other IP from my employer.

I don't know if this is actually true in SSDs that are actually used, but there is a real cost/write-performance tradeoff. You can make writes faster by using SLC rather than MLC and by having a higher proportion of physical capacity to reported capacity in order to reduce write amplification. Looking into the future, there could be a journal in an even faster kind of memory like PCM or memristors. Still, the controller can only do so much to hide the properties of the chip, and if your drive is only flash, then the trend in the recent past for the chips is that reads are getting faster and writes are getting slower (based on the move towards smaller process sizes and more levels).

Lwn and linuxnewbies aren't perfect, but they're some of the best resources we have. There are also a few books out there that are pretty useful, but they are always out of date. No need to start from scratch and replace these things;it's easier to just work on improving what we have already.

Linux is a huge program developed by tons of people over decades with limited focus on docs in the past and high code turnover today. Getting useful information about it is just hard.

Can we really credit this entirely to the increased efficiency of iPads? There are lots of usages of RAM--now people carry around cell phones with more RAM than before. And data centers can make use of tons of RAM--the main constraint is its price.

Maybe this is actually a result of bad general macroeconomic conditions. Nobody wants to spend too much money, whether it's consumers or people setting up data centers. That's always been the case, but we're poorer than before now, and RAM is one of those things that you can cut down on, or stop increasing as quickly on.

This news is unfortunate.

Tell your Google recruiter about your competing offers, especially if you're considering choosing another job on this basis.

Why do you want to do this, anyway? Linux already isolates processes' memory from each other. With cgroups you can ensure that resources are allocated fairly, and with chroot and namespaces you can ensure that they're securely isolated from each other. Why run a whole bunch of kernels on top of other kernels? It just adds inefficiency.

Good point about C. You misunderstand the intent of my post though.

First, as a small point, I was not seriously suggesting using thread pools to implement mlock_eventfd. That's just a way that we could play around with it short of a kernel implementation, which would probably be more efficient.

Second, I don't think I'm dogmatic about mmap. It's just a cool interface that would be nice to see more widely used, and I was suggesting that it might be more usable with a small addition. Commenters here, at my blog and on Reddit have pointed out other problems, and I'm happy to learn about these. If mmap isn't good and we should keep using DIO, then I think there are still possible improvements that we could make to the interface to allow better cooperation with the kernel.

Yeah, and there's HTML too! Complete bullshit!

I don't get it. He was spending $40,000/month on his living expenses while being afraid that he wouldn't be able to afford diapers and food for his baby? How was this guy running his life so wrong? Did it somehow cost him that much to maintain his unsellable house in the financial district of New York? Are we supposed to feel sympathy for him?

Yep, that's basically what we do. User-level threads are called 'coroutines' and they're scheduled cooperatively, so they don't incur the overhead of user-level preemption. Cooperative scheduling is fine for us because we can ensure that a single coroutine won't hog the CPU or issue blocking system calls.

It'd be nice if some pthread implementation gave us what we need, but I don't think they do. They probably are too heavy an abstraction, even if implemented in user space, providing too strong guarantees for context switching and scheduling. For example, we're never going to set individual signal handlers on different coroutines, but pthreads guarantee this capability. So even user-level pthreads which don't support preemption will require a system call on context switches, just as ucontext.h's swapcontext does for similar reasons. But as Slava points out, the performance penalty is something that we need to verify in the future with benchmarks.

What would be really nice is if the OS, or at least a standard application framework usable from C++, supported this. But it doesn't, so we build our own.

This is something which will be good to look into in the future. It may be that, in certain system configurations, that's fine. But if there are 10,000 coroutines each with 10 MB stacks, and memory overcommitting is turned off or configured to be relatively conservative (as some system administrators like to do), you'll need to reserve 100GB of physical memory + swap just for coroutine stacks. This may be problematic, but it may not. In the system I've implemented, the stack size is a parameter which can be varied when the database is launched, so this is something which we can easily benchmark in the future.

Thanks for the suggestion! I'm actually looking into doing something like that right now.

I guess telling Google what they should be doing differently might work, but I wasn't really prepared in my interview to give them advice. Still got accepted, though.

A problem with this is, most people don't have the money to just start investing in startups. This is one reason why startups tend to need VCs in the first place. It's not like the only thing holding people back from investing in startups is willpower.