Does someone know how these types of projects come to fruition in a big company like Facebook? Are people working on them full time (with no other workload)? Do engineers build them on the weekend? How do they get 'funded'?
I think the key is that these projects actually provide value - they make things faster, more reliable, more scalable - whether that's the code's execution or the people writing the code (or debugging issues, or whatever).
They generally aren't solutions seeking problems - they are responses to problems that exist.
Engineers generally don't build things like this on the weekend - unless they like to structure their time like that, I guess. It may or may not be a full-time job, but the job whatever it is isn't some search for abstract perfection, it is again to solve real problems encountered by others in the company. Often it is a part-time component built as part of trying to solve some more direct goal - like fighting spam, or serving bits, or whatever.
Often it is something the engineers just do - it makes sense to break things up into libraries, or services, or whatever, and they do that, and then that library or service is usable/useful elsewhere, and that's it. Other times they may suggest and motivate it as a goal-in-itself in a team goal setting situation.
I doubt that's a particularly useful answer, but maybe with further questions I can make it more useful to you?
It does raise the interesting question of whether facebook employees are doing this work just to avoid the work that is the "core competency" of the company. Especially given the fact they don't gain a competitive advantage from releasing the work that facebook paid for into the wild. By this I mean, the company benefit would seem to be attract other talent. And the personal benefit for the devs is to get their name out there are something cool and interesting. Certainly, working on the best way to advertise to users is a lot less exciting/sexy than working on static type checking for javascript.
I don't think anyone is "avoiding" any work - there's just tons of work that needs to get done, and people mostly choose to work on that stuff that interests them. One could say that product engineers "avoid" infrastructure work and infrastructure engineers "avoid" product, but I think that would be inaccurate in most cases.
It seems like the vast majority of companies are not far enough out in front of their production issues and requests from the business side that engineers could do this sort of thing. So I guess it's impressive that Facebook (and probably Google) are in that position.
Documentation costs productivity to write, but when there are many people who would be made more productive from it, it makes sense to do it.
I think "slack" is the right way to think about it. It isn't a free-for-all - the business still needs to run - but there's enough space to explore, to rewrite, to document, to polish, and so forth. That's where the magic happens - the unplanned and the unexpected things around that which you thought you were going to do.
(It's a different matter that we would have done this on our free time anyway, because it's so much fun!)