Facebook first level interview for Linux admin position
bsdpunk.blogspot.com
bsdpunk.blogspot.com
I had some good questions posed to me during a Google DevOps interview, though I have trouble remembering what specifically they were about, I do remember that they were of a much more "holistic" and general bent. Instead of asking me to provide irrelevant details (i.e what are the switches for ls that list time oldest to newest) that are easily forgotten, they would pose a problem that seemed to have an easy and obvious solution and then make the problem more and more esoteric and difficult to solve.
These were the sysadmin trouble shooting problems and I found them quite interesting because they directly related to my experience and opinion that sysadmin trouble shooting is unfortunately sometimes quite complex and involves less an understanding of command syntax and more of a problem solving mindset.
Another point to take away is that if the interview is about linux, it should test fundamental linux things linux things such as testing the candidate's understanding of linux process management and signals. Perhaps filesystems as well.
My take away point is not that I think the FB interview was overly easy, but rather that it asks useless questions. This interview isn't looking for the sysadmin mindset at all.
This is just the qualifying interview.
There are certain trivia or data that the entire community of linux sysadmins know, just by having worked with the operating system - and, if you miss more than about 20% of those questions - one can say with pretty much certainty that you are not a linux sysadmin.
I thought the questions were roughly the ones I would give to a technical recruiter to screen.
(PS: ls -lart - muscle memory, I type it 100+ times a day)
>(PS: ls -lart - muscle memory, I type it 100+ times a day)
Place that in your .bashrc, or whichever shell you may be using.
Reduce typing.
ctime is not when the file was created, it's when the inode (the file metadata) was last changed, as compared to the mtime which is when the file's data was last changed. Note that changing the file's data also updates the ctime.
time_t st_atime; /* time of last access */
time_t st_mtime; /* time of last modification */
time_t st_ctime; /* time of last status change */
The field st_atime is changed by file accesses, for example, by
execve(2), mknod(2), pipe(2), utime(2) and read(2) (of more than zero
bytes). Other routines, like mmap(2), may or may not update
st_atime.
The field st_mtime is changed by file modifications, for example, by
mknod(2), truncate(2), utime(2) and write(2) (of more than zero
bytes). Moreover, st_mtime of a directory is changed by the creation
or deletion of files in that directory. The st_mtime field is not
changed for changes in owner, group, hard link count, or mode.
The field st_ctime is changed by writing or by setting inode
information (i.e., owner, group, link count, mode, etc.).
;)
You don't have time to google anything when your system is down.
Likewise, I wouldn't hire a Senior Javascript developer who can't describe in her own words what a closure is and what it's useful for, or talk about execution context (what 'this' refers to) and how to control that. You just can't have seriously written any javascript and not have encountered those things enough that they've been burned into your mind (though maybe an intern or a Junior developer that'd be okay)
Essentially, more often than not if I'm doing something shell related I'll need to turn to man pages or doc files for a specific method/flag to use on a command unless it is the 'muscle memory' type commands like ls related flags, etc. There's only so much that is important to have memorized, and only so much you can memorize. The rest, look to the docs my friend!
Namely, question 14 is odd,
What Signal does a child send to a parent when it is finished.
Yes, because wether or not it is trapped is up to the application..
b) in acutality when it comes to the kernel code if a process/thread/task has a signal ignored (the blocked semantics are slightly different) it is not actually delivered x.ref. prepare_signal() in kernel/signal.c in Linux.
Heh
If a malicious attacker can modify a kernel, no process running within the kernel can check it for consistency.
I presume 'not much' or 'you get an error', but I don't have an instance I'm prepared to sacrifice right now, just in case something hilariously destructive happens.
So my question is, is this all you need to be accepted at Facebook? or "First level interview" means more difficult stuff comes after this?
After a phone screen, they setup another call that was probably more intense. They asked me to be near a computer with a net connection so I assumed I'd be writing code or troubleshooting something.
I ended up turning it down because I received an offer from another .com that seemed more promising but I believe if I passed the next call, I would have been flown in.
Actually a child almost never sends a signal to the parent process on termination; the signal is sent by the kernel from the kernel context.
A child could manually send a SGIC(H)LD by using signal(2), but then any wait(2) would block; this is why you should always use a non-blocking flag to waitpid(2) or its ilk in case some idiot program sends your program a SIGC(H)LD.
i'm taking the fact that this article is on the front page of HN as my sign that i've outgrown HN.
It's been 5 or 6 years since I've done Linux sysadmin work, but I still knew a bunch of these.
I know next to nothing about Linux -- it's yet another multi-user, demand paged, virtual memory operating system with an 'embedded' operating system and a hierarchical file system, written mostly in C. Right? That's about all or a little more than I know about it. There have been a lot of those, back to Multics. So, Linux is yet another one. Okay.
But I'm developing on Windows. Even on Windows I don't know that much detail. For such questions on Windows, I wouldn't know a good source (wish I did), and when I need answers to such questions have to Google/Bing a lot, hit Stack Overflow, etc., one silly little question at a time. Bummer, but the frequency with which I have to do such scavenger hunting in muddy swamps has been declining.
No doubt when I bring up Windows Server I'll have a lot more such detailed mud wrestling to do; similarly for serious work with SQL Server. Bummer -- again I'll be screaming at the lack of good documentation for such questions.
Still, so far my software development is going ahead well without such detailed knowledge. Actually I've long suspected that once my company gets serious with number of users arriving per hour and revenue getting up to the useful level, I'll get answers to such questions by calling for paid, expert support at Microsoft. It's their software, not mine! And while most such questions are simple conceptually, digging out the answers often is not.
Indeed, one reason I selected Windows instead of Linux is the hope, maybe false, that one way and another, say, from Microsoft or Stack Overflow, books, Microsoft's MSDN Web site and the similar site for SQL Server, I'll have better means of answers to such questions. Similarly, when I need a high end LAN switch, I'll try to pick Cisco heavily due to a hope for good sources, possibly from Cisco, for answers to such questions. LAN switching is their darned business, not mine; they can do their business, and I'll be a paying customer, and I'll do my business.
For hiring, there's a recent remark from Richard Branson that he hires for personality and not skills since skills can be taught, he believes quickly, while, implicitly, it appeared he was saying that personality could not be taught quickly. Actually, for such detailed questions for Linux/Windows Server, he is significantly correct. So, I'm hoping to hire bright people with good personalities who are at least okay computer users (even if they have never written a line of code in their lives) but are good at typing, reading, writing, speaking, 'critical thinking', getting some okay understanding from some horribly badly written material (i.e., nearly all computer industry documentation), and analyzing ambiguous situations. Right: Sounds like I might want to hire some good humanities majors! Nerd me!
Then have such employees start with simple things, say, plugging together a server, working up a simple, quick and dirty parts inventory system (maybe just a flat file with some keywords maintained with just a text editor maybe with a few macros -- maybe even a, yuck, spreadsheet), pulling CAT 5 cables and using a crimping tool for the ends, installing Windows Server, calling Microsoft for technical details, documenting what they learned about how to install, configure, and administer Windows Server, etc.
Then, teach them programming a few lines at a time, e.g., Visual Basic .NET (apparently about the same as C# for getting to the CLR and .NET Framework but, to my eyes, easier to read on the page than C# with its syntax from C and, thus, deliberately idiosyncratic).
For basic computer and network architecture, I'll give them a 30 minute overview lecture they can capture on video and use for other new hires. The hope is that such people will be able to grow in 'skills' as fast as the company grows. I.e., I'll show them what I already know, and as we need to learn new things it's their job to dig them out and document them for others. Did I mention I want them to know how to write (and not learned in the computer industry)?
Then for Facebook hiring, I would ask: Whatever the heck happened to the last person in that position? And, why the heck does Facebook have to go recruiting for such a person? You mean, Facebook doesn't have a backup for such a person? Or, Facebook has 400 billion executing instances of Linux, may be compiling some unique versions of Linux, and, still, has only one Linux system administrator? Not good.
Broadly, if Facebook needs such 'skills', then they should grow the capabilities internally. Or, I can do my startup without knowing all that stuff; if Facebook would want me to know that stuff, then they should pay me to learn it.
This stuff about having to walk in with a lot of detailed knowledge, learned on my dime and time, just as needed by Facebook for just an 'employee' position in a big company, that is, an organization that in a few years might just decide to layoff 20,000 employees, is getting to be a bad, old story. A startup, even for grass mowing, literally, looks more promising.
Let's see: For such scaling, it has long been the case that a first-cut quantitative approach has been theoretical and empirical network queuing analysis. So, how many such Facebook scaling architects are able to do such analysis, either theoretically or empirically? I'll bet, not many. Net, I doubt that the Facebook scaling architects have a very good approach to the problem.
What are they doing? They have a good grasp of the basic issues of the components, that is, the server hard/software, NICs, LAN switches, TCP/IP stack details, Linux thread switching performance, etc. Then they concentrate on the bottlenecks, solve those empirically one at a time, and otherwise just f'get about the rest. Right? I mean, what the heck else are they going to do, dig into the Poisson process hypotheses and the renewal theorem? I doubt it.
So, sure, a humanities major won't come knowing all about the time/date stamps on Linux files or about file locking. That may be stuff they have to learn.
So, for more, need people who can learn stuff, find materials for the learning, work with people for the learning, use some critical thinking about the material and how to use it.
Then for implementing the results of the analysis, have to involve other people, in software, hardware, cable pulling, HVAC, electrical power, ..., lots of other people. So, now have to work with people -- with technical papers, planning documents, foil decks, presentations, meetings small to large, one on one discussions, etc. So, have work with egos, power struggles, turf battles, office romances, human nature, etc., i.e., the stuff nerds ignored (I'm an expert nerd) in high school, college, and beyond. The Heine-Borel theorem, fine; human nature, f'get about it.
So, after I saw Branson's remark, I happened to visit the Web site of my college and, there, saw some of the undergraduate students they were proud of. I doubt that any of those students knew anything about Linux internals, but one of the students in just her extracurricular activities has organized nearly everything on campus short of the birds in the trees. Give her a couple of years, starting as I outlined, yes with her able to learn about time stamps on files, etc., and when it's time to scale the Web site, along with moving it to a colo or our own building, I'd bet she'd knock the socks off nearly all nerds, including me. And, if really need some good work, say, in queuing, bet she'd blow away the self-taught Linux hackers who started with C++ in middle school; for one, she'd be smart enough and good enough to pick a good consultant in, say, computer and network performance at a computer science department and make good use of some consulting.
Again, I bet that so far the Linux performance details remain nearly all fairly simple conceptually. Yes, good thread switching likely remains a challenge, but that's quite specialized work for which draw the larger lessons from specialists.
I'll provide an example: I was working in artificial intelligence (AI). We developed an AI language and did joint work with MET LIFE and GM and gave a paper at an AAAI IAAI conference at Stanford.
My wife wanted to learn some of this material. So I gave her a lecture of about 15 minutes on how to connect to the server and use it and then another 15 minutes on our AI language. She then took two weeks and wrote a nice, first AI program. Right: BAM, with next to no background in programming. I looked at her program and said, "It's a program. It works. It does what you wanted. There's nothing fundamentally wrong with it. But 'AI' has a 'belief', a 'theme', maybe just a 'hope', and you didn't really illustrate that ...." So, building on some of what she'd studied as an undergraduate, I explained the 'dreams' of the MIT 'school' of AI. She smiled. Two weeks later she had a really nice first AI program. "Nice"? Our project had as programmers some of the brightest computer science students in the country, who were taking computing and AI very seriously, and they had written first, sample programs. Her program was several notches better, the best first AI program our group ever saw. She did that work with all of about 45 minutes of instruction from me and four weeks of time.
Of course, she was 'cheating': She was brilliant, not just a little but the real thing. Yes, high school Valedictorian and in college PBK, 'Summa Cum Laude', Woodrow Wilson, NSF, etc. She got her Ph.D. in essentially mathematical sociology from one of the world's best research universities and where two of her professors became President of the American Sociological Association. Did I mention she was brilliant?
This detailed computer stuff can be picked up, fairly quickly, especially if it is well presented. Good students can be darned good at picking up such stuff, including from some horribly badly written documentation, e.g., as in the computer industry.
I'm guessing that maybe Branson had a point. That is, it may be better to get good smarts, ability to learn, especially from poorly organized materials, really good at the '3Rs', and really good working with groups of people and just pick up over, say, two years, what need to know about a growing server farm based on Windows Server and SQL Server and my software (a particular, just one, growing server farm, not all possible growing server farms) instead of hiring for Windows Server, etc. right away. Maybe.
I'm not fixed in my view but wondering.
Where am I right/wrong?
And this is what Facebook and other companies do; they hire those consultants.
Spot on! Another way of looking at it is that tech companies don't want to spend $ training their employees (including a slightly longer ramp up time for an employee to begin contributing). If they did, the hiring process would be: you had a 4.0 in college. This shows you are good at learning new things. Lets check for personality fit and some other things that a college transcript does not capture. Do a one hour interview. Hire or No Hire based on this.
As a startup founder, you should be happy about this state of affairs. The attitude of large companies makes it easier for you to hire :)
So, they have great SAT scores, know how to study and learn, know what high quality work is, are good at working with people, are really good at reading comprehension, can knock out an A+/A+ term paper between 10 PM and 7 AM with just a quart of coffee, can give a good presentation, etc.
So, teach them about Volts, Amps, Watts, Ohm's law, DC, AC, resistors, capacitors, inductors, basic electronic signals, the basic conceptual principles of how a processor core works, a little on caching and main memory, data representation 101, how a disk drive works, the basics of an NIC, Ethernet, a LAN switch, and TCP/IP, and IP router, a little on memory protection, an 'embedded' operating system, virtual memory, virtual machines, a file system with locking and security, console sessions and command lines, text files and a good text editor, the Internet from DNS, SMTP, POP3, to base 64, MIME, HTTP, HTML, and CSS, the basics of programming languages, e.g., Visual Basic .NET and its memory management, classes and instances, the .NET Framework, relational database, SQL, SQL Server, how to use Microsoft's MSDN, about Stack Overflow, HN, and how to Google/Bing to get answers to questions, how to use our technical support account at Microsoft and Cisco, all in about one afternoon, with some exercises, and then have a graduation party.
Then get to work on some basic stuff, say, plugging together another 20 servers, repairing a server with a busted hard disk, etc.
Clean, indoor work, no heavy lifting, and better than grass mowing or retailing.
How is this lost on people??
*Sorry for the self promotion, but I don't have any better examples right now.
[1]: http://serverfault.com/questions/320181/dnated-packets-disap...
[2]: http://serverfault.com/questions/542381/openvpn-forwarding-s...