Linux Sysadmin/DevOps Interview Questions | Hacker News Reader

Linux Sysadmin/DevOps Interview Questions

github.com

62 pointsmetmac62 comments

Generally good questions.

However I would not ask a candidate or more specifically I would not show him the code for fork bomb. I would ask him what fork bomb is or demonstrate it on some demo VM and ask him how would he prevent himself..

There's too many technologies already. I would not ask specific questions about the staff we use but instead a very generic ones.

Like really it's almost impossible to be a specialist these days and you can pretend your company is full of rockstars and works on the next big thing but in fact you are just bunch of posers.

Nobody really knows everything to the detail. I really hated when my colleague asked new people really deep stuff about Puppet and then we have switched to Ansible and half of the team was clueless and had to learn along the way. That's what good engineers do. They adapt. They learn the concepts and focus on delivering a good service. They do not memorize man pages.

When it comes to scripting / programming do not print scripts on paper and make the candidate to describe you what is it for or find a bug / mistake just by reviewing the paper. It's insulting. Especially if you do not follow any conventions or your code is not idiomatic.

Also I would never ask a question "What does the following command / parameter do". These are not good questions. Ofc they are asked often during interviews but are just plain stupid.

How would you display last / first 10 lines of a file ? How would you remove an X string from a line / row of text ?

and so on...

Honestly I think this is a good list of questions to test out a search engine.

As far as interviewing a DevOps guy, maybe? I consider myself a decent DevOps person out of necessity, but questions like "How to add a new system user without login permissions?" would kind of annoy me, because that's the exact kind of thing I can't remember because I automated it years ago at my current job. I probably add a new user twice a week... from a Fabric script. I couldn't tell you if it's useradd or adduser because my workflow would be to google the exact wording of the question if I actually needed to do it manually for once.

it depends on the OS and that's not the important part of the question, as far as I can tell.

I think it's more important you are able to understand and be able to communicate about the questions more than have accurate answers. Being able to hold a technical conversation about a question you don't understand shows that you could solve the queston in a non-interview setting. Everyone knows and expects web access in the field, it's more important that you understand the overall concepts.

One could also be pedantic because a "system" user has special meaning and does not have a login or shell when you create one.. A lot of people however say "system user" to mean "a user on the system" and not necessarily different from a normal user.

I'd still ask the question, your answer - even though you don't know - is good: it indicates that you've probably used useradd and adduser (it's both) and I'm sure if pushed you'd mention what makes a user not be able to log in. You're obviously familiar with the principles so as long as people aren't pedantic you've demonstrated sufficient knowledge.

So reply with an answer like "I can't remember the specific flags, but I'd look in the man page for them. I automated that years ago and can't recall whether it's adduser or useradd". That kind of answer shows that you still know what the problem domain is without knowing the exact details to answer, and how to find the solution.

I don't remember the flag, but this could be my answer:

Most linux distributions, come with /bin/false as a valid and registered shell, for login to the system (but it's a program that returns false and does not open a shell).

I know that useradd and adduser allow a flag for change this at creation time (probably --shell /bin/false).

It could be more safe to be explicit since the creation time, than using chsh (changeshell) to avoid race conditions in security, having a user created with a shell until chsh is run.

You see, I didn't remember the flag, but I did try. That's the kind of answer that maybe some interviews are waiting for.

/sbin/nologin actually does some cool logging stuff that /bin/false doesn't, which is why the distros generally use it.

Cool stuff as in display a message that the account cannot login, which leads to brute-force service, process or other stuff system user names.

Failed logins will be logged either way.

I think still could go with the output of `type -P false`. Unless the project requests that feature of messaging to the (forbidden) user, its forbidden logins.

It could work even on embedded devices having only 'false' available.

Oh my.

Nothing about knowing if the sysadmin knows how to find information (man, docs, info, apropos, perldoc) offline and online (advanced bash scripting guide), which shell to use to be the most portable, and where to go to be up to date regarding vulnerabilities.

That's a good pool of questions to choose from. How long do you think would not take an applicant to fill all of the questions?

My questionnaire for the same purpose (hiring) has 30 similar questions, yet including some that query soft skills.

E.g.: "you got the following email from a customer, please write down your answer to her inquiry".

Or: "the CFO, the CEO and the CTO are approaching you with the following different requests, how do you react to them and in which order would you process them".

There's some decent questions here, but also too many "how to" questions that most people won't bother memorizing since they take 2 seconds to google.

The key thing I found in interviews (from the interviewer side) is that getting someone to sit down and write the test is not very useful.

Its far better to get them to talk you through it. It allows you to asses their knowledge (which can be taught) but more importantly it allows you to see how they work and if they can explain their shortcomings.

I want someone to say "I know that x does y, but what happens when z comes along I'm not sure" I can then provide a nugget of more information and see if they are able to grab onto it.

It shows you their ability to communicate, and will bring out more of their latent personality. You're going to spend more time with your co-worker than your significant other, so you need to have some idea that you won't hate them.

Its far more important to have someone who knows when they are out of their depth and handles it gracefully than someone who seems to have more knowledge but hacks about wildly when they fall off the deep end.

This is a horrible interview question for all but a tiny handful of jobs:

  Is 300.168.0.123 a valid IPv4 address?

...but it is a great late night fun trivia question about "how can adding one character to that address make it into an address accepted by many tools".

                                                                                                                    Stick a zero in front?  0300 octal == 192

Edit: Fixed so it's not such an obvious spoiler.

"Unpack test.tar.gz without man pages or google"

Ha, guess I'm not the only one :)

Relevant xkcd : https://xkcd.com/1168/

Remember to 'eXtract Ze Files', twirling your moustache while you say that.

Ist just remembered only "xvzf" 20 years ago, not knowing the meaning of each switch back then.

You can actually drop the Z these days in a lot of cases; tar will detect the compression algorithm.

I occasionally log into older GNU, or non-OSX BSD systems where tar can't detect the compression automatically. [1]

[1] this isn't true, I just enjoy saying 'extract Ze Files' in a silly accent

    tar --help   # they didn't rule this out :)

> Write a get_prim method in python/perl/bash/pseudo.

What does get_prim do?

Returns prim

Isn't the biggest skill of a sysadmin to solve problems that he has not seen before? Learning all those answers to the questions wouldn't help you much in a real life scenario.

Instead of asking "what is/how do you do this?" it is better to give example scenarios and listen to what someone would do, to see how they approach the problem.

Being a hobby linux sysadmin myself I'd say you're right. The hard and often interesting part is to figure out why everything suddenly stopped working. sometimes I see the same problem several times and try to figure out how to stop it from happening again, most of the time though I see completely new problems. But it always helps to know your Linux system and how to hunt down error messages and log files. Buy then I'm no professional.

How can I use this as a self-placement test? I'm in college still, and I have answers to many of the general, simple Linux, and devops questions. What is expected to be known for varying levels of employment? (Intern/junior/normal/senior)

I am right there with you...

I was a sysadmin, a devops engineer, and a build/release engineer. Now a developer.

These questions are all decent. There's nothing here that would be out of the question to ask in a junior interview.

Usually for junior interviews, they want to see if you understand the fundamentals and not advanced details that a senior might need to know.

One example of this is the question "What is swap and what is it used for?"

For an intern, I'd ask if they knew what it was for. For a junior, I'd add questions about any ways to control swap. For a midlevel/normal, I'd add scenario questions about configuring swap and the performance effects it can have. For a senior, I'd add troubleshooting questions.

To be honest though, I've only ever been asked the base question in interviews. I'm guessing that my answer sounded confident enough that the interviewer determined that it would be more worthwhile to explore areas I was not so strong in to see how I problem solve/deal with foreign scenarios.

Most decent interviews understand that people have different levels of understanding for different topics which might fall under the umbrella of "DevOps". A candidate could be a docker wizard, knowing all of the best practices of versioning, CI, and deployment, but could not know much about VLANs because "they never had to use it".

Keep in mind that most companies use a certain tech stack and are usually looking for people familiar with it. Interviewers always seem to be impressed when I told them I "checked it out over the weekend" and was able to answer questions decently about their stack. Skimming the documentation, setting up a vm to play around with, and finding the most popular blog posts about current best practices for a technology can go a long way towards closing the gap between your current knowledge and the ideal candidate's knowledge.

Hope that helps.

Here are a couple I use:

- How many hosts are there in a /23 network?

- Estimate ping time from your workstation to your local gateway? www.google.com? Server on the east coast from the west coast? Server in Europe from the west coast?

- What law(s) of physics control(s) the round trip times?

"How many hosts are there in a /23 network?"

Having interviewed a lot of sysadmins, I see this trip people up, mostly because the slash/bits network notation isn't taught/documented well. The "memorize a table" approach seems to be common.

Most of them, with a bit of prompting, get that an ipv4 is 32 bits, and that a /23 means (32-23==9) 9 bits of host addresses. Most also can figure out what 111111111 is. Some remember to account for the network and broadcast address not being available for hosts.

> How many hosts are there in a /23 network?

I've recently learned that this is a bit of a trick question. How many routers are in embedded the /23? How many addresses are reserved for multicast? Are you on AWS VPC, where they reserve even a few more addresses for internal management?

All of those are good points. But a surprising number of people can't get to the starting point 2^(32-23)=512. I usually subtract 2 and say 510. No IPs are reserved for multicast. Maybe you meant broadcast?

The "simple questions" really reads to me like a proof of the superiority of the gui over terminal. With "how to delete a directory and its content" winning the award.

It could almost read like a wishlist for linux improvments.

Change it to "how to delete the contents of a directory, but keep the directory itself" and it becomes less simple.

But even this can lead to an interesting discussion. From a CLI, there's quite a few things to consider. Do you need sudo? If not, you shouldn't use it, IMO. Is the directory name dangerous? If so, would rm -rf -- help? Does rm support -- as an argument at all? I wouldn't know. And would it always prevent problems, even with truly weird directory names (with spaces, asterisks, slashes, $HOME, unbalanced quotes)? Of course you shell should tab-complete names and correctly escape them, but should you check your shell (brand and version) first if the name is weird, and know what versions have a known problem?

You've just let on that you don't write scripts often. Sysadmins and devops write 'glue'. All sorts of stuff that seems stupid if a human is in direct control becomes quite useful in a script.

For example, what purpose would you have for telling grep not to output the thing you searched for? Not useful for a human operator observing the results, but used all the time in scripts. "if grep -q foo /path/to/target;"...

So, if you're after a person who is going to be writing 'glue', being familiar with how glue 'sticks' is a good thing to look out for.

Are you not a developer? Do you know how to remove a directory and all its contents in, say, Python?

Does it involve a GUI for you?

Does anyone know where to find a similar list for Microsoft Sysadmins?

"How do you enable DHCP or $feature on Windows Server 2008?"

"Click the "Add Role Wizard" and then click next until you're done."

Q: "Ah but how would you enable it in Windows Server 2012/2016 Core editions?" / A: "PowerShell and Google, not necessarily in that order"

(Of course, Bash and Google would be the equivalent in *nix-land, so, maybe not that great an answer...)

Taking it more seriously, well, I'm not a sysadmin, I'm more of a developer (usually on Mac) with a passing interest in Windows servers. But here's my list: at least one question could relate to knowing any advanced Windows terms they might come across in a sysadmin context, like VHDs, different types of installers/drivers, differences between client, server, core, nano editions of Windows. Bonus points if they know of Git, Nuget or Chocolatey? Oh, and they're good in my books if they know about Microsoft Virtual Academy and Channel 9, especially the early Scoble days ;-) I do like the direction Microsoft's been going lately...

Q: "Ah but how would you enable it in Windows Server 2012/2016 Core editions?"

A: Most likely, add it to the Server Manager in the RSAT machine, then use the add roles wizard and click next till done.

Not seen it, but you could start with the same topics - assuming a powershell focus:

- process management (get-process, stop-process)

- hardware (Get-WmiObject)

- networking (Test-Connection, Resolve-DnsName, Get-NetIPAddress)

- finding / changing registry entries etc.

And a bunch of Windows specific stuff like AD accounts vs local accounts, group policy, etc.

Ex Amazon syseng here. Please be advised that the questions are really basic, even the "hard" ones and won't get you very far in an interview in various companies.

Thanks for the effort.

These type of questions are OK for a filter phone interview or entry-level Linux job imho as in they rely on "what is" questions (which can be memorized or replied without much experience) rather than "how would you" questions (the "demo time" at the end, that's the good part). You want to ask scenario and troubleshooting questions rather than trivia for jobs other than basic ones. (You can certainly ask more advanced non-scenario -still relatable or practical- questions about things like system calls, interrupts, CPU scheduling etc, "system internals" stuff that I don't see at all or barely in these questions. A Linux questionnaire without mentioning strace or perf hurts my heart.)

I can basically nit-pick, find a fault or find a better way of asking in pretty much every question.

The Expert section can be dropped (only one question).

Mysql section can be dropped, maybe replace with basic SQL section.

DevOps section can be dropped, we can see here how definition of DevOps is still fluid, just add Linux/Networking/General questions to their respective sections (also "GIT", really?).

One reference link at the bottom doesn't work.

----

Examples of issues with questions and how to improve them:

- What commands do you know that can be used to check DNS records?: instead ask how to troubleshoot DNS issues, like how to check if a name server has an entry. How does local DNS work? (as in what's the order in which files/servers are checked to resolve a name, where is that stored) how does DNS in general works? (as in you ask for a record to a name server and it doesn't have it, now what). How much TTL would you set in a record and why. Why does DNS run on UDP and TCP?

- "tac", (really? and no "grep" "sort", "uniq" or "sum"?), rather ask how to print a field from a command like 'df' or 'free' or how to add column values from a text file (basically if they can slice horizontally with grep and vertically with awk / cut and then count occurrences or sum numbers.

- What does & disown after a command do?: rather ask when you would use 'disown', or better, inversely how to de-attach a running command from terminal

- What is Virtual Memory? : not sure what you want them to answer here, maybe rather ask about a server with most of its total memory used, what does Linux cache (there's buffer and inode/dir caching besides files but only two memory caching areas), how to find that, when does the OS write stuff in RAM to disk (what are dirty pages), how do you change that frequency, when is better to do it more/less frequently or what's the tradeoff etc

- What is swap and what is it used for?: this is more advanced but you can ask if a server can have no swap, is it possible for a server with plenty of free RAM to page (write to swap)? can you reduce/increment tendency to swap? why would you do that, how much swap partition would you set up (OK these are a bit of tricky questions, more to gauge experience)

- How to search for the string "my konfi is the best" in files of a directory recursively? : I actually like this one

- How to connect to a remote server or what is SSH? : you can split in "list all the possible ways you can think of so you can communicate (or send message/file) between two servers on a network" and a question about how SSH works from Linux point of view (not protocol), sshd settings (idea, not memorized params), how to set up passwordless (key-based) SSH, hardening tips, how to troubleshoot if you can't ssh to a server (no firewall, you can telnet/whatever), what does it mean when you get a message "key has changed etc", what's a private and public key...

- What is the sticky bit? : this depends on OS, I guess explaining different behaviours is good.

- What does the immutable bit do to a file?: this is not part of linux VFS but of particular filesystem implementations, btw no questions about what's a filesystem, what's a journaled one, difference between ext4 and xfs for ex. Also what about LVM?

- What is the difference between hardlinks and symlinks?: while a good question I'd rather ask when to use one or the other, also just to confuse, ask how to "link/map" a directory in a partition that is running out of space to a directory in a new partition with plenty of space, so that it works transparently.

- What is MAJOR and MINOR numbers of special files?: I guess device files, rather or besides ask difference between block and character devices, about pseudo dirs in file system: /dev /proc /sys , how to find hardware or process info (everything is pretty much in /proc or /sys , also dmesg )

- Describe a scenario when deleting a file, but 'df' not showing the space being freed. : this is good but it can be better built into a scenario: there's an "no space left on device" error, how to find biggest files/dirs. You find it's a huge web server access.log file, now you delete it and disk space doesn't seem to shrink, why (the df vs du), what can you do?, can you close the open file?

- Instead of "how does ps/top work?" ask something specific or scenario based, like how do you find if there's a disk I/O problem by looking at 'top' (there's at least two possible indicators). Zero questions about I/O, or for the matter, for server utilization metrics, which ones are they, how to measure each, how to find if they reached saturation, what happens in that case for each, how to look for errors in each. If a server is using 80% CPU is that good or bad?

OK I got tired here.

If you have read until this dear reader, would you be interested in my trying to create a Docker container or Amazon image with a Linux box and have these types of scenarios set up?

Consider me extremely interested.

oh no, now I have to put my time where my mouth is :-)

> Describe a scenario when deleting a file, but 'df' not showing the space being freed.

I came across this, seems to occur when disk is almost full. Is there any probable reason why this happens?

if the deleted file is open 'df' doesn't show cleared space, 'du' does

Any additional questions anyone can think of?

One ultimate question: "describe a project of yours, current or past, personal or or job-related, that you are most proud of". You don't need to ask a professional anything else if he's anywhere near worth his salt; he'll volunteer all the information you could judge him by.

Having that said, most of the questions from "general questions" part come either from assuming that the candidate is dumb and possibly doesn't understand how computers work or from some lousy layman psychology.

Many of the questions further down are incorrectly assigned. "What is an A record?" is not a Linux-related question, and git is hardly a "devops tool".

Very poor list, in my opinion. In an interview you need to assess whether the candidate can do the job. Since the only heuristics available to the interviewer is how the candidate did in the past, the interviewer needs to allow candidate to show what he has done, how, and why.

And just for fun:

> A careless sysadmin executes the following command: chmod 444 /bin/chmod - what do you do to fix this?

My favourite way of fixing it is by running /bin/chmod. It blows minds of most smart-asses who ask this question.

Most distros symlink /bin to /usr/bin. I think the question is asking you to use /lib/ld.so, but it's a bit of a weird question to ask.

> Most distros symlink /bin to /usr/bin.

Uhm, no? I've only seen it in recent CentOS (older versions didn't have it), and maybe Fedora does this as well.

> I think the question is asking you to use /lib/ld.so, but it's a bit of a weird question to ask.

Not really. You can do it in any number of ways, including using Perl/Python/Ruby, writing a short C program, or overwriting another file with execution permissions. My favourite way is a surprise to most people only because they don't know how ELF binaries work.

> Most distros symlink /bin to /usr/bin. > > Uhm, no? I've only seen it in recent CentOS (older versions didn't have it), and maybe Fedora does this as well.

Arch, Ubuntu, Debian all do it.

> Not really. You can do it in any number of ways, including using Perl/Python/Ruby, writing a short C program, or overwriting another file with execution permissions.

Only the last one is a clever solution. "Just write a program" isn't a fun answer. :P

> My favourite way is a surprise to most people only because they don't know how ELF binaries work.

What do you mean? They didn't know there's a copy of the same binary?

>>> Most distros symlink /bin to /usr/bin.

>> Uhm, no? I've only seen it in recent CentOS (older versions didn't have it), and maybe Fedora does this as well.

> Arch, Ubuntu, Debian all do it.

I haven't checked Arch, but neither Ubuntu 16.04 nor Debian Jessie haven't created /bin as a symlink.

>> You can do it [restoring execution bit on /bin/chmod] in any number of ways [...]

> Only the last one is a clever solution

On interview you don't need to have the most clever answer. It's better to show you understand the mechanisms underneath by having several sensible ways.

> What do you mean? They didn't know there's a copy of the same binary?

No, I meant this: https://news.ycombinator.com/item?id=11627670

And there is no copy of the same binary.

> I haven't checked Arch, but neither Ubuntu 16.04 nor Debian Jessie haven't created /bin as a symlink.

Odd, I could've sworn they did. I stand corrected.

> > What do you mean? They didn't know there's a copy of the same binary?

> No, I meant this: https://news.ycombinator.com/item?id=11627670

> And there is no copy of the same binary.

That's what I said, you use the ELF interpreter. :P

So... when the first third of the questions in the 'general questions' section is asking about what the applicant has done, how, and why, it's "lousy layman psychology", but when you ask for the same thing, it's "the only way to do it"?

> > A careless sysadmin executes the following command: chmod 444 /bin/chmod - what do you do to fix this?

> My favourite way of fixing it is by running /bin/chmod. It blows minds of most smart-asses who ask this question.

----

Not sure what I'm missing here?

    $ docker run -it ubuntu
    # chmod 444 /bin/chmod
    # /bin/chmod
    bash: /bin/chmod: Permission denied

ELF interpreter/loader.

  objdump -s -j .interp /bin/chmod
  /lib64/ld-linux-x86-64.so.2 /bin/chmod

I only see one reference to /proc in here, which is kinda lame. As a sysadmin who's frustrated with some inflexibilties of lots of commands to the point of iterating through it fairly regularly (and know others like that, too), it'd be a great/necessary add.

Ask about 'ip' and why they might want to use it. Distros themselves don't use ifconfig any more and ifconfig can choke on more complex (virtual IPs, on vlans, bonded) setups.

How to get the serial number of a VM or physical box (dmidecode for Intel, bonus points for /proc for powerpc)

Reply on news.ycombinator.com