Comment by nostrademons | Hacker News Reader

Comment by nostrademons | Hacker News Reader

Parent Full thread

Google tries to keep interview questions confidential - that's why candidates sign an NDA - and periodically rotates out questions that have appeared in public. Many engineers are also continually trying to think up new questions as well, usually based on their work.

For most questions, there's no "right" answer, but there are a set of points that the interviewer wants to see you touch on. For example, they might first want to see that you can code up a naive brute-force variant of the algorithm, checking whether you know the programming language claimed and can think through the problem, and ask you the algorithmic complexity. Then they'll want to see if you can get a divide-and-conquer or dynamic programming variant with lower time complexity. Then they might ask "What if it has to be an online algorithm, where new input arrives before the computation finishes?" Then they'll ask "How would you distribute this over 1000 machines, and what are the failure modes?"

At each stage, they're watching how you answer, and where you get stuck. If you ask clarifying questions or spend time to think before diving into coding, that's a plus. If you have never heard of the problem before (this is frequent - many questions are not in textbooks), they want to see how you would reason through it, and break it down into subproblems that are similar to textbook problems. If you miss language trivia, most people don't care; when I did interviews I'd usually volunteer the answer if they missed some API call, and when I interviewed my interviewers did the same. If you don't know how to solve the problem and can't make any effort to move forward through a solution, that's a big negative. Similarly if you don't know what the concept of big-O is or why it's important.

Maybe, but that doesn't sound like my interview at Google, which was interesting but really rather random. Perhaps most notably, one interviewer became irate that I didn't know the "ps -o" flags off of the top of my head. :-/

According to `man ps` on my Mac, there are nearly seventy of them - how many did he want you to know?!

It's been a long time since I interviewed an ops person. But when I did something like this, I'd ask, "Name as many arguments to ps as you can." (Or ls, also good.) I didn't really care about which flags they knew. What I was looking for was a pattern. Anybody good knows some arguments cold. Often they know them so well as part of a phrase that they have to think about what the individual ones mean, which tells me that they've done something enough that it's become a habit. Then they'll know a few others that they use occasionally. And then they'll stretch to name a few more that are obscure for them.

But the real magic comes when they run out. What do they do then? Good people say that they don't know. Often, they will take guesses at a few more based on general principles. E.g., "I'm sure there are arguments for sorting, but I only use top for that." Or, "There must be more ways to filter, but I only do it by user or command." They'll usually look a bit uncomfortable, because they are the sort of person who likes knowing things. But what they won't do is bluff or make things up.

And honestly, that's my biggest tip for hiring: find people who like understanding things and know a good amount for their level, but are willing to say when they don't know. People who can't do that are dangerous.

This is a stupid trivia question. I use ps -ef and leave it at that. In my entire career I've rarely needed anything else and if I did I would look it up but never enough to remember. To base an interview question off of that speaks volumes about you.

I agree, and it's interesting that you know -e, while I've memorized -axuf ; I guess it's more portable?

Again, this is one of those, "Here's what I've got cached, and here's where I'd look up what I don't know offhand" questions.

No, "-axuf" is a weird mixture of BSD and POSIX syntax.

From the ps manpage:

Note that "ps -aux" is distinct from "ps aux". The POSIX and UNIX standards require that "ps -aux" print all processes owned by a user named "x", as well as printing all processes that would be selected by the -a option. If the user named "x" does not exist, this ps may interpret the command as "ps aux" instead and print a warning. This behavior is intended to aid in transitioning old scripts and habits. It is fragile, subject to change, and thus should not be relied upon.

Well, yes and no. If somebody just scored it as written, giving points for each argument known, it would be a dumb trivia question. But as I said, that's not what I do.

Try it next time you're mingling with ops people. Say in a bar at a conference. Don't look at what they answer. Look at how they answer. Some people are comfortable not knowing. Some people are excited to discover the ones they forgot they knew. some people get curious and eager to fill the hole they've just noticed in their knowledge. And some people get defensive and peevish.

Some people focus their curiosity on more important things than arguments for a command line utility.

I feel pretty comfortable with my Unix abilities, and have held ops roles in the past, and I would have bombed this question.

I doubt that. The only way to bomb it is to make shit up, and I don't think you're the sort to do that.

I don't score it by how many option-letters people know. I score it by how well they demonstrate familiarity with a basic ops activity, which is finding out what's running.

Even if somebody said they never used ps, or like below, only use one incantation, that isn't a problem, because my follow-up there would be to ask how you'd find out what's using all the RAM or what's using a lot of CPU.

Maybe they're just used to using different tools, in which case I can ask for details for those tools. (And then check manually later to make sure I'm not getting snowed.) But if they've done a bunch of ops stuff, they'll demonstrate the sort of know-from-doing level of familiarity with something.

In theory I could just ask the broader question to start. But some people are good talkers, and I'm looking for evidence that people are good doers. All that said, I've shifted mainly to more experiential interview processes, as actually doing is the best way to see if somebody's a doer.

They'll usually look a bit uncomfortable, because they are the sort of person who likes knowing things.

So the process is, in fact, designed to make people feel uncomfortable, for a little bit (or at least you're definitely aware that this is a frequently occurring side effect).

Does that not suggest to you that there may be a negative tradeoff at play here? (To wit: yes you do manage do efficiently extract a few bits of information from them... but at cost of having them feel like they're being, well... interrogated. And are already "losing points" by that point in the conversation with you).

I think part of the point may be to select for people who don't consider "being interrogated" to be a contest. There are plenty of folks who are just like "Here's what I know, here's what I don't know, you decide whether you would want me on your team. If you don't, that's fine, I'll find another team to join." Then there are other folks who, when they're asked a question and don't know the answer, feel like their self-worth is under attack. The latter group can be really hard to work with, and can do a lot of damage to a team.

I had a PM friend who would try to push the bounds of a candidate's knowledge until they gave up and said "I don't know". The actual knowledge wasn't the point of the question: it was that they could say "I don't know", because PMs who can't tend to make life miserable for the engineers & other teammates who work with them.

Yes, exactly. I think to be a good developer, you have to be willing to keep learning. And to be a good developer in a team context, you have to be willing to learn from your teammates, ask for help, and share knowledge in a way that's supportive, not prideful.

And I agree with your PM friend; I think willingness to admit ignorance is even more valuable there.

If somebody goes into an interview expecting never to be asked a question they might struggle with, I'm not sure they're somebody I want to hire.

But no, I don't think they walk away feeling interrogated. My style is pretty conversational, and when people don't have all the answers, I definitely work to make them feel comfortable with that. E.g., for this question I'd close with something like, "Of course nobody knows all the arguments to ls. I sure don't. So you did fine."

dangerous because they prefix things with sudo?

I imagine there's a correlation there. But for me it's the danger that instead of admitting when they don't know they'll just bluff, going off and building things that look good but aren't. If somebody knows their limits and can admit them, you have to supervise them much less closely than a bluffer.

I would almost assuredly stop the interview at this point and try to figure out what your malfunction is.

To me any question where the obvious answer is "I can trivially look all that up in the man pages" is a gigantic red flag that I don't want to work at that place.

If I'm feeling curious that day, I might begin a conversation about what you think you are testing for with that question, as I've spent a fair bit of my career studying hiring pipelines. But if I hadn't had my coffee yet or were irritated or something I'd probably just ask to talk to someone else.

That's fine. If you go into something assuming that anything you don't understand is a malfunction, and you excuse your rude behavior with your mood and/or caffeine level, you're probably not the kind of person I'd want to spend a lot of time working with.

That said, as I've explained elsewhere in the thread, the point isn't to see how many they know. It's to see that a) they know some portion of it that people doing the work would know, and b) have reactions to the rest that indicate useful work habits and attitudes.

A perfectly find first answer to is "I only use ps -aux. I'd just look in the man page if I needed anything else." Because then we could have a good discussion of how they use that output to figure out things, what sort of things they'd expect to see in the man page, and what other tools they use to see what's going on.

I'm pretty confident both of you are people I'd be happy to work with in the future and so I want to use this as an opportunity to point out that your reaction to what Kasey said his reaction would have been to your interview question practically guarantees that you would fail to acquire Kasey for your team. I know Kasey a little bit better than I know you (he's a Chicago person) so I'll just sum this up as "that's not a point in the win column for this interviewing strategy".

I know you're not asking people recite "ps" flags from memory. The problem is, you're looking for a subjective "X-factor" in how someone answers a trivia question.

I used to really like this approach too. I'd think of things that you'd only know if you'd actually done the kind of work I'd done. I had some favorite interview questions: "what are some of the functions you carry around with you from project to project in your 'libyou.a' library", and "where would you start looking if you had to debug a program that segfaulted in malloc"? I think the logic I was using is the same as the logic you're using here.

Ultimately, I think it's a bad approach. Equivalently strong candidates look at their work through different lenses, and find different things memorable. Far more important to me, though, is that some people really suck at interviewing --- and, what motivates me more, having succumbed to this repeatedly as a hiring manager and also being myself I think an example of the phenomenon --- some people just interview way better than they should.

The issue with this question falls into 2 main categories. The first is that its a rude question. The basic premise seems to be, "if you are an experienced ops person you have to know something about ps". Which may be true, but by asking it of an experienced operator you are hinting that you don't believe their resume. There is a much more straight forward way to see if they are lying on their resume, call their references and prior employers. For really good operations folks, the answer to this question might be "I haven't needed ps in so long I've forgotten how it works. I've automated away the process running abstraction and have a library of python/bash/salt/chef/esoteric top commands/etc that I find more useful than remembering ps flags". This question implies their experience in that regard is also suspect.

The other problem with this question is that, by its phrasing, it implies that there is a right answer. If I can name more ps flags than an other person, I'm a better candidate. Which when put that way, I think points out its faults. Maybe you don't intend for that to be true, but you'd have to fight your own cognitive biases pretty hard to not at least bias towards the person that can name 17 flags off the top of their head, or the one that teaches you a new flag combo you didn't know. Even though those things are likely not that good of predictors of a good candidate.

You've said what you really want to get to, is can they admit when they don't know something. If that is important to your hiring process (and I'd encourage you to validate that with data), ask them a question no one could know the answer to. Ask the same question to every candidate, and grade it purely pass fail. Did they say "I don't know" or not.

  >> admit when they don't know something... encourage you to validate that with data

Correlating this with employee performance data might not be the only metric.

My experience is that people that won't admit they don't know something are not necessarily bad employees. Often they are capable employees, although this is a trait I rarely see in the best employees. However, I find they are toxic to a good work environment, since they won't listen to people who do know something.

They do seem to get promoted to management had a much higher rate :)

I'd probably be snarky and say "I don't know - but I can google it!"

/actually I'd probably hit up the man page first...

He probably had at least five or ten in mind. I prefer other methods of extracting this information. Usually 'ps -efly' piped into tools like awk/etc to gather what I want, or on occasion, extracting directly from the /proc filesystem. He did not consider these to be proper alternatives.

Since the 'ps -o' flags are only useful in that one command, I just look them up in the (rare) situation where they seem the best alternative. Having been at this for decades, I try to reserve my limited memory capacity for factoids that will pay off, and being able to process columnar data generically at an instant seems more useful than knowing a special way to do it with that one command.

If the stuff online on Quora etc. are to be believed, the hiring committee will disregard such trivia interviews (and possibly send a node to the interviewer asking him/her to correct their technique).

Not sure how well this works in practice though.

Last time I was job-hunting I experimented with Google to see if their interviews were as bad as everyone said.

They were. I was asked a ton of this kind of Linux/Unix trivia in one session. As well as just random stuff. For example:

https://twitter.com/ubernostrum/status/659182356171874304

https://twitter.com/ubernostrum/status/659182564309995520

https://twitter.com/ubernostrum/status/659182708996706308

Another way to answer this kind of question is to know 2^24 = 2^10 * 2^10 * 2^4 and that 2^10 is 1024 which is roughly 1000.

So 1000 * 1000 * 16 = 16 million would be an easy estimate to make.

If you are designing a system and want to have an idea of how much space or memory an approach will take, being good at this kind of math can be really helpful.

Then they should probably ask people to estimate a thing that's even remotely relevant to the job.

(I mean, I get it, this kind of general estimation and/or ball-parking can be incredibly valuable, but this doesn't sound relevant in any way.)

This kind of general estimation and/or ball-parking can be incredibly valuable, but this doesn't sound relevant in any way.

To be fair to Google (in regard to this one filter question only), they actually are up front about expecting a significantly higher baseline of general mathematical awareness (even for non-mathy roles) than most other shops. At any rate, this kind of estimation comes up in capacity planning (both in architecture planning, and for algorithms on a single box) all the time. Which after all is what Google does, at nearly every level of their stack, (nearly) all the time.

It "comes up all the time"... and these ballpark guesses are probably(!) completely useless for anything remotely practical. At the scale Google's operating you want a Statistician, not a Guesstimator.

So, no, still not useful questions.

The problem is that there are some people who will need to go through that process in their head to produce an estimate of 16 million - and then there are people who have had cause at some point in their life to know of the existence of the concept of '24 bit color' and that the number of colors expressible in 24 bits (8 bits each of R, G and B) is exactly 16,777,216.

The question there was something like "roughly how big is 2^24?". I actually think that's a reasonable enough question, but anyway there's a nice story about this same question in another context. The mathematician Solomon Golomb was (as an undergraduate, I think) taking some sort of biology class, and the lecturer described some process to do with cell division and said "... so the number of ways to do that is 2^24, and we all know that that is, don't we?" (meaning, of course, "and no one knows what that is but it's OK because I'm about to tell you". But Golomb happened to have been memorizing numbers of the form n^n, and 2^24 = 8^8, so he immediately called out "Yes, it's 16777216".

As a result of having read this story, I can now always instantly remember what 2^24 is too :-).

What about when you know what the concept of big O is, know why it's often touted as important, and disagree that it's as important as it's touted?

My opinion is that big-O often ends up being used as a premature optimization effort hindering "just get the right answer first". Maybe that brute force method will work in a reasonable clock-time cost, despite having egregious algorithmic cost. You won't know if you're busy overengineering and optimizing before you even have a working solution.

That's a big red flag, at least in a Google setting.

The reason is because when your dataset is in the petabytes, any algorithm bigger than O(N log N) is not going to terminate. You're not going to get any answer at all; your MapReduce is going to sit there burning CPU time for a day, and then you'll kill it, and you won't have any idea what went wrong. (This is learned from experience, if you couldn't tell.)

In a startup context, I'd certainly agree with you - that's exactly the approach I'm taking with my startup, where I'm doing a lot of the debugging and code-wrangling on my local laptop after verifying that the data I want exists and calculating its size with a couple cloud-computing passes. It can sometimes be useful to prototype & iterate with a small subset of the data until you've got the algorithms down.

But many of the algorithms Google cares about don't give any useful results on small data sets. I remember running a few collaborative-filtering algorithms and finding that they gave zero answers because within my small sample of 0.01% of the total data, nobody liked anything else that anybody else did.

In my experience as a Google engineer, this is mostly false. None of the work I or any of the people I know well at Google is on petabyte datasets; most people just aren't on the hot path of indexing/websearch/youtube. In fact almost none of the stuff I was asked about in my interview has been relevant to my work. I have thought about complex algorithms two or three times in the past year.

Not only that, but (in my experience as a Google engineer), factors relating to the location of data (distribution and disk vs ram, etc.) often greatly outway algorithmic performance. Constant factors often dominate variable factors.

Exactamundo! Cache/memory/disk locality dominates almost all other factors[1] unless you're doing something very special. (Speaking as a non-Google person who also likes to think that he knows what he's talking about.)

[1] Well, it's really the amazing amount of throughput that modern architectures can achieve. Latency hasn't improved quite as much given the inescapable limitation of the speed of light.

What department? I was in Search and used that knowledge all the time, and also did rotations on GWS, Google+, and GFiber. Certainly I used less of that knowledge for GWS (where everything you need is in-process) and for GFiber (where I was working on marketing & customer signup - the job req for this was actually different, and both interviews & job duties were much more like a traditional SWE job in other companies). But even Google+ needed a lot of algorithmic complexity knowledge; G+'s scaling needs are similar to Twitter's, and we know how that went.

Perhaps it has changed as well; when I left (2014) they had just started encouraging engineers to focus on one particular task, with the architecture already defined, while when I started (2009) there were still a number of problems of the form "here's a feature we want to add; here's the data we have available; how can we build it?"

I'm on Ads and Commerce front-end, and I also know a number of people on Android. These of course are mostly constrained by users' devices, so don't have the scale your talking about. But I also think the trend you're talking about of smaller roles has been continuing and has a lot to do with it as well.

Everyone loves to put Big-O up on a pedestal, however many times the constant costs can outweigh the rest of the algorithm. Cache access times in particular can be brutal for non-linear access patterns.

> What about when you know what the concept of big O is, know why it's often touted as important, and disagree that it's as important as it's touted?

What if you're having a conversation with someone about local politics and you're convinced the local zoning rules don't encourage growth in the way they think they do. Do you force the subject of conversation so you can make sure they know just how wrong they are? Not if you want them to walk away having a good impression of you.

Instead if the talk turns to zoning you put out feelers to see if they'll want to talk about the larger issue, and only engage in the conversation if it seems like the time for it.

The analogy isn't perfect, but if the interviewer wants to talk about the larger issue, you're probably well matched and will have a great interview. If not, and you know enough about runtime complexity to discuss the tradeoffs, then you know enough to just answer the textbook big-O question and move on.

The questions about BigO isn't about premature optimization or actual implementation. It's about understanding exactly what a particular program is doing and how it's doing it.

Any sort of technical solution given at a Google interview would generally have the following questions:

1) What big O time does this algorithm run in? Why? What big O space requirements does it take.

2) If space were more/less expensive, or time more/less important, how would you change the solution and why?

Understanding those tradeoffs and being able to analyze code at that level is a big part of most software engineering jobs.

There are situations in which you want to just get a right answer and ship, like at an early-stage startup.

Then there are situations where the code you write must play nicely within a massive, already-complex system, wherein it will work with potentially huge inputs. Like at Google.

So, saying in a Google interview that you don't think Big Oh is super important, and that you prefer shipping whatever correct solution you come up with first and then worrying about efficiency if it ends up being slow, would likely not get you very far.

> What about when you know what the concept of big O is, know why it's often touted as important, and disagree that it's as important as it's touted?

Let's say you created your own company and you're interviewing engineers.

Would you be comfortable hiring someone who doesn't know the difference between O(n^2) and O(2^n)?

Or someone who doesn't even know what these concepts represent?

I've always called this kind of thing "trying to get the sizzle before the steak". You gotta have a steak first; get that, then work on cooking it - then work on cooking it to perfection if that is what is needed or wanted.

There's also the thing I picked up from playing around with graphics demo coding; get the algorithm working first, even if it takes 10 seconds per frame. Then look for the slow parts, concentrating first on the inner loops. Whatever you do, don't try to prematurely optimize the code as you write it, as tempting as it may be. Because almost certainly you'll make things worse than if you waited until after you have a working first pass.

This applies to way more than just fast graphics code, of course.

Reply on news.ycombinator.com