Show HN: 15-question programming quiz with answers | Hacker News Reader

Show HN: 15-question programming quiz with answers

quiz.triplebyte.com

227 pointsammon228 comments

This is very python specific. While I got a 15/15 and can easily answer these, the reason for that lies with the fact that I am very fluent in Python, had I not been I would have not gotten at least 6-8 of these wrong.

This isn't a generalist quiz, nor is it a good baseline for interviewing candidates for the hiring process, this wouldn't tell me anything about the application of their skillset.

I took a quiz at the Facebook booth at PyCon this past weekend. It was Python specific, but had a few annoying gotchas. There was one question in there twice, but with 2 different values for the input (which is what tipped me off that it was a trick).

It basically relied on this fact...

    x = 256
    x is 256  # True

    y = 257
    y is 257  # False

Which for one, you shouldn't use `is` to compare integers in Python. I wish I could have just written that as my answer.

Secondly, you need to be aware the CPython uses an integer cache which causes the answers to be different. You'll actually get different answers based on whether you run it as a file, enter it line-by-line into the interpreter, or use a different implementation such as PyPy.

https://stackoverflow.com/questions/15171695/whats-with-the-...

There wasn't any point to the quiz, so it's okay that they put in silly questions like that. But it's clearly there just to try and trick you, and not quiz you on anything relevant. Most of the other questions were tricks of the sort, or or Python 3 specific gotcha's.

That's kind of nasty. It's certainly fun to play with pathological language features, but it's not exactly a useful thing to test on when the answer is "don't do that". I'd enjoy a JavaScript quiz on some of the "wat?" features, but I wouldn't want to be evaluated on my ability to guess them.

Seriously! I think these sort of tricks are really fun (I read IOCCC and other stuff), but I only know this because of a famous codegolf answer that takes advantage of this trick [0].

I wouldn't expect anyone else to know it, and don't consider it something positive or negative as far as programming skill is concerned. It's kind of related to interest, but admittedly I waste a lot of time playing with stupid details like this in programming languages, so maybe it actually makes me less productive than the average person.

[0] "Write a program that makes 2 + 2 = 5"

https://codegolf.stackexchange.com/a/28851

edit: wording change

> There wasn't any point to the quiz

Knowing fb, there might well have been; if they had a facial recognition webcam up and noted applicant scores no one would hear about it. They'd probably just call it an "experiment in hiring"

As a Python beginner I'm completely flabbergasted... How can this be? And why can't I use "is" to compare integers? This is insane. The quoted stackoverflow is not very readable to me...

By the way y == 257 does work.

"is" is for asking whether two objects are the same exact object in memory. "==" is for asking whether two objects are logically equal, ie, that they represent the same value. As an example:

    >>> x = [1,2,3]
    >>> y = [1,2,3]
    >>> x == y
    True        # two lists with the same values; equal
    >>> x is y
    False       # two different lists; doesn't matter that they contain the same values
    >>> z = x
    >>> z is x
    True        # same list object in memory

Because here you want to know if the two integers are logically equal, not if they're necessarily the same object in memory, you want "==" and not "is". Due to an optimisation, "is" might work for small integers, depending on implementation, but the takeaway is not to use "is" for value comparisons; that's what "==" is for.

To expand on this, python creates some values as singletons to save memory/speed things up because they are used so often. The most well known examples are True/False/None. That's why you can always do None is None, because all None are the same None in memory. Python takes this a step further by creating lower integer singletons. So any integer from 0-256 (not sure about negatives) will always be the same number in memory. Above 256 python will start actually allocating memory for integers, so two of the same number won't be at the same memory address, and there for won't be comparable by 'is'.

0-256 not sure about negatives

-5 to 256 (inclusive) [https://github.com/python/cpython/blob/master/Objects/longob...]

For performance reasons, python re-uses the same objects for small integers, which are 0 to 256 if I recall right. This is done so the interpreter doesn't have to allocate a new object whenever your code uses the rare constant "1". Object allocation is slow, after all.

Given this, the expression "1" on the current version of cpython returns the same object from the small integer cache every time. Not just an equal object, the very same object. Overall, don't depend on it - it's an optimization and can change all the time.

0 to 256 if I recall right

-5 to 256 (inclusive) [https://github.com/python/cpython/blob/master/Objects/longob...]

I took the same quiz, and noticed the exact same. When I saw the first one, my thought was "crap, I don't remember how many integers python interns", and then I saw the next question and figured it out, but it was sort of silly that a lot of their test was stuff like this that can easily be looked up and can change by implementation (e.g. does pypy do the same? ironpython?) Plus, I was a bit mad that they wouldn't tell you which ones you got right or wrong, or even how many. Why did I bother taking it then?

Yeah, they never emailed me the scores either, and told me there was no way to know at the booth... I even joked with them that they were just trying to link it to my fb profile.

I actually didn't catch the trick until I asked someone later if they had gotten the same problem twice, "Almost, but the values were different" which is when I realized it (I blame the beer during opening night). However, as I found out today, because the way they worded the question it was all on a single line (if I remember correctly). So both cases should actually have been True, and I was right both times :P

> (e.g. does pypy do the same? ironpython?)

I wasted... ahem, spent a few hours today playing with this and found a few things, it's pretty neat!

In PyPy, `is` will always be True for primitives. This also means NaN is NaN (but still never == NaN). It doesn't use memory location for things like id() like CPython does, but it guarantees you get a unique representation. [0]

In CPython (3.5.2), it will always cache [-5, 256]. Here's the code that creates that array at startup [1] and each possible init/copy function calls this first to see if it's a cached int [2].

But it can sometimes cache other values! It also depends on if you do it from a module or the repl, and it's different depending on what scope or line each name/value is on. This one weirded out some coworkers, all from one continuous repl session:

    >>> x = 500
    >>> x is 500
    False

    >>> x = 500; x is 500
    True

    >>> def wat():
    ...   x = 500
    ...   return x is 500
    >>> wat()
    True

    >>> x = 500
    >>> def wat_wat():
    ...   return x is 500
    >>> wat_wat()
    False

If you run it from a script you instead get True, True, True, False. You can disassemble it to get a hint at why. One uses STORE_FAST and LOAD_FAST, and the other LOAD_GLOBAL.

    >>> disassemble(wat.__code__)
    2           0 LOAD_CONST               1 (500)
                3 STORE_FAST               0 (x)

    3           6 LOAD_FAST                0 (x)
                9 LOAD_CONST               1 (500)
                12 COMPARE_OP               8 (is)
                15 RETURN_VALUE
    >>> disassemble(wat_wat.__code__)
    2           0 LOAD_GLOBAL              0 (x)
                3 LOAD_CONST               1 (500)
                6 COMPARE_OP               8 (is)
                9 RETURN_VALUE

Now I'm still trying to figure out why STORE_FAST and LOAD_FAST cause it to use the same object, and why those only get called when it's interpreted together. When I figure this out I might put together a longer post about all this.

Also don't use `is` for comparing integers. ;)

[0] http://doc.pypy.org/en/latest/cpython_differences.html#objec...

[1] https://github.com/python/cpython/blob/master/Objects/longob...

[2] https://github.com/python/cpython/blob/ba85d69a3e3610bdd05f0...

Wow, this is amazing research! I had no idea the rules actually varied this much. The same like one is especially odd. The LOAD_FAST seems to make sense because it's defined as : "Pushes a reference to the local co_varnames[var_num] onto the stack." (emphasis mine).

Also see the global case:

  In [2]: x = 500
  
  In [7]: def wat_wat_wat():
     ...:     global x
     ...:     x = 600
     ...:     return x is 600
     ...:
  
  In [8]: dis.disassemble(wat_wat_wat.__code__)
    3           0 LOAD_CONST               1 (600)
                2 STORE_GLOBAL             0 (x)
  
    4           4 LOAD_GLOBAL              0 (x)
                6 LOAD_CONST               1 (600)
                8 COMPARE_OP               8 (is)
               10 RETURN_VALUE
  
  In [9]: wat_wat_wat()
  Out[9]: True

But this only goes further to prove my point - it's a bit of a silly question to quiz someone on :-) For one the question didn't seem to be well defined enough, but also it tests something that no one in their right mind would do, and also I don't know if the authors realized these nuances themselves either. I guess it makes for good trivia.

Haha thanks, I was inspired by David Wolever's talk this year. This is the first time I've gone through the disassembler module or the CPython source, it's much less intimidating than I thought it would be.

Also wat! The bytecode for the global case makes sense, but I wasn't expecting True?!

Yeah, now I'm even more confused!

I consider myself barely Python literate (I've modified maybe 5 python scripts in my life) but also managed 15/15. Most of the relevant core semantics are extremely common to dynamically typed scripting languages:

- Reference semantics for pretty much everything, except perhaps basic types such as integers - but those are generally "immutable" (e.g. no 5.add(6) mutating the "original object" in-place) making by-reference vs by-value moot.

- Arguments copy the reference, they do not alias the original reference.

- Mutating functions, if the return value isn't being used, are either being misused (e.g. the code has a bug) or mutate the original object in-place.

- Closures alias the same variables

This could be JavaScript, ActionScript, Ruby, Powershell, Squirrel, PHP, ... hell, even a lot of the compiled statically typed languages share similar semantics. I'd expect someone coming from, say, a pure C/C++ background to have some trouble though, which is perhaps the point you're getting at.

Particularly the question(s?) about list mutability. That is such a language specific property.

>This isn't a generalist quiz, nor is it a good baseline for interviewing candidates for the hiring process

I believe that it tries to be a bait and rubbish filter at the same time.

+1 ditto. So far questions 3 and 4 were about how Python handles references, question 5 is then about Python scoping. The rest are pretty good.

For a few questions, I also just booted up my Python shell, rather than comprehending the script. :/

I've written maybe 10 lines of Python in my life, and I got 15/15. They did a good job explaining the Python-specific stuff with comments.

Yeah, I have very little (basically none) python experience, and stumbled on some of the earlier quesitons. I found myself guessing, not from lack of problem solving skills, but purely on how the language works.

It would have been helpful if each question had the relevant language specific information - for example, reference/value semantics.

Agreed. No knowledge in Python. How am I supposed to know specifics of x.append() and x = y within the same function?

The fact that it's python specific wasn't stated anywhere. I'm like 'is this argument a reference or a copy?' what am I supposed to do then.

The problem is that the function makes no frigging sense without the argument being a reference, so you're naturally forced to assume that it is.

I have managed to get 15/15 without =any= prior knowledge of use of python (treated it like Java in terms of local variables). Actually I don't know if the Python has integer types. Hardest question regular expression one (mostly because regular expressions are the type of write-only code to me)

that being said. Some remarks about questions:

1st fails for all NaN/empty. Returning negative infinity is hardly a good choice for an array full of NaN.

2nd I didn't like this one: "children" comes from nowhere. There is no 'definition' of order, so 'first' is quite confusing, perhaps 'any' suits better.

5th I'd have expected white spaces in the toString rendering of the array (checked via inspect element). Had to make some assumptions.

7th since only variables i1 and i2 are defined and 3rd/4th answers make no sense as loop predicates, the entire merge function body is rather pointless.

8th About the correct answer:: One problem with ASCII is that it only support English characters (and a few other European languages). The characters are Latin and even Ural based languages like Estonian/Finnish/Hungarian use the same base for alphabet. "Unicode is like ASCII, but larger.", I'd never consider them a"like", for instance converting hexdecimal numbers to their asciis code is 5byte assembly for 8086. Also ascii is sort of a code page while unicode tries to combine all code pages together.

10th But because they share memory with other threads in the same process, threads must synchronize access to shared data to maintain data integrity. "Synchronize access" implies certain data model/locking paradigm. It's possible to have entirely lock free/copy-on-write multithreaded code that will exhibit no need for explicit synchronization. Alternatively some code could still be correct with benign data races (but that usually comes close to poking the dragons)

11th best practice is generally to use prepared statements, which lets the database itself protect against SQL injections. The statement is not necessarily true, the prepared statement could be just standard execution with parameters being escaped by the database driver (client), technically still performing the escaping part.

12th Side channel, timing attack is correct but virtually impossible to observe due to the very short nature of hashes and the fact they are all in the L1 by the time the check is performed -- even SHA512 is just one cache line, processing the comparing function yields 10ns or so time difference. The real concern is "the user_submitted_hash" -- users should not be submitting hashes directly as that precludes salting and invalidates use of any =slow= hashing function as the attacker just submits more hashes. IMO a bad question.

14th One advantage of DFS, however, is that it uses much less memory. BFS must store a queue of nodes to visit, and this queue can grow to (order) the size of the entire graph. The statement is not universality true and the worst case scenario is the same, a good answer https://cs.stackexchange.com/a/299. More importantly DFS can be executed in parallel.

I started reading the green part after first few answers, hopefully the remarks are useful (and not snark).

The only thing that was truly python specific was the UTF8 question - you'd need to know the library. The stuff about reference equality is pretty common in almost every language that abstracts away pointers - js, ruby, c#... The thing about closures was a basic closure question that anyone with experience in a language about closures could answer.

I only looked at three questions, but two of them were entirely Python specific. Question 2 asks you to choose between:

    queue.extend(node)
    queue.extend([node])
    queue.extend(node.children)

Sure, I could take a guess which one of those appends the contents of "node" to the end of "queue", but I'd be guessing about syntax, not programming.

After that, and question 3 about what happens when you assign to a function argument, I realized it was a Python quiz and stopped reading.

That's not python specific. When you're running BFS, after visiting a node, you next visit each of that node's children.

From a python-specific viewpoint, dequeue.extend appends a given list to the end of the queue. The first example will raise an exception, but the other two are perfectly valid, and their behavior isn't special to python - it's just asking whether you'll want to vi it this same node again, or the node's children.

Incidentally, it's not as if there's any standard library 'node' class with a children method. This was a structure they defined, but did not share the definition of.

> Incidentally, it's not as if there's any standard library 'node' class with a children method. This was a structure they defined, but did not share the definition of.

Ahh, that makes things much clearer. Since they weren't otherwise mentioned, I supposed that "value" and "children" were properties of some existing Python collection object - and that the question was asking how "queue.extend" expected that collection to be passed in.

I guess this is the inverse of a language-specific question - if one knows that "children" has no significance in Python then I suppose the question would be clearer.

The intent was that node.children would be the giveaway. The question is about BFS, and you'll notice that only one of the answers references the node's children (in a sensible way). The intent was that that would be enough to answer it without knowing python queue syntax (for what it's worth, I know python well and don't have the deque API in my head)

Thanks for the explanation, please see sibling comment for context. I think that a comment like

    # assume "node" is an object with "value" and "children" properties

would make this question clearer.

I would argue that UTF8 was a character encoding knowledge question. Two of the answers were clearly wrong and the array did contains values over 127, so there was only a single possible logical answer.

Not entirely applicable to this particular quiz (most of the questions were relatively simple until the final few), but the interview questions like the ones that Hired and Triplebyte focus on honestly make me feel like an imposter.

I've been a software developer for over 4 years but I don't have a formal CS education. I've gone through CLRS along with MIT's open-courseware--but I rarely use any of that directly in the workplace and so have forgotten a lot of it. And I think I'm a pretty good software developer, yet in the past when I've tried to get into Hired or Triplebyte, I always leave feeling like I don't belong in this profession because I can't tell you the big-O complexity of DFS off the top of my head, much less write the algorithm on a scratch pad.

I've been doing technical interviews for a few weeks now (I was recently laid off) and I'm honestly just a little tired of the brain teasers and non-applicable algo questions--and more often than not, the interviewers flexing their "knowledge" muscles.

Case in point: I'm in final stages with multiple companies for a Rails position, yet only one of those had even mentioned Rails in the technical interviews--the others decided to quiz my CS knowledge and one of them even quizzed my FP knowledge (for a Ruby position…). I don't understand it.

Hello Ezekg,

I am sorry that our process makes you feel bad :( The simple truth is that there are a lot of different ways to be a skilled programmer. One ways is being really good at CS (and to people who want skills in CS, talking about DFS is a reasonable low-bar question). However, another way is writing great, well-structured rails code. We work with people all the time who don't have CS backgrounds (we help them get jobs at companies that don't care about CS). The intent of our process is to measure both CS and practical skills (we actually break it down on more axes, but you get the idea).

This does mean that if you go through our process, you will be asked CS questions. But you'll also be asked to code. And not knowing the CS questions is not an obstacle to passing.

Do you truly believe that being really good at CS will magically make you a skilled programmer?

Are they not different skill sets? One is a practical skill, the other an academic one.

I've met "programmers" who had depth of theoretical knowledge but couldn't code their way out of a paper bag.

I've watched online lectures where Doctors of Computer Science lecture about OO but clearly have no understanding of what works and what doesn't.

I will concede that being good at CS will give programmers an advantage in solving problems in a subset of domains.

I suspect I could add the word small to advantage and subset and that statement would still be true.

Do you truly believe that being really good at CS will magically make you a skilled programmer?

Isn't the problem more often that lacking awareness of theoretical CS issues can limit your potential as a programmer? For example, someone with that awareness would immediately recognise some of the later questions as standard graph theory problems and could then solve them appropriately. Without that context, maybe a good programmer could still figure out a working solution from first principles, but it would surely take them longer. And while I don't suppose many of us will be working on intergalactic transport infrastructure any time soon, similar recognition scenarios do come up in plenty of day-to-day programming jobs.

There's a difference between expecting candidates to know when to recognize graph theory problems and expecting candidates to know (and implement) Dijkstra's algorithm as the correct solution to some abstract problem. I don't remember what Dijkstra's algorithm is exactly off the top of my head (all I remember is that it's for graph traversal and has a memorable name), but I do know when to recognize graph theory problems so I can start looking for the correct solution.

I definitely agree that general CS knowledge is highly beneficial (it's why I put in the time to learn), but knowing implementations for specific algos doesn't seem as important in the day-to-day as interviews make it out to be.

That is nice, however when hiring people it is useful to get somebody who can actually implement the solution too, rather than just sitting back smugly having identified a problem as "Dijkstra's".

If somebody can get as far as recognizing the problem as requiring Dijkstra's algorithm, they can go pull a book off the shelf, or use Google, and find the details of the implementation. Or, more likely, find a pre-existing implementation[1] that can be re-used.

Personally, I doubt that many people are walking around with the deep details of Dijkstra's memorized and readily available, unless they have a job that very specifically focuses on graph operations and they do nothing else day in and day out. So you could memorize it just for the interview, and then forget it again by the next day. But is there really any point to all that?

[1]: https://www.google.com/search?num=30&q=graph+algorithm+libra...

There are however people who can fairly easily write graph traversal and recursive code on a whiteboard. I wouldn't mind someone not knowing Dijkstra's well enough to code it from memory, but I'd be leery of someone who struggled with basic recursion, either writing the code or running through it and reasoning about it.

I wouldn't mind someone not knowing Dijkstra's well enough to code it from memory, but I'd be leery of someone who struggled with basic recursion, either writing the code or running through it and reasoning about it.

Fair enough.

I agree with this--but I don't think the interviewer should be asking for an implementation of Dijkstra's in the first place. That's my whole point--it's not a real world problem to need to know an algorithm from memory. If I need a particular algo, I'll look it up and find a supported open source library that implements it 10x better than I could have.

My point is that you can gauge somebody's knowledge without making good software developers feel inferior e.g. by asking them how they would go about finding a person's likely connections given some weighted data for an imaginary social network. And then you can maybe get responses like, "That's a good question--I've dealt with that before when I was working on X, Y, Z and ended up using weighted graphs according to how many friends a person had in common …" or "I'm not entirely sure on the exact algorithm I'd recommend, but that sounds like we would set up a weighted graph and go from there …".

Rather than making talented people feel stupid, you encourage conversation. But what do I know, I don't interview people every day so take what I say with that in mind. I just think tech interviewing is broken. I'm always left frustrated and my excitement to join the team dies down due to thinking I won't fit in because I'm not "smart" enough.

I mean, you're speaking as if you had the magic cure-all to technical interviewing, and if you did I'm sure you'd be telling us what it is instead of somewhat condescendingly criticizing someone else's methodology.

I certainly didn't mean my reply to be condescending. Are you objecting to the word magically?

He literally says that one way to be a skilled programmer is to be really good at CS. How is that not magically? What word would you have used?

Unfortunately, your reply is clearly condescending. And adds nothing to the discussion.

> I will concede that being good at CS will give programmers an advantage in solving problems in a subset of domains.

> I suspect I could add the word small to advantage and subset and that statement would still be true.

I doubt that's even close to true. You need CS knowledge to optimize programs and to design reliable services. It would only be true if you squint really hard and cherry pick some domains where CS knowledge isn't valuable and cherry pick what counts as 'academic CS knowledge'.

I would have edited my previous reply, but it was too late.

> Are they not different skill sets? One is a practical skill, the other an academic one.

I'd like to see you design and program a P2P networked program without any CS knowledge, like Kazaa or Bit Torrent, that actually works as well as they do.

> I've met "programmers" who had depth of theoretical knowledge but couldn't code their way out of a paper bag.

That's true for lots of fields, but it's not hard to see that that doesn't mean theoretical knowledge isn't useful.

It's a lot, lot easier to read papers on the algorithms needed by, and understand the specific requirements of, a particular project than it is to learn the entirety of software development outside algorithms ad hoc.

Honestly, even though I do know a fair bit about the Bit Torrent algorithms, I wouldn't trust that knowledge since it's most likely out of date or incomplete. It's deserving of research for anyone, even people who 'know it already'.

That's why I don't understand these narrow algorithmic tests - very, very seldom is a priori knowledge about DFS vs BFS going to be something that makes or breaks a project.

It would be akin to testing civil engineers on the memorization of precise ratios of the ingredients of concrete in current usage, instead of on the broad applicability of particular materials to particular purposes.

Yeah, I'd love to know how they test civil engineers in interviews.

For a working peer-to-peer network, who would you bet on to get a working solution first:

1. a programmer without any formal CS training

2. a CS graduate without any practical programming experience

Most programmers without CS degrees would pick up the concepts needed in a few days, most CS students would take months to even be able to finish a basic program. And it would fall over as soon as you sneezed at it.

While there will be exceptions on either side, run that race a thousand times, and the programmers would win the majority of times.

This is why I said you'd have to squint really hard to find something like this. You're clearly moving the goalposts here. I didn't ask about just a working solution, which could be almost anything. Even just something as hacky as everyone running an ftp server might fit such an extremely vague criterion.

But to run with your new scenario, professional programmers could come up with just a working solution quickly. However, lots of CS students could do the same. Pretending like no CS students can program reasonably well is disingenuous.

> most CS students would take months to even be able to finish a basic program

I don't have statistics on that, but I suspect most seniors and juniors could get something working in far less than a month. You're just trying to paint them in a worse light here. It's not like CS students don't have larger projects they complete in relatively short time frames.

> And it would fall over as soon as you sneezed at it.

I bet the that would be even more so for programmers with no CS experience. The reason I mention P2P programs is that they require CS fundamentals to function reasonably well. And CS students would be far better equipped to look up and understand and implement those things. Your average programmer who thinks CS is unnecessary is likely to write something that indeed 'falls over as soon as you sneeze at it' here because under a real load, it will not perform well.

> While there will be exceptions on either side, run that race a thousand times, and the programmers would win the majority of times.

For writing yet another Rails website or somesuch, maybe. For an example like mine, that requires in-depth CS knowledge, not a chance! I think the reason you are so confused is that you assume CS students can't be 'programmers' and/or you're mentally pitting experienced devs against new CS students. I doubt a new programmer that didn't have CS knowledge would do well at any of this!

> that actually works as well as they do.

I doubt most CS grads could do this. One hard part of it is the p2p (but that's already been solved [0]). Having a project that large where you don't eventually shoot yourself in the foot because of architecture and finish in a reasonable time without using too many resources on a user's computer is also a hard part.

[0] https://github.com/bittorrent/libutp

> I doubt most CS grads could do this.

Of course not, but the point is that the information can be valuable for making real things.

>I suspect I could add the word small to advantage and subset and that statement would still be true.

Sounds like you're only familiar with programming to solve easy problems. Lots (most) of us do it, and that's fine, but there are a large number who tackle the more complex domains and need both skills.

It's because I do realise there are a small number of domains that do benefit from formal CS knowledge that I couched it that way.

You are, after all, agreeing with me. Most.

I'm disagreeing that I don't believe it is a 'small' number of domains, only that they are domains you're not familiar with (or aware of.) Other than that though, yes, we agree.

They probably assume if you know programming, Ruby, and everything else you should be fine with Rails. Quizzing people about minutia that can be easily Googled doesn't really sound like a great way to find apt candidates.

Been programming for a living for 30 something years, my last contract ended recently so started investigating the technical interview process so sympathetic with your point of view. I've never had to do any of the basic algorithms/puzzles from scratch that one typically has to do in these interviews for a shipping product, based on the samples I've seen from hackerrank and leetcode. The questions marked easy on leetcode were ones I was asked in the 90's.

I don't understand people who don't do prep for interviews ... https://news.ycombinator.com/item?id=13888581

and I don't understand people who want to waste a month (or more!) of their life every time they change jobs... https://news.ycombinator.com/item?id=13888703

Well different people need different amount of time to prepare for interviews, but if preparing for one month gets you a much better job offer, that sounds like a good investment to me.

You've completely missed the point. If you need to do this every time you change jobs, then throughout your career you're spending years! of your life on interview prep. Getting into Google for the first time, sure, it's worth it then. But what about the next job? Or the job after that?

I've always just kind of assumed that everyone does a fair amount of interview prep. Maybe not a month, but at least a week or so to refresh your interviewing skills and practice. The skills needed to pass an interview have almost nothing to do with the skills you're employing day-to-day in your job, so you should expect them to get weaker the longer it's been since you've applied for a job.

I mean, what do you guys do, just walk in and wing it?

Yes. Just walk in and wing it.

Ok, so I generally do research about the business to make sure I understand the problems they are solving and have good questions to ask, but never CS interview prep. And I'm not some coding genius...I couldn't rattle off big-o for any algorithm off the top of my head, for instance...maybe a little above average.

Then again, I've only worked for 5 employers in a 19 year career. I think that was something like 18 interviews total in my career. A grand total of 1 or 2 required any deep CS knowledge and those were questions like "write fibonnaccci on the whiteboard*" or write a linked list in a code editor. I don't consider those to be worth studying...you should be able to do those off the cuff. And also, I'm pretty sure I can come up with a reasonable solution to most problems presented under pressure...something I know throws some people off.

Anyway, those 18 interviews yielded something like 9 or 10 job offers. And the fibonacci one...no offer...stared at it for 5 minutes after getting stuck on one of the conditionals before finding my mistake and didn't get the job. But whatever. There are always more interviews.

Now, I've also never approached a job like "I really want to work at Google/Apple/Facebook/etc so I have to ace this interview". I've always taken a broader, more laissez faire approach to try to find the right fit for me. So it might just be a matter of your approach to a job hunt.

I used to prepare for about a week every time I switched jobs early in my career. That's back when every company just asked typical DS&A whiteboard questions. Now, every company has its own method. I don't bother to prepare because I never know what I'm walking into. I just wing it.

This comment[0] says it better than I can:

unless you're interviewing at one of the big companies with a known interview process, you'd basically be wasting your time.

I've done hundreds of tech interviews over my career, and especially in recent years (now that companies are trying new things), it's completely random.

Some will do "Cracking the coding interview" type shit, some will do code reviews, some will give you a take home thing you have to present, some will do pair coding. Some do algorithms only, some do design discussions only, and so on and so forth.

So any prep I do will be a shot in the dark and 99% of the time I'll have to do something I did not prep for. So I don't bother trying.

Notable exception is my current job, which is a big tech co with a semi well known interview process (not as infamous as google or facebook, but still), and they gave me a fair amount of info up front. I also -really really really- wanted to work there because the team I wanted to work for did things no one else does. So i bit the bullet and studied/practice.

That was literally the first time (and probably last) time I did.

[0]: https://news.ycombinator.com/item?id=13888703

I never prepare and I received offers in maybe 13 out of 16 interviews.

When I hear people are getting offers in that high a proportion of interviews, I wonder if it's time for them to set their sights higher.

Geez, how many jobs are you planning to have?

Changing jobs gets you a higher pay increase than interviewing well once.

I dunno, I just ask for raises based on what I did in the past year. I've heard many employers say no, but if they don't want to lose you and you have a good case (management loves numbers, show them numbers) you should get it...

Changing jobs often is not rare in CS and related fields.

I'm amazed that it takes a month to refresh basic concepts like BFS and DFS.

This was an interesting quiz, but perhaps not the way the authors intended. The second half was straightforward first-year computer science, but the first half challenged me to speculate about the syntax of a novel scripting language.

Agreed. I've been a professional developer for nearly 20 years. I've shipped kernel C, windows C++, java,python, ruby, clojure to production, but the last time I wrote python was nearly a decade ago. Since then, I've been firmly in the Clojure community, with no intention of leaving.

I know I'm a good programmer, and I got nearly every python-specific question wrong. I understand closures and call-by-reference tyvm, but I don't remember or care about the quirks of python.

Triplebyte in particular seems to not care about the difference between "good at programming" and "good at programming in python". The skillset needed for C, python, clojure and JS are wildly different. They do a disservice to their users by pretending otherwise.

For what it's worth, we run our interviews in the language of the engineer's choosing (including clojure), and get people jobs at Clojure and Haskel and C++ companies. We went with Python for this (toy) quiz because it's the most common language we see among applicants. I tried to add comments to explain anything python specific, and make the question about the underlying logic. But it seems it's still easier for Python folks.

Sorry, not buying it.

I caught 3 "Whoops sorry you missed a weird bug in a specific programming language that you weren't explicitly being tested on". Sure, if I was doing a Python3 quiz, I'd expect those. But this wasn't what the quiz claimed to be.

Turns out, the "fill in the blank" code was easy, got all of them. Along with all the 1st year theory questions.

I agree. Coming in with a LOT of experience in JS and a good damn chunk in c++, but none in python... this is really bad test.

You ask dev about aspects of the language that are essentially memorization:

Does matrix.append(x) in python append by reference or by value? Both are equally likely without knowing anything about the language.

the question is also pretty easy to answer correct IF I know that, but I have no reason to know that. You're not testing skill in any sense of the word. You're purely testing familiarity with this specific language (and worse, this language's standard library)

That's bad.

> Does matrix.append(x) in python append by reference or by value? Both are equally likely without knowing anything about the language.

I don't think they're equally likely. The question would have the same answer in Javascript, Ruby, Lisp, Perl 6, Java, and many other languages. You can see from the code samples that Python is a high level, dynamic, imperative language, so I think it's fair to assume you would expect that behaviour from it. There's a bit of guessing, but you can make an educated guess.

Way late, but your response sort of proves the point. You're having to argue about what I can infer about the language.

What JS/Ruby/Lisp/Perl/Java do is irrelevant. Hey, C/Go/D/C++ would all be more likely to append by value, but neither of us care a whit because it's asking about Python, which I know nothing about.

So (and this is really hard) try to forget that you've got experience with the language, and approach it as a random pseudo language: this question has no clear answer, because it depends on the memorized implementation details of the standard library of a language...

That's bad.

---

Finally: JS doesn't have matrix as a base type (boolean/number/string/null/undefined/symbol is it, baby), so for this same question, I'd strongly argue that you're simply wrong about bucketing JS with those other languages. Again, for JS the question would revolve around the implementation details of the matrix object and its prototype. Trust me, I can write a useful matrix implementation that appends by value in JS.

> What JS/Ruby/Lisp/Perl/Java do is irrelevant. Hey, C/Go/D/C++ would all be more likely to append by value, but neither of us care a whit because it's asking about Python, which I know nothing about.

I guess my point was that you should be able to infer that Python would be more likely to behave like JS/Ruby/Lisp/Perl/Java than C/Go/D/C++. If you know or can tell by looking at it that it's much more in the first camp then the second, then you should be able to guess. You're probably right that I overstated how easy that is to tell just from the syntax if you don't know anything at all about Python, but it's also pretty short of memorizing implementation details.

> Finally: JS doesn't have matrix as a base type (boolean/number/string/null/undefined/symbol is it, baby), so for this same question, I'd strongly argue that you're simply wrong about bucketing JS with those other languages. Again, for JS the question would revolve around the implementation details of the matrix object and its prototype.

The 'matrix' in the question wasn't a special matrix type, it was just a list of lists. Word-for-word translated into Javascript it would look like:

    let matrix = []
    let row = [0, 0]
    for (let i = 0; i < 2; ++i) matrix.push(row)
    matrix[1][1] = 1

Question: If you were given a new block of code below, what is the result of X if X is passed by reference?

[Block of code]

     1. 34
     2. 17
     3. 43
     4. 22.5
     5. Not listed

____________________

This would be a much more clearer way of handling pass by ref/value. 2 of the answers are correct, so this could be reused to ask the other one. If they're being tricky, they can award half-credit for choosing the wrong pass-by-*. And 3 other answers are junk. And this abstracts out any sort of programming neologims or fetishization of $random_language.

But that LITERALLY is memorizing implementation details.

Here are the rules for JS, which I happen to know, because I use it often:

1. Javascript is always pass by value, but when a variable refers to an object (including arrays), the "value" is a reference to the object.

2. Changing the value of a variable never changes the underlying primitive or object, it just points the variable to a new primitive or object.

3. Changing a property of an object referenced by a variable does change the underlying object.

Those are not trivial "just infer it" details. That's highly language specific, and absolutely unrelated to the skills the test claims to be evaluating.

Agreed - this is a really Python-heavy thing that shouldn't be. I got 13/15 largely through guessing the syntax and semantic weirdness of a language I only rarely touch and never for long (like, the matrix question, by-value or by-reference? I had a 50/50 chance to get it right).

Not a big fan. Feels really weird.

Yes I didn't see those comments.

Possibly, the "Is Python call by reference or call by value" question could be rewritten in a language independent form by providing correct syntax Python, the correct python output, and asking verbally "Given either this evidence OR your Python coding experience, is Python call by reference or call by value?"

I've never used Python. I breezed thru the theory and regex questions, presumably I could be a halfway decent Python programmer in a week or two, it looks reasonably non-tricky. Thats another issue, some jobs merely require syntactically correct code, others require good algos, and its a lot easier on the job to learn syntax than to learn algos.

I like that idea for the question!

There's certainly nothing objectionable about choosing a common language for a toy quiz like this. It did feel jarring to have the early questions be language-specific, and the UTF question was sort of unknowable, but I got it by guessing what the most likely meaning of the syntax was.

I was frustrated that language features felt trickier than actual coding knowledge in the early questions, but given that it's nothing with actual stakes there's obviously no harm in it.

Those comments definitely helped. But there were a couple questions which came down to "does Python copy this data structure or simply pass a reference", and all I had to go on there was "if I was designing the language, how would I do it?"

In one case, I apparently think that Python got it wrong. :-)

If you wanted to make a generalist quiz, why not ask the questions in psuedocode?

Asking the user to choose between 'queue.extend(node)' and 'queue.extend([node])' is purely a Python syntax question; it has nothing to do with programming per se.

Neither of those are the correct answer, though.

The correct answer is node.children, which isn't syntax-specific, it's about the pseudocode of whether you want to visit the node, or the node's children.

Using pseudocode is a great idea. We've talked about that. We've not done it because we still need to decide what style of syntax to use (and that will still preference certain programmers). But maybe we should think about it again!

Maybe start off with a The Elder Scrolls style background quiz, where people are asked things such as "Ever since childhood you've been having terrible nightmares about (a) curly braces (b) dynamic typing (c) memory management.

And then, after 20 background questions, you could know with high certainty what style of pseudo code (or even language!) a person prefers.

Yeah, I think this is a great idea.

Although it would be a fairly decent extra amount of work ... make the quiz available in the top 20 languages and have the person choose which language to view the quiz in.

I have only ever written maybe 300 lines of python, and I was able to get every python question correct. The questions could easily be translated line by line to Ruby of Javscript and the answers would be the same. Even if you only knew C you should be able to get 5 out of the 9 python questions. If you are familiar with Ruby or Javascript then you should get all of them. The only reasons I can think of for getting all or most of them wrong as a competent programmer are that you are not good at multiple choice tests (the answers give a lot of hints), and/or slightly different syntax ruins your understanding.

Breakdown:

findMax: language agnostic - should be easy for any programmer

BFS: mostly language agnostic - requires knowing about the .extend method, but the meaning of it can be deduced from context (I did)

modify: requires knowing how variables work in python (storing references to objects), this is the same as Javascript, Ruby, Java, and other. If you only used languages with a different model like Haskell or C, missing this is totally excusable

matrix: same as modify, just a little more complex

factory: requires the variable knowledge + knowledge of closures

int_to_base: language agnostic - only requires knowing recursion

merge: language agnostic - just requires thinking through how merging works

decode-utf8: pretty much just a python thing, but it shouldn't be too hard to guess well if you know about utf-8 and notice the name "bytearray"

I think it's reasonable for a person with good experience of multi-paradigm dynamically typed languages to work out the solutions to those questions.

As a person who has strong JS/Lua knowledge, I didn't recognise the language as Python, but managed to get them right. I don't think my past experiences of C++, Java, C# would have been at all useful. I've also had a look at languages like Clojure, Elixir, Swift and Rust in my own time, and again that's not useful for this quiz.

It's a fun quiz, but maybe some of the assertions they make ("Only 3% of programmers can get all the way through") are not very fair, given it's not a true assessment.

Oh good, I wasn't the only one. The second question had me wondering if it was testing my knowledge of tree searches or python queues. I don't know python at all.

Exactly, I know what a breadth first search is. I don't know the specifics of how queues work in python.

I can attest that you are not the only one. Don't know python either, Nor should you need too.

"There should be a warning label on this thing!!!"

Interesting. As a datapoint - I've never programmed in Python (mostly Java/Clojure) but had no trouble answering the questions.

Java turns out to be a solid reference point for this. JavaScript might be even better. But there are still issues - not only would this be hard for a C++ programmer, it's hard for a Java programmer who overthinks the thing. Sure, I know how Java handles pass by reference, but unless I know that Python works the same way I'm still guessing.

The wording on question 9 (the regex one) is a little unclear: both b[l].e and ([^b]b[^b]*){3} will match blabber but not babel. Presumably the intended meaning is that the entire string must be matched, but the question does not make this explicit.

Just added anchors to the expressions. Thanks for pointing this out!

I got the same one wrong, for the same reason. I picked the first one that worked instead of looking at all and picking the best one.

Glad someone pointed that out. I chose b[l].e and got it wrong and was scratching my head...

I'm ashamed that the only one I missed was this regexp question, even after the anchors were added.

Cute quiz. Question 9 is incorrect, however – or at least poorly worded. It asks about regex matches and only considers the last one to match, while the second one also matches, albeit incompletely. The last regex is the only one that matches the entire input string, but that's not usually how regexes work – e.g. in Perl or Ruby. This verbiage seems to be specific to Python, which uses `re.match(str)` to mean "match the entire string" as compared to `re.search(str)` which means "search for any matching substring". Arguably a decent distinction, but very Python-specific.

[Update: this is now fixed in the quiz.]

The given regular expression does not match `blabber`, it matches `blabbe` (minus the `r`).

I got that one wrong as well. The question should have been "which regular expression FULLY MATCHES X and not Y"

Nope:

In [3]: re.search("b[l].*e", "blabber")

Out[3]: <_sre.SRE_Match at 0x7f2db79ed030>

"Matches" means "matches". If they wanted "fully matches", they should have specified "^$". That's why anchors exist.

Yes, I agree. I just explained where it went wrong and that "Match" can mean multiple things. It matches `blabber`, but the match result is `blabbe`.

Thanks, I had to check the Python docs to be sure I wasn't crazy! I went A->D, stopped at B because it 'worked', and got it wrong. D also works, obviously, but I'm glad the question is cleaned up now.

Got a few easy ones that were about language implementation details wrong because I've never done Python. Only realised it was Python and not some kind of beautiful pseudocode when I kept getting answers wrong

We've been using a quiz as part of our hiring process (at Triplebyte) for the last two years. However, a lot of people do our quiz for fun. So we decided to create a version optimized for this (and hopefully a little education). The problems come from the most common mistakes / misunderstandings we see when giving interviews, and ramp up in difficulty. We put a lot of effort into writing interesting answers.

Let me know what you think!

If you're going to pick one language for the quiz (python) it seems a little unfair to non-python developers to focus the questions on idiosyncrasies of python. Whereas if all questions were of the more general algorithm variety (like you see toward the end), non-python developers could infer the answer without any esoteric python knowledge.

We talked a bunch about this. Python and JS are the two most common languages we see on our site, and Python has slightly more compact syntax, which works well in the questions. The idea, however, was not to require any python-specific knowledge. Which question do you think is looking at syntax?

> The idea, however, was not to require any python-specific knowledge

But the quiz specifically mentions on multiple answers that "In Python, unlike language X, Y, Z" OR "In Python, like [similar language] A, B, C"...I mean this says nothing about my programming ability, this says whether I know the quirks of the language in the same way that I can tell you all the quirks of JavaScript but not in Python. This to me is not a programming quiz, but a Python quiz designed to determine how much someone knows about what can go wrong in Python.

Well, the questions get into other things too.(Character encoding, timing attacks, graph algorithms.) Writing a good single-paragraph description of iterative deepening depth-first search was a challenge!

Yeah except that character encoding question assumes I know how Python handles bytearrays. That trick answer of "it will break" threw me for a loop because I needed language-specific knowledge about the types that Python could handle for that input.

And then with timing attacks - I mean SQL Injection is a fair web security question to ask someone, but the timing is like, very fringe knowledge. Most devs let their framework handle all of those kinds of things. Not sure why I would need to know this unless I was specifically in the sec world.

The last 3 questions were actually general CS because they went through search trees and shortest paths algos, which are language-agnostic.

Hey, not the GP, but just glancing through the quiz - Question 3 focuses on the fact that Python passes things by reference to functions. This is definitely specific to the two languages you guys use (as is your prerogative), but if you're expanding this for others, you should definitely explain that these focus on the JS and Python languages.

Additionally, question 2 amounts to "what is the usage of python's queue class", question 4 requires knowing whether array indexes start at 0 or 1 in python, and question 8 amounts to "what is the usage of python's bytearray class".

I'm not sure that's fair about question 2. It's hard to say because I know python (but not the deque class), but none of the incorrect answers seem like they'd be correct in any sensible language.

Like, whatever language you're using, you need to push the children of the current node on to the queue. Only one answer looks like it could reasonably do that.

Yeah, agreed. Question 2 as asked demanded Python knowledge, but once you checked against the answers provided it seemed much clearer that 3 options were invalid.

Also, strictly speaking, Python is not "pass by reference" as the term is used for other languages. It's more accurately "pass by object" or "pass by object reference." Somewhat of a pedantic distinction, and the quiz question is clear regardless.

[1] http://robertheaton.com/2014/02/09/pythons-pass-by-object-re...

[2] http://foobarnbaz.com/2012/07/08/understanding-python-variab...

[3] https://jeffknupp.com/blog/2012/11/13/is-python-callbyvalue-...

Good point. I think that objects are passed by reference in a high enough % of language in common use (PHP and C++ are the two exceptions I can think of) that this is a fair question. But if enough people think it's bad, we'll remove it.

You got me on that too. A high enough % of the languages pass stuff by value (C/C++), copy on assignment (C++, D, I'm sure there are more), or do not have the concept of references or changes to start with (Prolog, Haskell).

Anyway, if you want to bias it into Python, go for it. It's a fun quiz.

Copy on assignment was what got me, and I didn't feel terribly bad for missing it. No harm done though, seeing as nothing was riding on the quiz.

Every functional programmer in the world is looking at you askance.

Q2: THis isn't a syntax thing, but it does require knowledge of python queue objects to get right

Q3: This requires knowledge around the semantics of argument passing, which vary from language to language.

Q4: this again requires some understanding of how values are stored. Some languages will copy, while others will use a reference to the originally inserted value.

Q5: python closure syntax. python scoping rules.

Q8: unless you know what "bytearray.decode" does, the best you can do is make an educated guess from the context.

they didn't say syntax, they said Python idiosyncrasies. Knowing move/copy rules in python counts as that. It's something that isn't intuitive unless you know python, ex: the matrix question. An experienced dev who isn't a python person would pick the intuitive choice, see the matrix is malformed, then Google how to clone objects in python.

On the other hand, how many languages do you know where arrays are passed by value?

My primary language (Erlang) does. I assume all functional programming languages do.

That's true. I suppose my next argument is that seeing matrix.append(row) on its own line signals strongly that this is not a functional language.

Is that true? I thought most functional programming languages did call-by-reference. I mean, if you have immutable data you can't tell the difference (outside of performance).

Under the hood it's probably by reference, but the semantics are by value. By reference indicates that the original value can be changed in the called function.

I think. Not a language lawyer.

lol, the same mistake can be made in a lot more languages~

I used many programming languages - C, C++, Java, Ruby, Elixir, C#, Swift, Objective-C & Javascript.

I've never used Python.

I only did the first 3 , but both 2 and 3 focus on python-specific names (extend(), queue) and pass-by(reference/value).

popleft is shift in many other languages.

Not all languages manipulate arrays in the same way re ref/value.

Not all languages have the concept of range() - you could have used a simple array to be more language agnostic.

(I'm sure I could have found more, but I'm Python isn't a language I spend much time in)

Just to be fair, i think the comments in the source code has explained it well

Yes, but finding answers required to understand what was set by ref, and what was set by value, which wasn't in any comments.

It feels like a lot of this quiz is just trying to "catch me out" on subtle behaviours of Python I didn't know or might have forgot about. I guess this is fine if you're making a little trivia toy to distribute to Python developers on the internet, but I don't really think it achieves much as a hiring tool. Getting questions on this test wrong would probably demoralise a lot of great candidates.

I got them all right, and I agree that there were many gotchas. For example, the ones about the matrix, the array modification, etc. The only thing worse would have been something like:

https://www.pastery.net/mbeghh/

(it'll print 3)

Hmm. We included those issues, because we see a lot of programmers get stuck on exactly those two bugs! (In all sorts of languages, not just Python). I guess maybe that's the definition of a gotcha -- a thing that a lot of programmers get stuck on? In any case, I don't think it's an arcane gotcha! Being clear about references and shallow / deep copies is important.

Being clear on reference vs value is certainly important, but you lose the ability to honestly call it a programming quiz. Those questions make it a Python quiz - which is fine! But not what you've sold it as.

In that case, by far the biggest Python gotcha is:

    def do_stuff(mylist=[]):
        mylist.append("thing")

Which will lead to the list containing N items after N invocations (i.e. the second function call won't start with an empty list).

Yup, we see that issue too. But in that case, I think it's python specific (how python handles default values). Ruby, for example, does not have that behavior.

These issues might have a place in technical interviews, but framing it as a question the user can get "wrong" feels a bit like you're setting a trap. In a real interview, this would just be a talking point. What would be interesting to an interviewer would be seeing how quickly the interviewee grasps what's happening that is unexpected, and why. Marking the candidate down because they didn't know about these quirks beforehand is kind of missing the point.

do they get stuck on it in quizzes or actual programming? I haven't seen devs, other than new devs, get stuck on this, very rare to see it on stackoverflow other than for students. I think the problem is the lens you are seeing things through is this kind of quiz like thing and what trips programmers up doing a quiz. I'd like to see repeatable research into correlations between a vetting quiz and actual capability of producing software.

Fun exercise. Thanks for sharing.

I'd suggest fine-tuning the question about hashes. My first impression from reading the code, specifically the name user_submitted_hash, was that we were looking at some sort of client-server system where the hash was being calculated remotely and then submitted instead of the original credentials, and that this was a question about not trusting the client. Even without that, in a client-server system the attack you give as the mostly likely wouldn't make much sense given randomness in network latency, so the misunderstanding about the context still breaks the question. Maybe a little more context about how this hypothetical function would be used would help here?

In terms of the presentation here, if you want to show the answers immediately for each question, you might also like to provide some sort of confirmation button. It's frustratingly easy to hit the answer next to the one you intended, particularly on mobile devices. If you're also keeping statistics based on earlier attempts by other people, this is probably distorting your numbers as well.

Timing attacks like this still work over a network, you just have to make an LOT of duplicate requests, and average the times. The lucky thirteen attack (https://en.wikipedia.org/wiki/Lucky_Thirteen_attack) is a real example.

Good point about the accidental answers. I'll look at that.

"A lot" is going to be a mighty big number in this case, though. In some networks it will surely be prohibitively large. But even if the latency is predictable enough for that not to be the case, what real world system is going to let someone make that many failed authentication requests without doing something about it? You might be better just trying to brute force it.

The trouble with hypothetical security questions is that it's so difficult to conceive a scenario where you have a deliberate vulnerability that you want someone to see, yet no other implausible aspects that will throw off anyone capable of seeing it. I am reminded of a murder mystery party I once went to with a group of mathematicians, programmers, and other logical people. When we got to the end of the evening and the true murderer was revealed, I think the majority of the participants had long ago ruled out that suspect on the basis of one of several subtle deductions from the clues provided, none of which had apparently been intended or considered by the authors of the scenario. Instead we were supposed to have ignored all the minor inconsistencies in clues and ambiguities in phrasing, and just gone for the person with the flashing neon sign over their head in the first place...

I am actually curious to know of cases of insecure string comparison vulnerabilities being exploited. I think there are real cases. But I tried recently to to this against a toy server over my local network, and could not get it to work without adding a delay to the comparison.

Question 9 appears to have two answers that are correct, but only one is accepted.

Hmm. Just checked again. Two of the expressions match neither word, one matches both, and one matches only one.

EDIT: I see. The confusion is over partial match vs whole match. I thought whole matches were standard when talking about regular expression in the abstract, but looks like I was wrong. Just changed the problem to include the anchors. Thanks for pointing this out.

That's not the standard meaning in my experience. Regex's are often used for extracting information from larger strings, forcing a full match by default would be pretty counter to their purpose.

Yep, I just came here to post that:

In [2]: re.search("b[l].e", "babel")

In [3]: re.search("b[l].e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030>

It looks like they meant "match" as in "re.match", i.e. "^b[l].*e$", which is not the usual definition of the word.

That would also make this question specific to python, which wasn't at all what I was thinking about. It's a question about regular expressions, it should be as agnostic as possible to implementation.

Yeah, agreed, although there are different RE flavors, so you can't get too agnostic.

To be honest, this quiz makes me feel that I'm learning more about you than you are about me. I especially find the reactions to the complaints interesting. This response could be read in a negative way, which is not as I intend it. Hopefully it can provide a window to view things from a different perspective.

It is not possible to really determine the ability of a programmer in an interview -- the scope is too large. It takes time and effort to understand how well a person will perform. In an interview we try to find indicators of how well a person will perform. So, a quiz that tries to unearth those indicators is interesting.

Your quiz makes me feel that technical details are foremost in your mind when you asses performance. Will the candidate make technical mistakes that you have seen many times before? Similarly, there appears to be considerable bias evident in your selection of questions. All of these questions asses whether or not a candidate deeply understands the functioning of the code snippet, or whether they understand it shallowly. For the candidate who admits a shallow understanding, their only recourse is to guess (and potentially fail).

From that I infer that your cultural makeup is one where you have a bar that you expect all candidates to surpass. This bar is single dimensional, though, and especially your responses to the complaints imply a lack of realisation that you are narrowly optimising for a single useful ability.

Finally, the quiz is set up in a "run the guantlet" fashion. Can you avoid the pitfalls that others have fallen into? This tells me that you fear candidates who do not statically measure up to your bar more than you are excited to find out what a candidate brings to the table.

While you clearly will have other interview techniques for other aspects that you value, this particular quiz would worry me as a candidate. It's essentially the same question 15 times in a different context. When people complain that the quiz is too Python specific, the answer is essentially "But it covers concepts that are similar for many languages and you should be able to figure it out". It's a kind of "Well, I'm sure I could do it pretty easily, so you should be able to too". It makes me worried that there is a lack of empathy on team and that there will be problems if I don't think exactly like the leaders on the team.

On the other hand, this may be fine if that's what you want in your interview process. Personally, I probably would not apply to your company having seen this quiz. This may also be a good thing from your perspective :-). However, if you ever get to a point where, despite having excellent people on staff, you can't seem to get to the next level, I would concentrate on examining potential biases of what is "good" on your team.

Good luck!

"True programming generalists are rare. This quiz covers basic programming, web development, algorithms and low-level systems. The questions start easy, but get harder fast. Only 3% of programmers can get all the way through. Can you?"

This test doesn't cover any of that, makes me worry if they think it does.

it's more :-

do you understand python syntax?

do you understand a few algorithms?

do you understand a few security issues?

not sure how you'd go from this test to working out if a person is a programming generalist. In fact I'd be worried that someone passing this test would be fresh graduated from college where they used python.

imo, this is the right way to do interviewing. I'm currently interviewing with several companies, and one thing that stresses me out is the expectation that

1. candidate must discover algorithm to solve unseen problem

2. candidate must code up algorithm discovered in #1

3. candidate's code must work

4. steps 1,2,3 must happen under 1 hour

I think this is a bit too much, and completely misrepresents the current working state of professional programmers.

If I am asked to do #1, I usually do a literature survey - I read papers before rolling my own algo from scratch. 99% of the time, there's a good enough algo already out there.

If #2 - ok.

If #3 - not ok. Typically you need a debugger & 1-2 passes before you get it right.

If #4 - definitely not ok. Step 1 itself can take you forever, and you don't even know if its correct. Then you've got #2,#3...adding time constraint to this mix is crazy.

So yeah, this quiz has two highlights: showing a block of code & asking to fill in the blanks - much better, highly recommended. Also, looking at a block of code & figuring out what it does - definitely part of programmer's everyday work. I hope more companies adapt this technique.

My worst ever interview was for a mid-college internship. The task was a string-matching question, and I was explicitly failed for not using the Rabin-Karp algorithm. Mind, I had never heard of the thing, and I produced a valid answer with slower runtime, but since I didn't know it or invent a publication-worthy algorithm on the spot we were done.

Most interviews are a bit better than that, but I'm always disturbed when people ask for algorithms that aren't either common sense or common knowledge. If it was worthy of publication and isn't relevant to the job, it probably shouldn't be tested on the fly.

This system (and a lot of others) beat the heck out of that.

A live example, from Uptake (Chicago): https://twitter.com/ch1gg1ns/status/867043938779222016

i thought those were just for free tacos?

Hard to tell if you're joking or serious. (Either way: "Why not both?") Here is the official tweet:

https://twitter.com/Uptake/status/867105586336456705

I'm sorry but I just can't take a quiz/website/company seriously that assesses programmer ability, and yet they put the freaking answer key in Local Storage. Seriously, quizState? I got a 100%. ;)

{"questions":[{"name":"max_function","answerIndex":2,"question":"Fill in the missing line of code:","code": ...

This quiz is just supposed to be fun :) We opted for speed over security, on the idea that anyone who cared enough to pull out the answers deserves a 100!

Gotcha. Just know it could give a bad first impression about the security of other assessments and competence of the team. While I find these questions to be quite flawed for the stated purpose, I really do wish you and others in this space luck!

Yeah plus with allowing full access to my own browser I could write a HTML doc saying 15/15 well done. How unsecure!

Eh - there's nothing to be won by getting the right answers! At best, you could share a fake 15/15 on Facebook, but could do that regardless by taking the quiz twice.

Given that Triplebyte doesn't store answers that way for their job-finding quizzes, the criticism feels a bit hollow.

I actually appreciated that, since my phone has a spotty network connection.

This quiz is not a programming quiz but a Python quiz.

If you only fail the regex, unicode, and hash security question you ought to get a badge of innocence you can print and cut out

15/15. Some questions felt like they were trying to trap me. Or maybe I'm just tired, I've had a long day of work.

Congratulations on making this! It's less fun for me than algorithm puzzles, but I can see it working well for hiring purposes. I'd love to see some software company use it and report the results.

The quiz wouldn't pass security review, FWIW. The answers are shipped with the javascript, and validation is done client side, so they're just an 'inspect element' away.

I'd cap scores at 5/15 for applicants who automate their responses.

I especially enjoyed the post-response explanation of the time and memory bounds of iterative deepening.

It would be a shame to miss the opportunity to test one's skills on such a well put together experience.

In a quiz with multiple security questions, I'd give bonus points to an applicant who found security problems.

Good point.

:-) Yup. This is just for fun

On the last two questions, I'm curious why the ability to memorise which algorithm to use is a useful skill for a professional software engineer, let alone a useful interview question. Isn't the best answer "I would look up the best algorithm by ..."?

If you understand the graph traversal algorithms in question, there's nothing to memorize. You can just think through the problems. Is it important to understand graph traversal algos? I don't think it is for everyone. But a lot of people who like CS find it interesting. And if you work on, say, a web crawler or a game AI, then you'll need them.

The last two questions are little more than "what is the name of the algorithm I should use to solve this problem?".

This doesn't test understanding in any way. How is it useful to determine whether someone can work on the examples you give, let alone the average software engineering job?

The goal of this quiz is to be a little challenging and fun and maybe educational. It's not to judge whether someone can do a software engineering job!

The answers to the graph problems are indeed fairly straight forward. But asking about why one might use a BST in place of a sorted list prompts interesting discussion. And going into Dijkstra's vs A* and iterative deepening depth first search is (I think) pretty interesting!

It's highly unlikely that you would be good at solving the single source shortest path problem on a weighted graph, and not be familiar with Dijkstra.

And I'd say it's unheard of that you would know the first thing about the general, unit weight, problem, and not be familiar with BFS.

Concerning 12: timing attacks are not a relevant threat for hash function computations. See https://github.com/codahale/bcrypt-ruby/pull/43#issuecomment.... Supported by our own tptacek here https://twitter.com/tqbf/status/866115075110825984

Whoah. Careful! Timing attacks aren't relevant to password hashes, because the user doesn't control the hash, only the input to the hash, whose output is unpredictable. But cases like the TripleByte quiz, in which a user is directly submitting a hash to check (typically: an HMAC on a message) are totally relevant; in fact, they're the most common kind of legitimate timing attack.

Yes, sorry, my comment is sloppily phrased. I should have been specific. And then I may have realized I was wrong, because it wasn't about password hashes.

(It seems that tweet and the link were fresh in my memory: I automatically interpreted the question as being about password hashes, although the code actually does not imply that at all. And the linked thread is clear about the difference between password hashes and other hashes, but it seems the 'simple' message was more available to me than the deeper truth)

Question for ammon : What's the time frame for expanding to remote jobs?

Question for other HN readers : How much did you score on the quiz? I scored 10/15 and it shows that I'm better than 50% of engineers i.e I'm somewhere right in the middle. Not very encouraging, but if I brushed up on the graph algorithms, I could have done better.

The truth is that there's just a lot less demand for remote engineers, and among companies that do hire remote, it's usually for very specialized roles. (There are obviously exceptions.) Looking at our stats, about 5% of the companies we work with are interested in remote engineers. We're not going to expand to remote jobs until we have that number up to a point where we can provide a good experince. I don't have a time-frame, but not in the next 3 months (over the next 3 months we're focused on expanding to other cities)

Re the % scores, they're currently set based on the people we had test the quiz yesterday. May well not have been a representative group! We'll reset them later today.

Follow up question : In your faq, you mentioned that you do not interview candidates who need visa sponsorship. Any plans to change that part?

Unfortunately probably not :( I would love to, but the jobs are just not there. H1Bs are just too hard to get.

What I do hope to do is expand out of the US, and get around the issue that way.

I scored 15/15 (top 3%). I was fortunate enough to already know some Python and to take the quiz after the regex corrections.

I appreciated that the Python questions were not especially Python specific: the reference questions would have been exactly the same if translated 1:1 into Java, Javascript or C#, and none of them tried to trick you with subtle syntax, types or parameter order.

Bonus points for not having joke answers. They're never funny.

13/15, top 25%. My supremely embarrassing errors were both because of not thinking carefully about variable scope for questions 3 and 4. I think the questions were pretty fair even though only a few seemed much applicable to what I need to know or deal with day to day as a developer. I'd guess most people with undergraduate CS degrees could answer all 15 right out of school.

I scored top 3% at 14/15. I had experience with Python so that helped. Honestly I don't think I am in top 3%, since I don't work as sw engineer in real life.

I scored 14/15, messed up on the regular expression question. Good thing I remembered the graph algoritms.

Seems extremely language-semantics specific.

In the 3rd question, if you give the wrong answer, it says Java is pass-by-reference as well, though IIRC it is always pass-by-value. Also JS passes primitives by val as well if I'm not mistaken.

Well, objects in Java are only accessible via references. So the result is behavior identical to Python and JS (primitives are passed by value, objects are passed by reference)

Java is indeed pass-by-value, but it behaves like pass-by-reference when dealing with objects/arrays (because object references are passed by value).

> claims this is a generalist programming quiz

> first half focuses entirely on weird Python quirks.

Wew.

On the plus side, I still somehow managed to get 9 out of 15 right, which apparently puts me above 50% of all programmers.

Mildly annoyed at myself for getting question 8 wrong: the true answer was what I thought that code would be doing, but the bytes didn't look like valid UTF-8. I had a brainfart and thought that continuation characters were always valid ASCII, so a string containing only characters >= 0x80 wouldn't be valid.

The (much saner) reality is that continuation characters are never valid ASCII.

10 I admit I got through elimination: three answers were obviously wrong, one could have gone either way.

I'm not very familiar with Python but I liked this quiz. I got two wrong (13/15), once for a stupid mistake and once for not knowing the algorithms well enough. In a couple cases I didn't know the right answer, but was able to determine that all other answers were wrong, which worked.

Overall, I think this is a good level of knowledge that isn't too nitpicky but has enough meat to really test if someone knows something.

This was fun. It's been a while since I took a multiple-choice test, and it called to mind the Cluss Test <https://www.recompile.se/~teddy/cluss-test>. This test was fairly cluss-resistant, as I don't recall making any inferences from the test's format or language rather than its content.

14/15 - I missed the one where Python doesn't copy the row before appending it to the matrix.

This is a quiz about python gotchas.

I first found out about Ammon and Triplebyte through this YC video interview. I thought it was interesting, so I'll share:

https://www.youtube.com/watch?v=haxZDcTWdOo

Too many language specific aspects.

Reminded me why I dislike Python...

Where do the percentiles come from that are displayed at the end of the quiz? Are these based on a previous test sample group or are they being updated as more people take the quiz?

Is there a JS equivalent of something like this out there?

If people like this, we'll be creating more version (in different languages, and drilling into different areas)

Judging by other people's comments, this would be a good quiz if it was written in their favorite language.

The quiz doesn't seem too hard to port (ha, I've heard "it won't be hard to write" before). I think that a good approach might be to let the user see the code in multiple languages via tabs or an interactive select element, like the MSDN docs do.

What's the deal with the warp searching question? There are both depth and breadth first algorithms that can handle cycles. Bit annoying since the example I had in mind was Djikstra's but it turned out that was classified as a BFS instead of DFS like I thought it was. Finding the shortest path from A to D in 4 steps for example seemed more "depth" to me, but maybe the algorithm as a whole doesn't neatly fit into either.

I think you're misunderstanding the point of the question. BFS is guaranteed to find you a shortest path in terms of "# of hops" in an unweighted graph. DFS is not.

Is this intended to be taken with access to references and a python interpreter, or closed book?

Either way. It's just supposed to be fun / interesting

I only got 50 percentile which I thought wasn't bad for not knowing python

I would have done better had I known how python does parameter passing...

I took the Triplebyte quiz on Facebook and after finishing, they didn't even bother giving the score. It was very outputting, and felt like a trap, asking for an email address instead of giving the score. I won't ever be doing business with them.

Ah, you're talking about going through our regular process and looking for a job? In that case, we try to make it clear that what you're doing is applying for a job (and we require an account creation at the beginning of the process).

This quiz posted here is something we created because some people (like you?) are applying to our process just to answer the questions. So we created this version (which does not require an account creation or ask for an email) and totally gives you a score and shows what we think the answers are.

I took the 35 question quiz. At the end it told me I passed and I shoul wait for feedback on how to move next but after a day I got an email informing me I failed.

I like the idea, but my first reaction was that this is a test of how well you know Python's quirks (like the problems with not deep-copying things).

But then I realised that that's not true. For example, call-by-value and call-by-reference are pretty pervasive across languages: many of those examples would've worked the same in Java, say. It's not familiarity with Python, but rather familiarity with a certain kind of language (which admittedly constitutes the majority of those used in industry ATM) that the test demands, at a time when even Java is getting nice FP-like abstractions and the JavaScript community is cooking up all sorts of ways to bring the benefits of that kind of thinking to their own work (from React/Redux all the way to Ramda and the like).

If you're more used to languages that "get out of the way" by eliding what could very well be termed "implementation details" (e.g. is there any reason[0] to use call-by-reference in the absence of global mutable state given a "sufficiently smart optimizing compiler"?), I'd predict (n = 1) that you'll be more likely to make mistakes.

I would've done far better on this quiz back when I was playing with matplotlib as an eighth- or ninth-grader: apart from the token map implementation, the quiz really favors knowledge of counterintuitive behavior common in "mainstream" imperative languages over an understanding of "composability" that users of modern functional languages are used to. (And no, that doesn't only mean Haskell or "ultra-Haskells" like Idris, but also languages like OCaml, F#, and so on.)

It would be much nicer if the quiz could be less about the candidate coaxing her preferred meaning out of a language implementation and more about testing the meanings a candidate knows how to implement with the help of the language.

[0]: For instance, the standard reaction to things like

    sum (map (\x -> x ** 2) (map (\y -> sin (y * pi / 10)) [0..9]))

is that a single for-loop is better than this for "real-world purposes". That is very false: the Haskell vector package is usually capable of efficiently fusing the code above into exactly what one would've written by hand.

https://www.schoolofhaskell.com/user/commercial/content/vect...

At the risk of going all peak-HN, these things are more a problem with implicit, nonuniform behavior which well-designed languages solve with abstractions like the Copy trait in Rust or even C++ move semantics.

Failed two questions in the first part because I don't understand Python tree syntax and don't know what does it pass by reference and what by value.

Would be great if authors translated it to at least a couple of other languages.

Thanks a lot for that short quizz !

its a python quiz.

Someone please change the title to "15-question python programming quiz". Or python and js? I stopped at the 2nd python "trick" question.

Also, remove the "generalist" reference. Generalist those questions aren't.

A bit harsh to make a generalist judgement if you quit after two questions. I don't know any python and still did pretty well.

It was their 'generalist' statement that made me quit.

Expected to dislike it, but ended up liking it quite a bit.

I admire triplebyte's effort and the content they produce. Their narrative is really data-driven common sense: some engineers fit to some companies but not to others. Especially enjoyed: http://blog.triplebyte.com/who-y-combinator-companies-want

I am a programmer and I run http://www.coderfit.com, a tiny recruiting agency in Zurich / Switzerland (reach out to me via e-mail address in my HN-handle if you look for a tech job in Zurich). Recruiting is really, really hard and I am happy to see platforms like http://interviewing.io and http://triplebyte popping up that give programmers without classic CS background a chance.

Reply on news.ycombinator.com