Codility - Automated tests of programming skills | Hacker News Reader

Codility - Automated tests of programming skills

codility.com

71 pointsgrk95 comments

Very good idea, but I don't program that way. I use a competent IDE to tell me syntax errors and obvious programming mistakes while I write. If I can't debug my algorithm with trial & error I write a test first and modify my algorithm until it complies with the test. To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve. In my 10 years as a programmer I have never seen one wonder-kid that is able to do that. The real wonder kid assets in a team also tend to be those that are more creative than syntax-anal compared to those that can write syntax-error-free-code with pen and pencil and remember large parts of a phonebook...

Well, that's one more reason you should like Codility - you can run test compilations (the VERIFY button) which will show you syntax errors. And in bottom left corner you can apply your test data sets. In other words - you work like in your every day environment. Check it out and let me know what you think.

I did, and now I see comments stating that you should be able to run tests on your code while developing it. That's awesome and what I was missing. Next step intellisense and on-the-fly syntax checking. I hope no programmers click the "compile"-button nowadays to check if their code is correct...

To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve

I have no idea where you saw this being implied. The software highlights syntax and lets you check your code as many times as desired, as well as giving hints if your result isn't correct and providing compiler output. Nothing stops you from googling something. It's not like the software will destroy your file and egg your house because your algorithm is o(n2) instead of o(n); that's up to whoever is evaluating the results.

The fact that it has a polish and chinese version should also give you a little hint about the intended audience.

Me too.. I am totally ruined by Eclipse, so I depend on it to check my spelling/typos..

That said, as soon as the test popped up, I opened my eclipse, wrote the code, cut & pasted .. viola! (in python btw)

This is quite nice, even if the exercise isn't all that difficult.

It would be nice if the boundary conditions were better specified (people seem to have been bitten by overflowing integers). This could be inferred if you told people that you're using a 32-bit machine (the sum of elements is at most SIZE_MAX * INT_MAX; if int and size_t are 32-bit, SIZE_MAX * INT_MAX < INT64_MAX.)

I was also somewhat surprised to see that my code compiles without #include <stdint.h>. I'm not sure this is correct (it could be solved by putting the code written by the candidate before the test code).

Finally, I assume you have proper sandboxing set up?

We deliberately underspecify the compiler version, we think in such situation a programmer should attempt to write code as portable as possible.

#include --- good catch! We are not very strict about missing includes, but this is definitely not something we want to penalize.

Sandboxing -- absolutely! :-)

I think this is cute but I would think hard about the kinds of questions you develop for this. Coming from the perspective of someone working for a big company with a ton of software people that is not a tech company, I could use something like this in general. But, the question you asked is the kind of thing I ask CS undergrads (almost exactly, in fact) because it allows for clever solutions without actually requiring someone to be a particularly good programmer. I can work through it with them and if the basic solution is easy, prompt them to give me something more clever, so I know how they think, which is good when you're looking at an inexperienced developer. However, for an experienced developer that I want to hire to churn out Java code, the difference to me between a correct but simple O(n^2) solution vs a clever O(n) is not terribly meaningful for determining whether they can actually do what I want them to do. I want something that requires people to write more than a handful of lines and demonstrates an ability to keep track of a few more variables correctly, and honestly I don't care all that much about their algorithmic complexity (although I might care a great deal about their space usage, usage of external resources, etc). So, if this is the audience you are looking to attract, consider thinking about this as a screening tool for people outside of the startup world that actually need "knows how to really do shit in language X" more than "knows how to write clever algorithm,"

I don't think Codility checks whether one knows how to write clever algorithm. Look at tasks from ACM Collegiate Programming Competitions, they require clever algorithms. Codility deliberately focuses on very fundamental tasks. Also it does not pretend to be a final sign-off, it merely tells you which candidates to reject due to lack of rudimentary programming skills. It's not that surgeons make injections that often in their daily practice, yet I would not trust a surgeon who doesn't know how to make an injection.

I like this comment. A typical surgeon earns a bachelors degree, then attends medical school, before finally spending three to six years in a residency program (in the United States). This is before becoming a practicing surgeon. Unfortunately, there is nothing similar (in the United States) that is related to software engineering -- though many here would like to see something similar to an engineering license for technology professionals; there is nothing currently on the horizon (that I know of). In some ways, I think that Codility can help with this state of affairs. This is particularly interesting when we consider comparing metrics for education and experience; and perhaps 'real-world' success. The Codility statistical comparison, taking these factors into account, would be very interesting indeed. And could possibly result in data that furthers the state of the art in general.

{In Pascal Language+Assembly} {$ASMMODE INTEL} function equi (A : Array of longint; n : longint ) : longint; var c:Longint; label noOverflow1; label noOverflow2; label ciclo; label fine; label Over; label tot; Begin Asm DEC n JS over XOR ECX, ECX {Somma1} XOR EDI, EDI {Somma2} XOR EAX, EAX MOV c, EDI MOV ESI, n tot: MOV EDX, A MOV EDX, [EDX+ESI*4] PUSH EDX ADD ECX, EDX JNO nooverflow1 ADD c, ECX nooverflow1: DEC ESI JNS tot; SUB ECX, c SUB EDI, c ciclo: POP EDX SUB ECX, EDX CMP ECX, EDI JE fine ADD EDI, EDX JNO nooverflow2 DEC EDI nooverflow2: CMP EAX, n JA over INC EAX JMP ciclo over: MOV EAX, -1 fine: end; End;

lol I got 101 on the test.

This is interesting. It might be worth noting somewhere how many bits is an Int on your test rigs.

I got 94 using C in 12min (didn't check for overflows of large numbers, silly haskell making me used to Integers).

Generally we don't want to penalize people too badly for assuming particular (but realistic) type ranges, nevertheless we think that perfect solution should be as portable as possible across different implementations of the language (e.g. one should not assume that C's int is 32 bits wide, since implementation with 16-bit int can still be Standard compliant)

That is an amazing piece of software. I hope you make a mint.

I'd give people access to their language's output facility for playing around to the solution, but that is a minor nitpick. (And I suppose I could open an irb instance on my machine for exploration and then give you code when I'm good and ready.)

My Ruby solution, in case we've got anybody who is struggling: http://pastie.org/771438 This would be my first cut if you asked me to whiteboard this.

Pasties are a good idea, but results pages are more interesting. Here's my sad effort after 8 mins in python: http://codility.com/demo/results/?id=demoXFXD64-GCZ

The site's very slick. My only feedback is the histogram on the results page could be more prominent - it's cool, useful information but I ended up inspecting the html elements to work out what it was.

Also the results page seemed to keep refreshing for me while waiting for results (Safari 3). Surely that should've been AJAX'd?

regarding your python code: do you know about 'enumerate'? It returns an iterable of (index, value) pairs of a sequence. :)

enumerate is very useful, here's mine:

    def equi ( A ):
        right = sum(A)
        left = 0
        for i,x in enumerate(A):
            right -= x
            if right == left:
                return i
            left += x
        return -1

What's interesting is that all the versions I've seen so far are pretty much identical apart from very minor things. This would seem to me to be obviously the correct way to do it but I'm interested if it's just a way. Did anyone solve this in a different way of equal or greater elegance?

Mine kept one cumulative list adding left to right and another summing right to left. This is conventionally slower, but conceivably faster if you do it in parallel, leaving the final loop to be something like:

    for i in xrange(len(A)):
        if right[i] == left[i]:
            return i

The final loop here is slightly tighter, so as long as your parallelism didn't add big overheads, this will be faster.

Original: http://pastebin.com/m6e742f56 Parallelised: http://pastebin.com/m32a60b1

+1! Thanks for the reminder. I'm fairly new to python and couldn't remember the keyword. I guess this is a reasonable argument for the time deadline - to get it done quickly I didn't look it up, instead doing the simplest thing I knew would work and thus exposing the pitiful state of my python experience :-)

I like how the performance evaluation seems to be fairly realistic - I used Ruby and my algorithm could have been more efficient (O(n) space, and O(n) time but with more traversals than necessary), but I scored 100 because I met all the stated constraints. (http://codility.com/demo/results/?id=demo8XKE77-H8G if anyone wants to refer to and/or mock my code.) It seems to me "ability to solve problems well enough under time pressure" is exactly what you want a simple hands-off test like this to screen for.

In face-to-face interviews you'd probably follow up by adding constraints: "your algorithm uses O(n) space in addition to the input, can you do it in constant space?". It would be cool if Codility supported that kind of workflow - following up the submitted solution with refinements on the original question. (Relatedly - do you support the workflow where you get a second try if you didn't get 100, maybe because the evalution revealed a constraint that wasn't clearly stated in the problem outline?)

(If you are looking at my solution and wondering what that monkeypatched Array#sum is doing there - it's a red herring, abandoned early on.)

You can assign up to 6 tasks per each test, to minimize the 'constraint omission' risk. Does it solve the issue you raised?

With paid account it is possible to reopen a task after one solution has been submitted, so the recruiter can "give another chance" to the candidate (although we generally suggest not to), which supports the workflow you describe.

Is the code editor they're using something publicly available? I'm looking around for a light in-browser code editor with programmer-oriented features (e.g. syntax highlighting, this also seems to auto-indent Python code to some degree).

bespin?

https://bespin.mozilla.com/

Edit: just noticed that you'll have to wait a few days to get an embedded release that has syntax highlighting ( http://groups.google.com/group/bespin/browse_thread/thread/0... )

Try http://www.amyeditor.com/ (demo). You can find the sources at http://github.com/aprilchild/aprilchild/tree/master/amy_edit... . Supports a bunch of programming languages, syntax highlighting, interactive interface, indenting, projects and colaboration, ect. The only thing it does not do is compiling.

Even I noticed that I had seen something ditto somewhere... Hmm so Codility's editor is copy of the one SPOJ uses, so which one do they both use? Can I too get it :)

http://twitpic.com/x6wkq and even the code matches line by line I guess - http://twitpic.com/x6wnp

Codility uses open-source component EditArea.

> Hmm so Codility's editor is copy of the one SPOJ uses

Please don't flame. Codility has not copied it from anybody, its EditArea-based interface is in place since 2008.

I beg to edit the word "copy", but I can't do it now.

Hmm so Codility' and SPOJ use a similar editor, so which one do they both use? Can I too get it :)

Thanks for pointing out. /me looks into EditArea

It's EditArea

http://www.cdolivet.com/index.php?page=editArea

very nice project on LPGL license.

Thanks for the positive feedback. Do not hesitate to ask us any questions.

best Marcin Codility

Nicely done overall.

You might want to lock down functions like glob() and readfile() in PHP -- they effectively give the coder read access to your file system.

It's blocked elsewhere in the system, so no worries :)

Good point, thanks! But no worries, this is not our real filesystem :-)

You should run a programming competition using this software.

We're thinking about it. Stay tuned on our blog :)

If I have an account and am taking a test that I've created, you should provide a link back to the dashboard after I've submitted so I can check how I did. Currently, I'm taken to the the "enter a password" screen and have to manually go back to the homepage and then to the dashboard.

That's because the recruiter and the candidate are supposed to be two different people, so the candidate doesn't have the access to the 'dashboard'.

It gave me a test concerning finding the "equilibrium index" of a sequence. It said to assume that the sequence is very long. (The question is repeated here: http://forums.sun.com/thread.jspa?threadID=676596&start=...) This problem seems underspecified, because it doesn't state whether you can iterate repeatedly over the sequence in question. If you can, it's easy. If you can't I'm not sure how to do it.

You can do whatever you want - it just has to return the correct result.

It isn't that difficult to solve, but I like the idea.

iterate repeatedly over the sequence in question

Does that ever seem like a good idea? The description suggests it may be a long list. That should give you a hint that you don't need to do this to solve the problem.

I'm pretty sure you have to iterate over it at least twice. I guess the specification is implicit in that.

The best case possible is to iterate the list once - you sum the list from left to right holding cumulative answers and if the final sum is zero, that index is a valid answer.

The worst case of the best algorithm is to iterate twice - firstly as above, and then again in the opposite direction, comparing values from the first iteration.

I assumed that "sequence is very long" meant "the time (and space?) complexity of your solution matters". Iterating over the sequence x times doesn't change the time complexity, so long as x << n (i.e. it's O(n) whether x is 1, 2 or 3).

Unfortunately, revealing whether my assumption proved correct would give away how they assess the solution to this demo problem, which some might view as a spoiler.

EDIT: since people are posting actual solutions, I suppose mere analysis of the testing methodology is okay. My assumption was correct.

Really nice site. Well done. The example problem that showed up to me (equilibrium point detection) was not trivial, but didn't require advanced stuff which a halfway decent programmer wouldn't use day in and day out.

I also like the way you tell me what you scored on and what my result was.

I wonder if you should relax the condition that you can submit only once.

I gather TopCoder offers something similar. What's your strategy to compete with them?

PS: got 94 using C++ in 16 min b/c I didn't check overflow of large numbers. Good thing you check for that.

Right now TopCoder is focused on coding competitions. We are focused solely on recruitment - those are pretty separate markets at the moment.

You know, now that I think about it...The description of the test was pretty incoherent. If I don't know what my test data is, and if the problem description is incoherent, I can't really make it work.

It's also taking a metric fuckton of time to evaluate my solution. Which apparently fails. For reasons I can't debug because I can't see the test data.

Any company that relies on code snippets, incomplete problem statements and undefined test data to judge coders will get what they deserve.

maybe there's more than one task, but the task i took appeared to be completely specified (the one where you find a balance point - i may have remembered the name wrong - for an array, so that the sum of elements to either side is equal).

it wasn't obvious to me what to do on first read, but when i paid a little more attention, everything was there (including edge cases, as far as i could see). in particular, they gave a formal numerical expression that was unambiguous (and which didn't seem to contradict the text).

Yep, it's good to read the specs in detail ;)

The spec was clear, but the fact that the test dataset used for the test was the very one given as an example wasn't. plus, it was a bit frustrating not being able to fiddle with the test dataset. I would have done it locally with my compiler, but the necessary boilerplate would have lost me time. So, I didn't test enough, and made a silly error which went undetected because of the trailing zero. I still got 75%, but that looks quite miraculous.

EDIT: It appears I can customize my test dataset, and I didn't see it. Silly me.

Yes, it is even possible to add your own data sets, Codility will tell you what you code returns for them. I guess we should make it more apparent in the interface.

5 minute version of the demo problem in Ruby - 100%:

http://codility.com/demo/results/?id=demoMPC2W9-JDG

As a student this is very interesting since it could also be used to measure self progress. There's a myriad of ways to test your abilities in most subjects, but if I want to find out if I'm any good at programming or to measure my progress, there are very few easy ways to do that. Are you going to sell this to just business's?

Have you tried topcoder? It's the same as codility, but with cooler/harder problems.

Thanks for the tip, topcoder looks pretty interesting in general, too.

Also Project Euler might be worth a look. It focusses on solving math-related problems with progams.

Codility tech team is from Warsaw University, we are more than familiar with TopCoder, ACM CPC, IOI etc. We deliberately keep Codility tasks at fundamental level. Codility is not a competition for 100 world's best programmers, it is a tool to tell decent programmers from poor ones. We can raise the bar arbitrarily high, if there is a customer who needs it.

If an edu focused business case emerges, we will definitely look into it.

For C++ it would be nice to know what compiler is being checked against. At least in the demo question I would use a 64 bit int to hold the sum if the arrays are really "large" but MSVC names the type __int64 vs G++ using long long and other such minor details. Same for the other languages aka python 2.x or 3.x etc.

We want to avoid sticking to particular compilers/interpreters. An expert in MSVC should not be penalized only because we use GCC. Also we design the tasks so that they are compiler-agnostic (actually they are even language-agnostic). It is sometimes a bit tricky due to non-standardized type sizes (e.g. C, C++, PHP), but generally the tasks can be solved using very limited and portable subset of a language.

Got my 94% C solution up to 100% by using long long ints for intermediate sums.

http://pastebin.com/m6e742f56

Edit: And won't this be a problem if interviewees can access the internet? The interviewers will have to make sure they can check the browsing history of the computer being used!

Why? After all, you are going to use the internet while on the job too. Just making the problems more difficult should fix that problem.

I don't mean the interviewers should shut off internet access entirely, but what I mean is that if they do allow access to the internet, they'll need to check the interviewee hasn't just googled for the solution to their codibility test (and they'll also have to make sure the interviewee can't delete their browsing history too).

Don't worry, we have some nasty surprises for the cheaters ;)

Codility people, if you're reading, here's a bug report.

The test window's layout is fubar if you use a small window size. I was using 800x600 and the language selection buttons were hidden behind the source code text box. Something (couldn't tell what) overlapped the first two lines of the instructions.

Firefox 3.5.7/Ubuntu Linux 4.10, FWIW.

thx! we will look into that! Chris

I'm not sure the timer is a good idea. For one, I found the seconds counting down very distracting. It also encourages quick and inefficient solutions that are easy to hack out, over more efficient algorithms that may take longer to code.

Maybe you should time the programs, instead of the programmers.

We are timing the programs as well - with the scalability assessment. Recruiter can change the timing anyway, but we feel that a bit of pressure is good. You don't always have comfy conditions on the job after all.

Right, but maybe not down to the second. I personally didn't mind, but counting seconds causes a distracting flickering. Maybe you should consider activating it on the last minute only ?

Hm, would something more smooth (analog?) be better?

I also found the clock annoying. If someone does this for an interview they're likely to feel enough pressure anyways. I'd let the test taker hide it and maybe redisplay it automatically with 5 minutes to go.

As for comfy conditions on the job - I would hope that default conditions are comfy. When interviewing I'm more interested in how people will perform day-to-day rather than in rare situations. Most programming jobs don't really require employees to be "cool under pressure."

Other than that, good job on the execution so far. Reminds me a lot of TopCoder, just applied to a specific niche in a good way. I would even consider using this or recommending it to colleagues for hiring.

I agree the clock puts extra pressure that can make one unable to focus on the task, but it's important to know your time limit.

Consider making the clock like on http://www.sudokuslam.com/ - with an icon right next to it to show or hide it.

Might need some way of making explicit which libraries are available?

eg. if the programmer goes off and writes a multithreaded solution, they might only realise too late that pthread isn't included. Admittedly they should have checked first using verify, but you never know with interview pressure.

Unless it is stated explicitly, we do not provide anything extra but the vanilla programming language and standard libraries. It is more clear for some languages (e.g. Java, C++) than other (e.g. PHP, JavaScript), but we try to stick to the reasonable canonical set of libraries.

Seems like a cool idea. If they were smart they'd not only target companies looking for people but people looking for jobs. e.g. Adding a "Allow the codility community to take this test" check box. Then you have the a whole world full of people taking your test.

Actually it's there - if you click on "Take a free demo test" (http://codility.com/demo/take-sample-test/) on the Codility page there is a free test and you will be given the option to tweet your result at the end.

Wow, beautiful design.

That said: Jesus I must still suck, or maybe Math simply isn't my strong suite.

Yeah, that demo problem would probably take me at least several hours to solve. I'm bad with sequences and I have a volatile short term memory which means I make mistakes and am slow to process numbers and logic. Interestingly these flaws cause me abhor any over complex logic or code where you have to remember more than 4 or 5 things at once to use properly, which makes my code clean. So maybe I'm not a good programmer (I can't write the complex stuff very easily) but the code I write is usually good. Try testing for that Codility!

BTW 10-20% of the population is like me.

Very nice product. Much-needed quantification in tech recruiting. Maybe consider marketing it to the stackoverflow people. They have been running a job board with much fanfare since recently. (94 points for me in 7 mins, yay)

Great. So now we're going to be hearing about TDH zealots (Test Driven Hiring).

I've been a TDH zealot, as you put it, since 1986. It's amazing how hard it is to evaluate coding ability in an interview without it.

I readily concede that coding ability isn't the only thing a dev needs, but it is absolutely required for most jobs.

For the record, I was kidding. :-)

Just a heads up:

I took a demo quiz and it worked fine. Great idea for a product.

However I think I'm too competent for the test because I was able to run something I probably shouldn't have been able too.

Yep, we've seen it... :-) Thanks for pointing this out!

This would be kind of cool simply as a brainteaser site with problems taken from Project Euler.

This is great!

I've already sent it around at the company I work for. I'm hoping we can put it to use.

also thought this was nice after trying it out.

one thing i missed, though, was the ability to write and run my own tests. perhaps a "scratchpad" that evaluates code and returns a result would be a nice addition?

Bottom left corner. It's obviously not visible enough :)

I just can't see it. Is it in the demo test?

My bottom left corner is empty. Except when I click "verify" it shows message whether the tests passed or failed.

But I don't see the tests themselves, nor did I notice a way of writing my own. I've tried it in Firefox, IE 8 and latest dev Chrome.

So, is it possible to write my own tests when I'm taking the test, or only when I'm creating a new one?

Sorry, some bug on our end - it should be there. We are fixing it right now. In the meantim - if you sign up for a trial account [http://codility.com/accounts/codility-register/] it works fine there (a blue "add a test case" button in the bottom left).

Great :-). I feared I was too stupid to see it.

Anyway, I like it a lot. It sure beats writing tests in front of a blackboard.

thanks. another smaller point i just remembered - i was a bit surprised to see that the evaluator could see my submissions to the "validate" function. given that you do say they won't influence the score i assumed that they would be discarded.

i can see there are several reasons why this is useful to the evaluator, and i can also see that you're getting close to saturating the user with instructions, but in a perfect world it would be good to have "full disclosure" (i guess you do, in that anyone can run the example....)

Good point, thanks for the feedback.

Very cool product. Well done

Reply on news.ycombinator.com