Hacker News Reader

Comment by harias | Hacker News Reader

> Once the students were selected, the researchers then administered the Major Field Test in Computer Science, an exam that was developed by the U.S. Educational Testing Service and is regularly updated.

There is a huge variation among countries in how subjects are taught and tested. I would like to add that many of my peers have resorted to streamlining for whiteboard interviews, and unsurprisingly, they end up with well-paying jobs.

View on HN

I took a look at the sample questions for the test, having never heard of it. It's 66 multiple choice questions, and many of them seem like “gotcha” questions where one of the answers is an off-by-one error or something like that.

And how about this lovely question:

> A personal identification number that opens a certain lock consists of a sequence of 3 DIFFERENT digits from 0 though 9, inclusive. How many possible PINs are there?

Putting aside the value of this question in the first place, the correct answer is 720 but it also includes the answer 1000 in case you didn't notice the emphasis on “different”.

Doesn't seem like a great test to me unless it's about your reading comprehension and ability to decipher code with single-letter variable names.

Funny. Yeah I totally would have missed the word "different"... does such a lock even exist in the real world?

I would prefer it be more explicit and say, "each digit can only be used once".

Different can be interpreted in multiple ways.

Reminds me of a physics test where the question is something like:

Susie is almost at her house and comes to a stop in 4 seconds. What was her acceleration.

The only other interpretation that is obvious to me is "not all of the digits can be the same", but 990 is not an available answer.

Yes, such locks exist, and are actually sold in the real world. They are often used in "outdoor key storage", which will find you images of what they look like. Though I can't guarantee any particular model has this particular PIN flaw (some are even worse, such as only allowing a single number from each row).

Which is a limitation from the mechanics of the mechanism used. Including such a restriction on an electronic PIN is stupid, but I've seen enough idiotic password policies that I don't doubt someone has done it. For example, my bank (at least 2 months ago, haven't tested recently) makes no distinction between case of letters, so 'aaa' = 'aAa' = 'AAA'.

It's considerably worse when you see bizarre password restrictions because they usually mean the password is stored in plaintext. For example, your bank definitely is doing that.

Don't use that same password for anything else!

Over the phone with my bank and my cabal company I've had them ask me multiple times to confirm my PIN or give the answer to my security question. I'm shocked that they will store everything in plain text like that AND give a large number of their employees access to it.

Well, my bank tells me that I can't have 3 equal digits in my debit card PIN.

The android lock screen, for example.

That question first and foremost tests English reading abilities. Many mathematically gifted non-native speakers (and indeed native speakers) would miss-understand the "different".

"The exam was translated for the students in China and Russia."

Would love to see results compared to a Chinese or Russian-first test later translated to English.

I wonder if the translator fully understood the questions.

There is no reason to doubt the translator just because you don't like the study's result.

No, but there might be reason to doubt the translation on the basis that the intersection of the skillsets "fluent in Russian" and "total command of the relevant computer science material" might not have many people.

For a fair comparison, the translator would have to fully appreciate the relevant connotation of every significant word like "different" in every question, and successfully convey the same in Russian. You couldn't just go to a regular translation service and get a good result.

A lot of computer-related words in Russian are borrowed with small modifications from English (or from where English took them). Nobody says "electronnaya vychislitel'naya mashina" anymore, they say "computer". Same for file, register, class, object, compiler (kompilyator), algorithm... It happens that tutorials for some computer-related subjects - especially some niche ones - are available only in English. It sometimes help that programming languages use English language base for keywords, so there is less confusion with Russian words.

Doubting that the translation caused some discrepancy isn't unreasonable. Technical and academic works are notoriously difficult to translate, and it's entirely possible that even though "different" was stressed, certain languages lost the distinction, whereas if the word had been "unique" the distinction would have remained.

Although I have no faith in a multiple choice test as a predictor of CS knowledge anyway. That is just a horrible test format anyway.

The funny thing is the question is actively harmful to real world problem solving. If I had to write a feature where passwords could not repeat any character the spec would spend a lot more than 1 word explaining that (and why there is that weird restriction in the first place). Assuming they wanted such a system based upon a single word would be a bizarre conclusion to jump to.

Yeah if this were a Software Engineering test the correct answer to the question would be: ”First I would ask the client why they needed this. If they didn't have a good answer I would advise them not to do it. If they still want it anyway: 720.”

Personally I really dislike multiple choice question format, as they often end in "gotcha" type of questions or overthinking of the question. In the exams of my university (for computer science) they are rare (most exams have at most 1/10 as multiple choice).

Does anyone have an assessment of reliability/validity/fairness of multiple choice comparing to other test types? (afair objectivity is very good in multiple choice tests)

Asking "how many 3 digit numbers are there where each digit of the number only appears once?" is nearly a mathematically identical problem but without the fake backstory.

Being able to go from some constructed story to a math problem is a skill that can be tested elsewhere (and really, should be assumed if someone clearly has proficient enough numeracy to do combinatorics) Questions should be testing one thing as isolated as possible, not the same things repeatedly.

Transforming the backstory into math is the important skill that it's all about, and no, you can't assume that a student who knows how to manipulate numbers knows how to apply them.

I can see this in my students in my introductory class. You can ask them to calculate a concentration of a solution and they will reply just fine. But when you ask them to go into the lab and actually prepare a solution of a certain concentration you draw blanks in 9 cases out of 10. Students are trained in AP Chemistry to spot problems according to keywords and apply the appropiate formula, but they have never thought about the underlying meaning of the things they do, and they cannot connect different subjects to save their lives.

Those are dramatically different skills.

The mechanical kinesthetics of combining things is not the same skillset as the mathematical process of computing it. Lab work is its own craft.

Based on what you're saying, I'm assuming you already put your questions in the "Story time" format and are getting that 1/10 success rate because the story problems are yet a Third skill - this time one about reading comprehension and language mastery.

Reading comprehension isn't kinesthetics and that's not computation. One may partially inform the other, but each skill is fairly independent.

i am lost. how can a sequence of 3 digits be 4 digits long?

Where does it say it has to be 4 digits long? Most PINs are I guess, but you could have 3 digit pins just as easily.

They're saying that 1000 was also a choice on the multiple choice test

i can't read. thank you

He's saying there are 1000 possible PINs (000-999) if you don't have the unique digit constraint.

It asked how many possible choices are there, 000 through 999 is 1000 choices, just as there are 10 numbers between 0 and 9 inclusive. (Yes, the Q also said each digit must be unique, hence the real answer is 720).

Anyway - IMO this is a terrible question.

Isn't this just the sort of stuff students do in a first year discrete math class? This question is hardly scandalous.

I agree. Looking at the same sample questions, these are not difficult questions if you know the relevant field of study.

The sample questions cover amortization, memory allocation, finite state machines, formal logic... If you just studied enough to pass a whiteboard interview, I'd hazard that you probably can't answer half of these questions without looking up what the symbols mean.

Sure, most people here may be able to figure out the PIN question, but it's not about whether any single question is easy or hard, but whether the test can sort the students according to ability. A good student may misread one or two, but a bad student is going to miss a lot more.

There is a general description of some nomenclature and symbols in the appendix which I assume the student get.

What this really measures is corricilum. And pressumbly it's more stuff like this in the US then Russia?

I didn't mean to say this question is scandalous. It is a needlessly confusing question. No PIN I've ever heard of requires unique digits, and the question does not actually have anything to do with PINs.

If a CS student taking this test does not get “tricked” by this question, they should definitely be able to answer it.

Thinking of the problem as a nested for loop it's intuitive to see how you get 1000 combinations without the numbers being unique but I dont see how to get 720 combinations as the answer when each digit is different. Can you explain where 720 combinations comes from in the context I mentioned?

10 * 9 * 8

The first time you pick, you have 10 different numbers to choose from, the second time, 9, the third time 8.

Or if you are like me (and a bit stupid):

1000 = all possible combinations from 000 to 999.

less 10 combinations with all numbers identical is 990.

now there are three patterns for numbers to be identical left that need to be taken out: 00x 0x0 x00. The x can be replaced with all digits != 0 in this example, so thats 3 (patterns) x 9 ('x' digits) = 27. That times 10 digits that can have double patterns is 270.

1000 - 10 - 270 = 720.

Hmmm. I wouldn't have approached it that way. Thanks for sharing! Lots of ways to approach any problem.

for i = 1-10 for j = 1-10 if i == j break for k = 1-10 if i == j or i == k break log(i,j,k)

for i = 1-10

for j = 1-10

if i == j break

for k = 1-10

if i == j or i == k break

log(i,j,k)

(I don't understand HN formatting still)

    #!/usr/bin/python
    r = [(i,j,k) for i in range(10) for j in range(10) for k in range(10) if i!=j and i!=k and j!=k]
    print(len(r))

The questions ask for all permutations of length 3. What you've described is the Cartesian product:

    import itertools

    print(len([x for x in itertools.product(range(10), repeat=3)])) #1000
    print(len([x for x in itertools.permutations(range(10), 3)])) #720

itertools.product essentially does a nested for loop.

The reason it's less is because each number has to be different, so you can't have 000 or 111, etc.

Permutation without repetition.

The formula is:

n! / (n − r)!

Since there is 10 (n=10) total numbers, 0-9. And it can only be 3 (r=3) digits long. Then it would be,

10! / (10-3)!* which would result in,

10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 1 / (7 * 6 * 5 * 4 * 3 * 2 * 1) = 10 * 9 * 8 = 720.

I'd like to highlight this comment for everyone who is saying the question can only be gotten wrong by someone who misreads it.

It can be easy for people to forget that many people (even coders) have never done this type of math before.

I believe CS degrees require discreet math (and highschool algebra 2), which means they've done permutations and combinatorics, so yes they've done these types of problems before. The real question is how long has it been?

> Doesn't seem like a great test to me unless it's about your reading comprehension and ability to decipher code with single-letter variable names.

Really? I mean, to first approximation, all software engineering work is debugging. And to first approximation, all debugging is about carefully reading results and the code that generated them. That sounds pretty spot-on to my eyes.

And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code.

I mean, obviously people do rely on those things for understanding, so when writing code it's important to have good semantically and contextually clear names for stuff (which certainly can include single letter names for local symbols repeated often!). But for debugging, you really have to train yourself to read the code, not the names.

This is a Computer Science test, not a Software Engineering test. I think it's even worse as a test for the latter.

I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points.

In my experience most debugging presents as trial-and-error (a terrible test-taking strategy) or seeing a bug and reasoning at a high level which component of the software is responsible and gradually narrowing it down. Carefully reading through lines of code is usually the last step, unless I've got an especially elusive bug.

> And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code.

I'm not sure what sorts of code you're reading but this opinion seems extreme. Do you really just wholly ignore comments and variable names? I have encountered inaccurate examples of both but I have never thought to distrust them by default. As far as I recall this has not caused me serious issues while debugging.

> I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points.

I don't think I agree. You're talking about heuristics for debugging code and I'd argue that many of us apply similar heuristics to reading comprehension. Computer Science is not the same thing as reading comprehension, but I have to argue that if you're really good at one of these; you're probably potentially good at the other.

Totally agree with what you say about multiple choice programming tests. I have a PhD in computer science and have worked as a programmer for over a decade. For fun, I took a practice AP CS exam with my son, which had similar multiple choice questions.

I did not get 100%. I actually thought the exam was pretty good, but it tested something like 1% of the skills I use as a working programmer. Skills like API design, software architecture, documenting code, and yes even naming variables, which I think are important to writing quality code.

Maybe that's because computer science is not software development?

> A personal identification number that opens a certain lock consists of a sequence of 3 DIFFERENT digits from 0 though 9, inclusive. How many possible PINs are there?

Do people really struggle with problems like these?

Students from U.S. educational system are doing beter in an

>exam that was developed by the U.S. Educational Testing Service

Who could have guessed that?

"We've evaluated ourselves and concluded that we are superior."

> many of my peers have resorted to streamlining for whiteboard interviews, and unsurprisingly, they end up with well-paying jobs

I switched into programming and am low paid, and am doing this right now for interviews. Practicing for whiteboard interviews, or coding tests (real time online and in person).

I re-read the language docs and then did (so far) 20 or so self-timed programming tests. Most of them I did in 5-30 minutes, 1 or 2 I got stuck on and subsequently reviewed. After finishing reading the language docs I will read though the platform docs, and may do some quizzes on that as well. As well as whiteboard high level class creation tests (HAS-A, IS-A).

The tests have a lot of array stuff, String manipulation, using long instead of int, casting from one type to another, doing a lot of stuff with maps and sorting maps by value. It also uses things then can be done quickly and compactly in streams. I do these things once in a while, but now have it memorized and down cold.

Hopefully this all leads to the latter result you mentioned.

Interview experience is just as valuable as the preparation in my opinion. I recommend you start getting some interviews at companies lower on your list and get some of that practice.

Also, what do people think about triplebyte for this?

There is also huge variation within countries. The curriculum is basically up to each college or university in the US. I never took this test.

I would assume ABET certified programs would have certain standards or requirements regarding he schools’ curriculums.

There are 5 Universities in India with ABET certified courses. None rank among the top 30 in the nation.

I’m not sure how much value I would ascribe to a certification that is held by DeVry University, but not the computer science departments of Stanford, Yale, or Princeton.

This annoys me to no end, since ABET is used to gate certain federal jobs and the Patent Bar, even in Computer Science.

The purpose of ABET accreditation is to ensure that students are prepared to take the fundamentals of engineering exam. Hardly any CS grads take the exam because it doesn't help you get a job. So it's not a big deal that their CS departments aren't accredited.

However, every school you listed is ABET accredited in other engineering disciplines (mechanical, chemical, civil, electrical). So ABET-accreditation is important to some fields.

It's not a huge deal, but it's not entirely trivial either.

I don't want to be too elitist, but where would you like to recruit reviewers for your computer-related patents from: DeVry Tech or Stanford? Do you really want to actively exclude Princeton and Yale grads from working on software that runs the government?

I get the appeal of having a standardized 'seal of approval', but for Computer Science, ABET is clearly not that. It looks like several other disciplines are also moving away from ABET (e.g., https://cen.acs.org/articles/95/i48/time-leave-behind-chemic...)

I've never seen a government requirement for ABET accreditation for CS degrees.

From the eligibility requirements for the patent agent exam:

Acceptable Computer Science degrees must be accredited by the Computer Science Accreditation Commission (CSAC) of the Computing Sciences Accreditation Board (CSAB), or by the Computing Accreditation Commission (CAC) of the Accreditation Board for Engineering and Technology (ABET), on or before the date the degree was awarded. Computer science degrees that are accredited may be found on the Internet (http://www.abet.org).

[page 6 of this pdf: https://www.uspto.gov/sites/default/files/documents/OED_GRB....]

A lot of fed jobs have engineering-centric requirements. This one, for example, wants an accredited degree or experience under a PE....to do network management. https://www.usajobs.gov/GetJob/ViewDetails/525193800

Typically CS programs offer a degree track that is ABET accredited, and a few other different options for students with different needs or desires.

> streamlining for whiteboard interviews

Can you clarify what this means?

They spend all their free time on interview prep websites. It is very easy to pass courses in India (except a select few colleges), so they spend next to no time on courses like computer architecture, networking etc and focus entirely on DS and Algorithms. It is quite effective though.

Reply on news.ycombinator.com