cspa

We view this closer to CFA, where it's not required (I don't think we'd ever get to this point anyway), but we want a world where many employers do recognize CSPA.

> Software Engineering is good right now because anyone, absolutely anyone, can dive in.

Yes, I wholeheartedly agree! CSPA actually bolsters this because there are no eligibility requirements to take the test. We believe there are a ton of great engineers WITHOUT a college degree and we want to surface them.

We're making it MORE accessible, not less. There's no real barrier to taking the CSPA, except the time commitment. Just register and take it.

Our initial goal is to replace the technical phone screen. We are asking companies to guarantee an interview (or skip the phone screen), if they score above a certain threshold.

Long term, we hope to replace as much of the onsite as possible. But we do acknowledge we can never fully replace it.

Thanks for the valuable feedback!

Re: GAN, we tried including ML in the Core exam, but have actually since removed ML since it wasn't working. We may introduce this as part of an ML specific subject test.

If you want to get involved designing questions, please consider applying to the TSC for next term: https://cspa.io/tsc/apply.

Thanks for the feedback! We try to have a good mix of trivial, novice, intermediate, and advanced questions.

Some of the multiple choice questions are fact-based and just require a Google search -- this is intentional, since proper Googling skills are still needed for the job :)

Problem solving questions are better formatted as open-response essay questions, which we do have.

Thanks for the tip! Can you elaborate? Do you mean teaching to the test, or something else? There's a whole research field for large scale assessment and evaluation methodologies.

Our third party proctoring service handles the KYC, as well as the LogMeIn. We did some vetting, and they are used by some well-known MOOCs and universities' online courses. It's a concern of ours as well, which is why we try to offer in-person proctored tests or appointments.

Currently the Core Exam is 50/50. It's designed for career-readiness. Our Technical Steering Committee is composed of majority of hiring managers, so it's intended to be useful for them -- that is, what are the qualifications desired in an entry level software engineer?

We are debating having a Core general exam, and separate per-subject tests, similar to SAT I vs SAT II.

We've found that that is indeed true. The more senior and knowledgeable you are, the most likely the answer is "it depends", because all the nuances and edge cases you know about.

We try to solve this with the standard "choose the BEST answer". We also do post-mortems and analyze the responses to disqualify or discount bad questions.

These are issues the CSPA Technical Steering Committee, which is responsible for the exam content, will deal with, in collaboration with our research advisor: https://cspa.io/about/team

Thanks!

(1) When we retire old exam questions, we will make them available for practice. We're working on creating more study materials! I can't promise anything before April though.

(2) Ah the 440 FE is a typo. We updated the scoring algorithm a few months back and didn't update those examples. Most people try completing all 6 subject sections. It's up to you, but we do discourage optimizing this kind of test taking strategy, and "teaching to the test" ;P

It sounds like this happened after I left Crunchyroll in 2015. The process I had in place was definitely not like that :/

Well FWIW, I personally wrote the coding exercise that you enjoyed (http://www.crunchyroll.com/tech-challenge/roaming-math/yourn...). I thought it was a good mix of basic algorithms (tree traversal) as well as some practical knowledge (CURL/http). I'm surprised they are still using the same one :)

That's a bug, thanks for reporting it. We don't have it in our bug tracker -- I don't recall CSPA having a Show HN post o_O.

Yep I agree! In fact, about 50% of the people who take the CSPA, do it more for self-evaluation than for employment purposes.

You're right, so far most of the test takers are entry-level or changing careers. To accurately assess specialities like ops/IT, we'll need separate, dedicated subject tests (or adaptive tests).

That said, no one has yet gotten a perfect 1600 :) . The highest is something like 1380.

Re: custom modules. One (arguably) interesting thing we do is to only take the MAXIMUM subscore of all 7 topics and use that component in the composite score, ignoring the other 6 topics. This allows candidates to focus on only certain topics if they want. Employers want someone who is strong in one area, rather than a jack-of-all trades and master of none.

Adaptive testing (https://en.wikipedia.org/wiki/Computerized_adaptive_testing) is something we're looking into, and would be extremely interesting (and challenging!) to build.

Great feedback! We do have a distribution of scores for our latest exam session here: https://medium.com/@cspa_exam/july-2018-cspa-results-509ace9...

Excellent points -- this is something we discuss extensively. After talking with so many employers, we found that each of them look for something different.

Adaptive scoring algorithms and ways to let employers create custom projections of our multidimensional data is something we're thinking about. If you have ideas on how to build this, I'd love to hear them! james@cspa.io

Great feedback -- thanks!

We're constantly improving the CSPA questions, and our latest iteration has quite a few changes. Some of the examples you mentioned assess how detail-oriented the candidates are. We're interested in how this data correlates with how bug-prone someone is.

That said, we trimmed some of those down in our latest version and replaced it with more substantial questions.

- james

Sorry to hear it was that bad! :( I'd love to hear your experience, especially if I was directly involved -- you can reply or email me anonymously at james@cspa.io. If you know anything about Crunchyroll's early culture, I always made a point to listen to feedback and improve ourselves. It'll be the same at CSPA.

Crunchyroll's interview process definitely was not consistent, which is why I spent my final two years there trying to improve our processes. I learned a lot from it, and that's why we're doing CSPA :)

- James Lin (james@cspa.io)