Launch HN: CSPA (YC S18) – SAT for Software Engineers | Hacker News Reader

Launch HN: CSPA (YC S18) – SAT for Software Engineers

51 pointscspa28 comments

Hi HN!

We're James and Angel, the founders of CSPA (https://cspa.io).

The Computer Science Proficiency Assessment (CSPA) is a standardized assessment for software engineers. We find software engineers to take our exam, and then share the assessment results with employers, who can then hire these candidates for free.

James was the founder of Crunchyroll and oversaw product and engineering. Angel also worked at Crunchyroll. We decided to start CSPA because:

1. There are many self-taught programmers out there who are really really good, but don't get a fair chance because they don’t have a degree. We want to give these people an equal opportunity to pursue their career. That's why there are no eligibility requirements to take the CSPA. You can come from any background, and if you get a high CSPA score, you’ll stand out.

2. As hiring managers at Crunchyroll, it was challenging to find great software developers. We wanted a service that quantitatively compares job applicants in an unbiased way, and ALSO did that assessment for us. In short, we wanted candidates to come to us pre-assessed. It would have saved us and the applicant a TON of time.

3. As software engineers, we hated applying for jobs. We did the same phone screens, the same coding challenges, and same whiteboard exercises over and over again, company after company. It's a huge waste of time. There has to be a better way. Why couldn't we just do it once, and then reuse the results at every company we apply to? Keep it DRY, right?

If you want to take the CSPA, you can register here: https://cspa.io/winter-2018/register. The first 100 people to sign up can use code "LAUNCHHN" to waive the exam fee.

If you're an employer looking to hire a pre-assessed candidate, sign up here: https://cspa.io/add-employer. We don't charge a recruiting fee.

If you're interested in joining the Technical Steering Committee, apply here: https://cspa.io/tsc/apply

I scanned through the "General" questions in the practice test linked on the thread, paying particular attention to the "Security" questions.

Not the best.

There are two questions regarding denial-of-service attacks. One of them I found hard to answer (had a plausible answer and a plausible "none of the above"). The second one was clearer but still ambiguous. Weirdly, the ambiguity in both questions stemmed from the use of an HTTP POST login as the possible DOS vector --- why? Why use that confusing example?

The 2FA question I saw had two valid examples of 2FA (push notifications and chip-and-pin) but no "all-of-the-above". I'm curious which of those two the author has demoted.

The phishing question I'm just irritated at the superficiality of; everyone will get the right answer, but the right answer misses the point and the power of phishing attacks, which isn't "websites that look like real websites" but rather the lures (like targeted emails) that get people onto those sites. But, whatever.

The password management advice question asked for a "best" from several subjective answers.

The SMS 2FA question had two valid answers and no matching "these-two-answers" answer. I'm pretty sure the answer that wss being looked for was the social-engineering phone-porting one (which is weird, because the first answer is "SMS 2FA is considered secure by current industry standards", which is certainly true for most reasonable definitions of "industry standards"). But also: that's not even the biggest problem with SMS 2FA.

There was a blockchain question. Who is this for? I'll buy that every working programmer needs to know how a phishing attack works (though perhaps not what distinguishes, in the test author's mind, a DDoS attack from a DoS attack). How many working programmers know how blockchains work?

Similarly: there was a GAN question. Come on. What was the point of that?

Finally: it's a six hour test. You weren't kidding when you called it "the SAT for programmers". It's not an especially pleasant experience (I like the interface, though). This test is a very big ask of candidates, and I think a short ways through the test it becomes clear that there's not much intrinsic merit to it; it's just a hurdle.

Six hour test on top of whatever time you spend interviewing. Because you know companies will still have you interview.

They would have to, since this is a general-knowledge programming test.

Our initial goal is to replace the technical phone screen. We are asking companies to guarantee an interview (or skip the phone screen), if they score above a certain threshold.

Long term, we hope to replace as much of the onsite as possible. But we do acknowledge we can never fully replace it.

May seem like I'm shitting on your product, but I'm not. I do a good amount of hiring[0] and have had to use services that provide similar outcomes[1]. It's mostly a waste of time and am currently working on removing that part of the process. My main issue with your product is that I don't see how it would provide me with a better outcome than what's out there.

I want to like it enough to try it out and pay you, but dont see value proposition right now.

[0] Corporate Fortune 500 types in many verticals. [1] http://derricocomputers.com/

Thanks for the valuable feedback!

Re: GAN, we tried including ML in the Core exam, but have actually since removed ML since it wasn't working. We may introduce this as part of an ML specific subject test.

If you want to get involved designing questions, please consider applying to the TSC for next term: https://cspa.io/tsc/apply.

Serious question: why would I do that?

Every version of this I have seen in the past 20 years always ends up worthless after the braindump sites pop up. How do you intend to mitigate this?

There are two requirements that I personally don’t feel very comfortable with: the first one is uploading a picture of my ID (How is it stored / backed up. Who has access: third party validator or your team?) and the other is adding logmein to let a proctor control my machine prior to the exam. I saw that one may take the exam in person. I don’t know if it’s allowed by your rules, but I suppose I can set up a VM for this, but either way that just makes it more difficult for me to justify finding the time to do it.

Our third party proctoring service handles the KYC, as well as the LogMeIn. We did some vetting, and they are used by some well-known MOOCs and universities' online courses. It's a concern of ours as well, which is why we try to offer in-person proctored tests or appointments.

I want to make another comment about how these questions are much more trivial, when compared to the mathematics SAT which has a lot of problem solving. I suggest have users solve more problems using computers, and less problems about computers.

I'm only making these comments because I don't want this to fail.

Thanks for the feedback! We try to have a good mix of trivial, novice, intermediate, and advanced questions.

Some of the multiple choice questions are fact-based and just require a Google search -- this is intentional, since proper Googling skills are still needed for the job :)

Problem solving questions are better formatted as open-response essay questions, which we do have.

thanks for the replies

It's good but... I don't want this to turn out to be anything like the SAT in that it's a requirement for schools and they won't have it any other way. I've found when it comes to schooling, people are ready to market aggressively because the students don't have a choice (we saw this with TI calculators)... So what's your promise to prevent this from happening? Software Engineering is good right now because anyone, absolutely anyone, can dive in. This is a step in the wrong direction in that regard.

Yeah that's exactly how it's going to work. Standardized testing is very rarely effective when employed in this fashion as a proxy for "talent" or "drive" or "good employee". It usually ends up being a proxy for something else.

This also pushed the onus on candidate selection even further on to the candidate, when it's clearly a lack of trust from employers that you are capable of what you put on your CV. Testing that should be the burden of the employer, not the employee.

One key difference between software engineering and other engineering disciplines is that there are many more ways to solve most problems, many more problems to solve, and an endless number of customizations. At a concrete level what does the canonical "web app" look like? We can say "3 tier architecture" but that's abstract -- which technologies should you pick? When should you pick ruby over python? What about when you should pick a document store of a relational database (OK that ones kinda easy) -- these are the questions that good engineers can answer and refute -- there's usually not an absolutely right answer, either.

I personally also feel the same way about the CKA (Certified Kubernetes Administrator) -- hire someone with a CKA and sure that means they know the Kubernetes basics/internals (as it stood whenever they took the test), but does that mean they keep up? Experiment with new tech/RFCs/features (so your company doesn't have to with the product on the line)? Can they choose not to use Kubernetes when it's not necessary?

Rather than just building good, multi-layered interview questions with subtly random parts and lots of room for interpretation (to see how they interpret and think about the problem), we just pick these solutions that are somewhat broken from yesteryear (see: it/sysadmin/networking certs).

We view this closer to CFA, where it's not required (I don't think we'd ever get to this point anyway), but we want a world where many employers do recognize CSPA.

> Software Engineering is good right now because anyone, absolutely anyone, can dive in.

Yes, I wholeheartedly agree! CSPA actually bolsters this because there are no eligibility requirements to take the test. We believe there are a ton of great engineers WITHOUT a college degree and we want to surface them.

We're making it MORE accessible, not less. There's no real barrier to taking the CSPA, except the time commitment. Just register and take it.

I have a strange fetish for multi hour standardized testing, so this looks great to me. Thanks for posting and for the code!

One question based on your responses in this thread: is this test intended to measure software engineering ability or computer science knowledge?

EDIT - one more question after looking through the grading: do you see people studying for specific roles, or the entire test? While I think the latter is possible, it seems much less feasible than studying for, say, the entire SAT.

Currently the Core Exam is 50/50. It's designed for career-readiness. Our Technical Steering Committee is composed of majority of hiring managers, so it's intended to be useful for them -- that is, what are the qualifications desired in an entry level software engineer?

We are debating having a Core general exam, and separate per-subject tests, similar to SAT I vs SAT II.

Hi James and Angel congrats on the launch! Great issue to try to solve, hope you succeed.

Thanks for the LAUNCHHN voucher code. I'd seen cspa.io on hackernews previously but the free voucher code was the motivation I needed to register for the core exam in April! I'm a uk-based maths grad previously in a datasci role, currently taking some time off to self-study computer sci and frontend/backend. Looking forward to checking my progress against the test!

Quick bits of feedback in case they're helpful:

(1) I'd love there to be a second practise/past paper available (there's only one currently right?). Ideally I'd like to go through one now to check where my knowledge gaps are... then take one a couple of weeks before the exam in April.

(2) I just gave this only a very quick read... and found it a bit unclear: https://cspa.io/scoring. I'm sure I could understand it no problem with a more serious read-through... but thought I'd feed that back anyway. [Specifically it's not clear in the examples section how the score of 440 in FE is achieved, as the max mentioned is 400. Also the 6 subject scores all differ... but then presumably are scaled to 400 points. Is that right? With my quick read through I'm not yet clear if in the exam I should try to complete all 6 or just complete 1 of the 6 subject areas. I guess as final score takes your best score out of the 6 you could just complete one or two. What do most candidates do in practice? Perhaps you could add some guidance on this to the site somewhere if it's not there already.]

Regards. Mike.

Thanks!

(1) When we retire old exam questions, we will make them available for practice. We're working on creating more study materials! I can't promise anything before April though.

(2) Ah the 440 FE is a typo. We updated the scoring algorithm a few months back and didn't update those examples. Most people try completing all 6 subject sections. It's up to you, but we do discourage optimizing this kind of test taking strategy, and "teaching to the test" ;P

Sorry, but 5-6 hours is a non-starter for me. I've never taken an exam that long in my life (the SAT is about 4 hours IIRC), and I'm so glad to be out of college and done with this type of BS. At least interviews let me talk to a person and learn something about them and their company - sometimes I get a story out of it.

Sample exam, for those interested: https://cspa.io/sample-exam

First off, I like the idea. Tech interviews are so broken that I'm happy to see folks trying to make things better.

I just started clicking through this and stopped at question 7: "How many bits are in 8 bytes?"

I don't know if this is intentional or not, but this is an ambiguous question. I suspect the "right" answer is the one that assumes 8 bits in an byte, but historically that assumption is not always true, e.g. lots of IETF RFCs refer to "octets" rather than "bytes" to avoid the confusion.

I bring this up not to quibble about historical architectures, but to address another item I recall from my SAT/ACT taking days--getting into the head of the question writer. When I saw this question I thought, "Are they looking for the current/common interpretation, or do they expect me to know mostly useless trivia about the history of the definition of byte?"

Perhaps I'm overthinking it. I probably am. But for that specific question, the answer is "it depends." And based on that, there is no 100% correct answer offered, so I'm left choosing the next best answer, which is probably the one most folks would select anyway.

Was this intentional? Again, it's been a long time since I've taken these types of tests.

We've found that that is indeed true. The more senior and knowledgeable you are, the most likely the answer is "it depends", because all the nuances and edge cases you know about.

We try to solve this with the standard "choose the BEST answer". We also do post-mortems and analyze the responses to disqualify or discount bad questions.

These are issues the CSPA Technical Steering Committee, which is responsible for the exam content, will deal with, in collaboration with our research advisor: https://cspa.io/about/team

Why does it take 3 to 10 days to process results? What's stopping you from delivering results within a second?

Edit: I didn't see that there was open ended responses or a programming project on some of the exams because I was looking at the sample exam, which only has multiple choice.

Hi James and Angel! You are doing a great job with your project. Does CSPA concentrate only on hiring processes or also helps to find partners, contributors and the possibility to develop tech community?

Was curious as to how tests like these guard against Goodhart's law (If goodheart's law is applicable here at all)?

Thanks for the tip! Can you elaborate? Do you mean teaching to the test, or something else? There's a whole research field for large scale assessment and evaluation methodologies.

Reply on news.ycombinator.com