The Lesson to Unlearn
paulgraham.com
paulgraham.com
For example, it's long been known in the physics education research community that students come away from introductory courses with very little physical understanding, even if they can do the plug and chug problems on typical tests just fine. Students can all recite Newton's third law, but immediately afterward claim that when a truck hits a car, the truck exerts a bigger force. They know the law for the gravitational force, but can't explain what kept astronauts from falling off the moon, since "there's no gravity in space". Another common claim is that a table exerts no force on something sitting on it -- instead of "exerting a force" it's just "getting in the way".
For research purposes, we measure physical understanding using a battery of tests, such as the Force Concept Inventory, containing only simple conceptual questions with unambiguous answers. So then everybody asks: if ordinary tests are so hackable, why not just switch to these conceptual ones? But that wouldn't work. There are less than ~100 distinct FCI-style questions. If these conceptual tests were the norm, students would just memorize the answers and parrot them back, with a flimsy understanding that crumples the second any follow-up question is asked. It would be just the same problem as before, except they would be worse computationally, too. The FCI only works as long as it doesn't count for a grade.
The problem isn't tests, it's scale. If the people aren't motivated, any standardized measure will miss the mark -- even the entrepreneurship Paul Graham advocates for. God knows I've seen a lot of bullshit in that direction.
Shouldn't that be a sign that trying to come up with standardized measures is a fool's errand?
Short of blatant social engineering, I don't see how we can have a system that could improve people's motivation and drive for education. And if we don't have a way to actually change human nature, why do we keep trying to design this system that would magically work for so many different people?
We could give more autonomy to individual schools or teachers in terms of designing tests, but then we'd be facing two problems. One, the grades given by different schools/teachers would not be directly comparable. An A in one school could be awarded only for exceptional circumstances, and in another just for above-local-average. Two, once you start using those grades to judge people in a competitive situation like the job market, you'll have schools selling grades, teachers inflating grades to help their pupils compete against those from schools that sell grades, etc. and the whole system will collapse to uselessness.
Standardized testing mitigates these problems by forcing everyone to grade by the same criteria, but in order to do that, the grading must be done on things that are extremely easy to measure - which removes both individual judgement of the teachers and any relation between testing and understanding of the material.
The more I think about it, the more I feel it all boils down to scale. The grading system needs to be scaled up with the size of the market people compete on, otherwise it'll be either trivially gamed, or deeply unfair to unlucky people.
Accounting and law firms are looking for people with the stamina and single-minded goal focus to get document-creation tasks done, one after another, to a high degree of "skill." Getting an A on the test and preparing an SEC-compliant bond prospectus are close cousins of each other.
In fields where real-world mastery doesn't have much to do with the orderly routines of getting a 97 on the midterm, employers are less likely to hunt for people who got 97s on the midterm. (John Steinbeck stumbled around Stanford for six years and never graduated. But he got more accomplished as a writer than a whole decade worth of creative-writing majors with high GPAs.)
Are they really, or is just this elaborate illusion supported by most decision makers to give some rationalization to their own selection processes? When you have Harvard being sued for discriminating against Asians, how can we keep a straight face and tell the kids that they need to study hard because the grades matter?
I think this is part of what PG wants to defend in the essay: why do we actually care about grades at all?
Because jobs are at a premium and evaluating prospective hires is hard, so employers look for anything they could use to filter and sort the stream of candidates. Education - both the what and the where is somewhat correlated, so that's what's being used.
I think I spent too much on this discussion already, but the point I am trying to make is that standard tests are bad both as a filter for selection and as predictor of performance.
While not necessarily agreeing, I do understand if someone says "I will hire someone good from a top-tier school vs a top student from a not-so-good university". The "what" and "where" are definitely useful data points to filter out candidates.
What I don't understand is any employer that says they will hire based on SAT, GPA or any variation of an IQ test. It seems to be so unreliable measures to the point of actually hindering the process.
I am not asking for any utopia. What I am asking is "Instead of setting ourselves for failure every time coming up with some scheme for testing performance, why can't we just use actual performance on the activity that we want to evaluate candidates"?
It works for people in sports. It works for trade professions. It works for traders. It works for performers, entertainers, even entrepreneurs.
Why can't we have an educational system that does not put people on tracks to mold them and instead just coach them to follow whatever path they seem fit?
Sports works, but only because sports are extremely popular, with tens of millions of people enthusiastically participating on a regular basis, and accessible to anybody no matter their age and income. This is not true for, say, working at a law firm. You can’t take the best clerk from a pool of a thousand kids who have been clerking since elementary school. It’s either who you know, or who got good grades and test scores.
One would argue that this is the current status.
How objective are, e.g, IQ tests when I can take two different tests in a span of two hours and get from 128 to 146?
How my grades in Computer Engineering can be used to predict my performance as an Engineer in a country where 85% of the graduates would work in non-technical and more profitable fields?
If a company looking for a Quant gets someone with a Master in Math and a good tracking record knocking on their door, would they stop and ask to see if they can get someone with a PhD instead?
You are right that "proper evaluation takes time". What I am saying is that standardized tests should not count as proper evaluation.
> How objective are, e.g, IQ tests when I can take two different tests in a span of two hours and get from 128 to 146?
You can't take two proper WAIS-IV tests in two hours. In fact, unless you have been part of an unusual psychology study, you've probably never taken a real IQ test in your life. There are a lot of gimmicky free things on the internet that call themselves IQ tests but aren't predictive of anything.
As for the IQ test - it wasn't online and it wasn't a study, as far as I know - but I can totally concede that it was not a "real" IQ test.
Anyway, the story happened over 20 years ago. Part of the driving exam in Brazil included a "neuro-psychological evaluation" and the doctor flagged me for some extra tests. From what I remember I went to the clinic the next week, did one test, did the "neuro-technical" exam and was told to wait for an IQ test. When I got the form, it was the same as I got before originally (different order of questions) Long story short I asked why I was re-taking the test and I thought I "failed" somehow. They actually said they were not aware of the first one and later on found it.
Now that I think about it, perhaps it was some weird psychology test? :)
Throughout my life and career I've seen so many statements of this nature from my peers: "the status quo will only ever be a 90% solution at best, therefore we should throw all of that away and just do everything on a case-by-case basis in isolation from one another, then we'd all live in an ideal meritocracy (or something)".
See also any discussion about flat management. "Explicit management hierarchies obviously aren't perfect, therefore let's get rid of them altogether and assume that everyone is awesome and will magically self-organize and do the right thing."
I guess the common theme is, someone is greatly overestimating their own rationality, and projects it onto other people, and ends up with an almost religious faith in the self-organizing competencies of unmanaged people, or the natural inclination of unregulated schools or industries to always do the right thing even when it's not in their own self-interest.
IME there are lots of fundamentally messy or imperfect things in life, where any choice you make has negative consequences. There's also a personality type that is repelled by this sort of messiness and will always be drawn towards throwing everything away and starting from scratch, often with a childlike belief that things they don't understand aren't important, and why can't everyone see how obvious and elegant their solution is? IME they're usually wrong about those situations, but I guess I'm still glad people less pessimistic than I am are tilting at those windmills.
The problem isn't hyperrationality per se; it's a lack of intellectual humility and self-awareness. The behavior you're talking about arises when someone looks at a problem, immediately wants to fix it and makes snap judgements to figure out how to do so. Instead, they should dispassionately consider 1) whether the problem is fixable in any meaningful sense, and 2) whether they have the requisite experience, skill and insight to solve it in five minutes of thought.
You could further distill this idea with the observation that a lot of people consider things too systematically, rather than holistically. "First principles" thinking is a powerful drug, which appears to crack open the world and solve every problem. But no problem-solving paradigm can deconstruct away unknown unknowns or inexperience.
What I am saying is that what you regard as a "90% solution at best" is actually not a solution at all. Standardized tests are terrible predictors of performance on actual jobs. If standardized tests (or IQ tests or anything that is supposed to be a proxy for measuring performance) shows time and again that it only manages to measure people that are good to "study for the test", then why do we keep trying to use them to measure performance? Is it because it "the only thing we have"? This seems like saying "I am lost in Rome but all I have is a map of Paris, let me use that anyway."
And no, I am not proposing some other solution to replace it.
There just has to be an immediate value in what you teach. If there isn't, there's no point in teaching it. Students will just forget it anyway after the test.
I'd actually take the lesson the other way. It's hilariously tough for model-makers to acknowledge real-world noise that prevents their theoretically precise models from delivering reliable results.
One of the things I think was the best ideas I learned in physics is that physics is all about approximations. Theorists tend to forget this annoyingly, but it helps keep naive scientists from using words like "fundamental" without qualifications, or saying garbage statements like "Newton's law X is wrong."
https://v.cx/2010/04/feynman-brazil-education
> There are no experimental results mentioned anywhere in this book, except in one place where there is a ball, rolling down an inclined plane, in which it says how far the ball got after one second, two seconds, three seconds, and so on.
> The numbers have ‘errors’ in them – that is, if you look at them, you think you’re looking at experimental results, because the numbers are a little above, or a little below, the theoretical values. The book even talks about having to correct the experimental errors – very fine.
> The trouble is, when you calculate the value of the acceleration constant from these values, you get the right answer. But a ball rolling down an inclined plane, if it is actually done, has an inertia to get it to turn, and will, if you do the experiment, produce five-sevenths of the right answer, because of the extra energy needed to go into the rotation of the ball.
> Therefore this single example of experimental ‘results’ is obtained from a fake experiment. Nobody had rolled such a ball, or they would never have gotten those results!
> “If I ask you a question during the lecture, afterwards everybody will be telling me, ‘What are you wasting our time for in the class? We’re trying to learn something. And you’re stopping him by asking a question’.”
In my experience there are almost just 2 types of motivated students: those who want to understand the content of the course and those who want a good grade and ultimately graduate.
There's often a conflict of interest between them. For example the first group wants to learn more, ask more questions, go deeper. The second group sees this as a threat because the additional content may now be on the exam as well. And they often misunderstand this behavior of the first group as just being teacher pleasers/pets.
Or from own experience: when you tell the prof that he accidentally started to cover the same topic that we already covered in the last lecture (because you're there to learn more), you will annoy a lot of people who hoped to get one fewer lectures to study in the course. Less content, less effort, same grade.
You hear the first group having passionate discussions on physics or programming over a beer, while the other group discusses all the latest info of which prof is how demanding, what the grading criteria are, what the deadlines are, which departments you should or should not take courses from, where they have mandatory presence, who does multiple choice tests and who does free-text exams.
I don't have much to conclude, just an observation.
I should clarify the last time I had this conversation, I'm talking about 40+ year olds building bikes. They were thinking a heavier bike would accelerate faster down a hill. That just is not how that works.
That is to say, the force from the air does not actively slow you down. It prevents you from speeding up, after a point. (Aerodynamics are, of course, more complicated. But this general model is pretty solid.)
Such that if I want to accelerate faster down the hill, I have to pedal to cause that to happen. Or add a motor.
Yes, acceleration due to gravity is the same, but air resistance has less effect, at all speeds, on the heavier bike. The rate at which the heavier bike gains speed is higher. It accelerates faster.
Take my statements to be, it won't hit a meaningful higher acceleration. Not that it won't necessarily accelerate for longer.
In general, you will accelerate down the hill at 9.8 m/s^2 modified by the incline. No matter how heavy the bike is. (Within the realm of realistic weights.).
Yes, the points you are raising are true. But within the realm of the biker and realistic bikes, not really relevant.
You can say that two larger things attract each other more than smaller things. Especially as they get closer to each other. Catch, of course, is all things on the earth are in the noise range compared to, you know, the earth. If you are talking about gravity and acceleration on the earth, you are talking about 9.8 m/s^2. Pretty much period.
And yes, there are terminal velocities that impact maximum speed. Acceleration, though, is not velocity.
It's clear that there's more force on a heavier object. How much force do you have to apply to counteract a bucket full of rocks from falling to the ground, vs. an empty bucket? Your arm is certainly applying more force to the former.
So then the question becomes Why doesn't the full bucket fall faster? Because it has more inertia, and thus it's more difficult to accelerate. But there's also more force, which overcomes this difficulty. The amount of greater force is exactly identical to the greater inertia, so the acceleration remains the same. The "exactly identical" here is no coincidence.
So yeah, that makes sense.
Similarly, I only mentioned terminal velocity because I said faster. Which could apply there. But my point was focused on acceleration of falling objects. I get that there are a lot of ways to muddy that discussion. As exemplified by my postings. :).
F = G * m1 * m2 / r²
This tells you that the acceleration of mass 1 is independent of the mass 1 and only depends on mass 2 and the acceleration of mass 2 is independent of mass 2 and only depends on mass 1. But the relative velocities will increase when either m1 or m2 is increased.
E.g. a black hole with the mass of the earth will hit the ground of a (perfectly rigid) earth in almost half the time as a feather would.
(I won't go into relativistic effects here.)
That is, would you expect adding ten pounds to have a meaningful difference in how fast a bike will accelerate down a hill? Even fifty pounds?
It will impact brake force needed to stop. But the speeds are going to be near identical.
You can use a bunch of words to explain what's really going on there, but it won't change that the brick hits the earth first, hence it falls faster. Unless you can predict the future correctly and communicate this simply to your peers, what's the point of being correct in the terminology?
My point was strictly that many people will falsely think gravity somehow meaningfully pulls lighter objects at a lower rate. This despite days off lessons that gravity on Earth is basically a constant.
That is to say, if your model of speed of things falling is hinged solely on gravity, it will not be predictive in the extremes.
Of course, most folks aren't playing in the extremes, so the simple model will be fairly predictive of everything they do actually toss around the room. Can easily explain why the stack of books fell at the same speed as the basketball when the dresser was knocked over. :)
If only they had told me everything (including friction), then maybe phyics would have made a lot more sense to me. But then again, school math never even mentioned imaginary numbers. School is very incomplete.
Same goes for everything underwater. Which follows from viewing air as a medium you are traveling through.
Are there bad teachers that need to more honestly cover some of the assumptions? Yes. They are a useful tool, though.
I did Physics 2 that way and really liked it.
https://en.wikipedia.org/wiki/Keller_Plan
(also, wow, I just got that Principle Skinner in the Simpsons doesn't accidentally have that name. Face to palm.)
I recognize it's not feasible for a class of students to constantly interrupt a lecture. The questions that other students had were never the same ones that I had, either. The problem is indeed scale, but I don't think motivation is a primary factor. Eventually you learn to be quiet and accept what the teacher and the textbook say.
Hear, hear! When people say that with better technology we could automate away teachers, I say that with better technology we could automate most everything else and spend the time mentoring one another.
To take your gravity example, the first question might be something about calculating gravitational force on the ISS. Then explain that it's rotating in space and therefore has centripetal force that accounts for this gravity. Then compute how fast it must be rotating. And from there you can switch to a completely different topic in the same context. Maybe something about EM radiation from the Sun that falls on the space station.
Goodhart's Law (Popular formulation): When a measure becomes a target, it ceases to be a good measure.
Part of the problem is the lack of time. Both teachers and students need to cover all the topics in one semester/year, and it takes time to assimilate and integrate the knowledge. Add to that that as student, you don't know what's important, even when you're told so, if you're told at all. That's why we can hold contradictory models in our minds. They're good enough to solve some problems, but conceptually wrong.
I think that a possible solution would be to keep the grades open. You get a B in physics-101, since you got a flawed model in your mind, you got to physics-201 where new experiences show the problems in your model and force you to correct it, and now you prove you achieved a knowledge that deserves an A.
I know, that proposal it's impractical, and time consuming, but I think that's how most of us really learn.
Standardized testing is good for:
- measuring system performance against set standards - giving basic insights into student's average level of knowledge of a subject (if you use a thorough theoretical framework to conceptualise learning outcomes and link test items to this) - maybe measuring school performance against standards - maybe measuring teacher performance against standards
But for students this individual measure of an exam is simply not a good measure of learning in any modern education system. Yes it is a useful measure in really bad systems where teachers are bad and teach wrong things, or where eg attendance and exam fraud are huge issues.
But modern pedagogy is not about teaching book knowledge which is easily measured. In any Western education system we are mostly past that point. It's about helping students discover own interests and abilities, develop competence and confidence, experiment, fail and try again. This learning is simply too individual and hard to measure for a standardised test.
The only option is formative assessment (essentially teachers grading students repeatedly throughout the year on things like effort, achievement of general learning aims, additional development, results produced, presentations, support to other students, etc - a bit how you would measure a good office worker). Or maybe let's do like Finland and Estonia and let's mostly eliminate the testing burden.
The basic principle of education is to start where the student is and then lead them step-by-step to what you want them to understand. That being the case, I would say physics should start with student's intuitive understanding and then explain, like through examples they can understand, why some of their intuitions are mistaken. Maybe this could even be turned into Socratic questioning.
Funding, growth, usage -- these are all still one level removed from something worthwhile or beneficial. Cigarettes have amazing usage numbers even in 2019.
I'm waiting for the entrepreneur who will say he doesn't care how much money he makes or how many customers he has, only that it's worthwhile and needed doing.
Closest to this today might be Elon Musk, maybe...but there are other complexities there?
The entrepreneur is by definition somebody that cares about money.
Regardless, Jack knew he needed to make money eventually is what that shows. Uber may be foolish in its attempts to make money, but it absolutely is trying to in the long term.
There are other people that intend to swindle people (make money for themselves, not the company).
There are other reasons to start a company, but then you have an expensive hobby. If all you ever do is blow money you didn’t earn (your inheritance) starting businesses that fail, at best you are going to be called a failed entrepreneur.
Unless the people involved are in a financial state where they are not dependent on revenue from the company, or we divorce survival from capital, I doubt it's really possible for "not making money" to disappear as a terrifying core motivator for every business and every employed person regardless of whether their marketing copies are disingenuous.
That's the root of the issue, not motivation or purpose or what the individuals value or the intentions behind the creation of the company.
Companies like Theranos and WeWork succeeded for a short while because they tried to "hack the test". E.g. Theranos hacked the metrics of "buzz", "investment amount", "notable people on your board", but they didn't actually have a working product. Adam Neumann certainly "hacked the test" to the tune of a billion plus dollars, but WeWork certainly didn't succeed (I'd note there is still a critical difference between Theranos and WeWork in that customers actually really like the product WeWork provides, the question is just can they provide it profitably).
I think hacking the test can get you so far in the startup world, but eventually reality rears its head and what matters is whether you can provide a product people want, profitability.
One quick note on your cigarettes example. The problem here is that companies are giving people what they want, they've just figured out how to "hack the test" so to speak with respect to what the human body wants. This is basically the case with all addictive substances - there is essentially a hackable part of the human "desire" subsystem that people have taken advantage of.
Framing it as lack of product is unfair to all startups who lacked good product, hacked buzz, got investment, got notable people and then did not committed fraud.
Plus it suggests that Theranos fraud is result of not having product. The same people would be engaging in unethical conduct with product too.
Plus, Theranos fraud was waaay more bigger then that.
For any non-trivial product there's information asymmetry problem, so pretty much everything is hackable, but to different extents.
I believe this is the role of the nonprofit sector.
I think it's PG who mentioned it in some other essays, but "non-profit" and "company" can be seen as different ways to make things happen, a different models of getting society's help with bringing your visions into the real world. Non-profits tend to depend on other people's generosity; for-profits have to earn their money directly. Both have to care about money to some extent, as money is the lifeblood of all large-scale endeavors in this world.
All of that is orthogonal to doing something worthwhile. There's plenty of non-profits that exist primarily to make money for the founders. Conversely, there's e.g. Elon Musk and SpaceX, which is a successful for-profit with an explicit, unchanging, non-monetary terminal goal: making humanity multiplanetary, by bringing down the costs of space so that it's economical to venture there and colonize Mars (you can argue whether that's worthwhile, but I think it's well proven that Musk/SpaceX are honestly pursuing space, not money).
This is literally all of open source software. The problem here is that you're waiting for an entrepreneur to say this, which is the wrong kind of person to expect this from.
I agree and can’t help but wonder if this is why we’re seeing more and more developers trying to make money from their Open Source projects.
I think they're really just trying to solve the problem of "how do I afford to eat while still supporting software used by Facebook to make billions of dollars"?
And these days, I think most of the open source software I use was actually contributed by a corporation.
The media buzz is dominated by mostly-not-very-good VCs and mostly-not-very-good startups trying to find and impress each other. The VC market is designed to seek out the "unicorn" companies whose value will grow 1000x, and so the companies that might be that and try to look like that get the most buzz. There's nothing wrong with starting or funding that kind of company, but it's not for everyone, and it certainly isn't all that's out there.
It's still viable to start a smallish lifestyle business that produces enough money for yourself and doesn't take a ton of work to keep going, but will never grow to be 1000x the size. It's just fine if that's what you want to do too, but know that such businesses usually don't market too much, either for customers or investors. They don't want VCs, and VCs don't want them either.
They're bad, not liked, etc etc. The usage in 2019 reflects something deeper: their effectiveness, perhaps a cohort effect and the difficulty of quitting, and a certain value proposition.
There's a term I'm perhaps misusing which is: revealed preferences.
This statement is perhaps a stretch but maybe what paul meant is that learning history for history's sake is helpful but learning potentially untrue test patterns (aka launching on a tuesday to try to maximize funding) is not.
This might not get a lot of traction but it seems your last sentence describes the Ayn Rand novel where the guy spends years toiling in anonymity versus his peer who is "fake" and successful, where the value is in the work itself.
PG: if you had to boil it down to one quality to look for, authenticity would be the most important one. “You’re looking for people who are real friends,” he said to Chang. “Not just for people who got together for the purposes of this startup. You don’t want people who were in it just for the money.
He also talks about how that startup founders that are more interested in the product or the challenge of building a business, make buttloads of cash as a side-effect.
And YC funds nonprofit startups too?
Also from another 2004 PG essay:
“””Money Is Not Wealth
If you want to create wealth, it will help to understand what it is. Wealth is not the same thing as money. [3] Wealth is as old as human history. Far older, in fact; ants have wealth. Money is a comparatively recent invention.
Wealth is the fundamental thing. Wealth is stuff we want: food, clothes, houses, cars, gadgets, travel to interesting places, and so on. You can have wealth without having money. If you had a magic machine that could on command make you a car or cook you dinner or do your laundry, or do anything else you wanted, you wouldn't need money. Whereas if you were in the middle of Antarctica, where there is nothing to buy, it wouldn't matter how much money you had.
Wealth is what you want, not money.”””
I work towards wanting less stuff. Living a simple life, I do feel richer for sure. Wealthier, perhaps not, I don't even care actually.
"Wealth" is still problematic, as per the cigarette example.
Also, most small businesses do stuff people want. But it's mostly commodities. So the hard part in many small businesses is marketing.
And marketing is mostly hacking.
As for startups,marketing is very important for a startup.
But is it common to sucseed without something people want, or at least get addicted to ?
The profs were literally 1 chapter ahead of the students.
I, however, was in love with the subject. I was the "student advisor" in our "machine room". We had 5 teletypes connected to a remote mainframe at Rutgers. I lived and breathed CS.
I wrote tests for the profs (even thought I was also in the class) and I answered questions for students about the tests when they came for computer help.
Pick a subject you love so deeply that you're always at the leading edge (which essentially means a new subject the school wants to teach). For example, CMU introduced an AI curiculum. Learn how to do NNs, DNNs, GANs, etc. Read the latest papers. Write working code. Chat with the profs. Hang around the dept.
Proof systems are a hot topic. Learn LEAN, COQ, AGDA. Learn to write proofs by machine. Read the papers. Get ahead of the curve. It is hard to "fail" a student who knows a lot more than you, especially when they help craft the tests.
Intel stuck an FPGA inside its CPU. Learn Verilog and learn to hack new CPU instructions (e.g. read Gustufson's book "The End of Error" that introduces a new kind of floating point arithmetic). Write code that outperforms BLAS code.
Learn BPF (Berkeley Packet Filter) so you can do impossible things about the kernel in User mode. Make your networking class look like it is stone-age.
I could go on but you either "get it" or you don't.
I understand that being at the leading edge has benefits, but it's not feasible to understand the state-of-the-art across an entire CS curriculum.
E.g., early in my career I stumbled onto some of what he is saying but then didn't appreciate the situation. Basically, just stay with what is hot. Get into a hot job, learn what is hot, jump to another job that wants the hot stuff, keep going for the hot stuff.
This way get to focus: (1) Actually can get okay with the point of the focus at the hot stuff because that point is not very large or hard if only because what is "hot" is from a public that doesn't know very much and (2) do not have to learn computer science broadly. If take the money from learning the hot stuff, get a good Ph.D. in what is arguably the better stuff, go back to the job market, and no longer be with the then current hot stuff, then can find that your career is in trouble.
Then using your actual competence in some actually important stuff, do a startup where need to please the users but not some naive public fooled by mostly simplistic hot stuff.
Pick a focus and whenever you get to the point that you're a strong expert (or whatever level you feel is useful) and can do the job you want to do you should reach a bit and pick up something you're curious about to add more diversity to your skillset.
Being a generalist isn't bad, I basically have a PhD in being a generalist and tons of people want to work with me because someone who only understands optics will be lost when they need to use a photodiode and a lock in amplifier or someone who wants to test magnetic properties will be lost when they don't know how to actually construct a custom magnet. Among many obvious examples.
Unless you're at a place so big that you can just glide on assuming other people will fill in the gaps in your knowledge, you need both general and specific skills. A PhD doesn't have to mean you are only capable of one thing unless you let it, even if that was the common meaning 20 years ago. There just aren't the jobs in academia to support that kind of singular focus anyway.
We're going to live long enough that it's a terrible career decision to pigeon hole yourself so badly 50 years before you're even theoretically going to retire. Especially with online classes it's only going to get more common for people to have multiple significant skills and major changes in their field over the course of their career. Required even. Very few people are going to go for 50 years with a single technical expertise, certainly jobs for such experts are far harder to find than jobs for people who can do a few things at a more modest level of expertise.
I know a ton about materials science and physics, have genetically engineered viruses, have built chemical reactors, have designed catalysts, have done machine learning, have designed lots and lots of electronics, have programmed as required (not my favorite kind of work, but I've done 8051 assembly up to ARM firmware, Matlab, Scheme, Python, and shudder LabView), am more than half decent at machining, teach karate, perform music, and make art... trying to be the best at just one thing to the exclusion of all else is unnecessary and limiting both professionally and personally.
I was terrible at all of these things when I started. The article echoes my philosophy as well -- you have to learn to learn because you want to and not because you want a grade. I got a C- in differential equations. Two years later in stat mech I was the only one of my friends who recognized LaGrangian multipliers being used (by name even) because when I took the course my goal was to learn differential equations regardless of what grade I got, so I didn't forget it all the second the final was done. If you are the kind of person who just won't do homework unless it's graded, you should work on that because it will hold you back and you're training yourself to rapidly forget information that isn't required to pass a test.
Don't be afraid to try new things that interest you. Be okay with skilled people saying you're bad at something you did your best at, grow a thicker skin since those people are the ones likely to be able to give you advice to become good at it. Professors want you to come to office hours, especially if you're struggling.
You have to be okay with getting that C-, who cares what grade they assign you as long as ten years later you still remember convolutions or whatever. Be solid at a bunch of stuff rather than extremely good at, I don't know, something super specific like the spin states in cobalt platinum alloys or matlab modeling of alloy surface energy same as you shouldn't refuse to learn anything but Python if you want to still be a developer in 2069.
The thing to avoid is becoming a person of one book, or a person of one tool, where you know one thing and only one thing, and you experience deep anxiety when you're out of your (one, singular) comfort zone. Being an expert in everything is impossible, but being an expert in one thing with no knowledge of anything else is definitely something to avoid.
Thus, the best way to learn is to be in love with the subject.
During school holidays, he actually starts to do school exercises for fun; those are the very same exercises he would fight against doing for hours on a regular school day. Go figure.
Btw, Gates, Page, Brin - all were not in a conventional school. The capacity to just build things without fear of being judged is invaluable.
We send our son to one. It's a leap of faith; the kids essentially "do whatever". But it seems worth it compared to the damage done by middle and high school. And the kids that I've seen come out of this school seem to do well in society (and college) and generally seem much more mature than high school graduates.
I went to a "traditional Montessori" school for a few months when I was very young. Long story short, their educational method insisted that we needed instruction in how to play with toys before we were allowed to do so, and we needed to do so on mats, which needed to be rolled up tidily at the end of the prescribed playtime.
The problem with that was that we already knew what Legos were. But we were supposed to pretend we didn't, in order to appease the teacher.
I think this was the same school that insisted to my parents that I couldn't read because they (the school) hadn't taught me to read. My parents taught me to read. So they had me demonstrate, and the teacher decided my parents must have... trained me to pretend to read.
The person who mentioned waldorf must have been confused. While they’re both small private schools, they are two very different approaches to education.
[0] Wiki Brin: https://en.wikipedia.org/wiki/Sergey_Brin
[1] Post Article: https://www.washingtonpost.com/local/education/montessori-lo...
[2] ABC Interview: https://www.youtube.com/watch?v=0C_DQxpX-Kw
[3] Wiki Page: https://en.wikipedia.org/wiki/Larry_Page
[4] Guardian Article: https://www.theguardian.com/sustainable-business/2014/jul/23...
[5] BI Article: https://www.businessinsider.com/tech-innovators-who-went-to-...
[6] Forbes Article: https://www.businessinsider.com/tech-innovators-who-went-to-...
For me, I had to unlearn self judgement. This has helped so much over the last few years.
I just didn't do my homework and got bad grades. Luckily in my country you can always get in via standardized tests instead of grades so I did well in life anyway. I'm not sure why homework is weighed so heavily in grades, it mostly just shows how much free time the students are willing to sacrifice. The biggest cost this had for me was the guilt that I hadn't done what I was supposed to do, which made me feel like a bad person, but in hindsight I did much better than most of my more studious peers so it really wasn't all that important.
This is Anecdata^3, you probably shouldn't make rash decisions about your children based on 3 successful technical entrepreneurs.
That does not sound like exercises themselves would be bad or that he would cease to be interested in learning overall.
It more sounds like after many hours at school, he wants to do different things that day. But when school is not present, he seeks similar activities and his willingness to learn is not deteriorated. That kind of sounds healthy and not wrong at all. In school, kids spend a lot of time learning or doing focused activities, a kid wanted to just play after that should not be sign of something grave.
The only thing that is wrong or unusual is the hours long fight against homework. I dont think that is normal for majority of kids. Only some parents I know report that much daily fights, most have fight here and there once in a while only.
The only reason they stopped is I was getting top marks in every subject anyway, and the only reason I had that was I had a tutor who pushed me so far beyond high school level that I was finally interested in the topics again.
The neat part was: to learn those topics, she gave me a crash course of most of the year 11 and year 12 curriculum in 3 or 4 months or so, because I needed the base knowledge to learn the cool stuff
Most kids with good grades don't have tutor, unless they struggle a lot or it is before entrance tests for high school and college.
I am curious, is this about some particular kind of homework? From my own experience, I would think that exercising a subject alone (which is what most homework was in my school system at least) is a very good way of understanding it better. I of course remember not wanting to do homework many times, and I remember excessive or idiotic homework assignments, but overall homework seemed to be one of the major modes of learning for me.
Also, related to Gates, Page, Brin - I don't generally think that it's a good idea to look at a small number of very successful people and try to emulate some part of their life experience - you are, în general, very likely to fall for some kind of survivorship bias, or accidentally home in on a less important detail (for example, Gates was probably helped much more by his extremely wealthy and well-connected family than any particular aspect of his schooling).
On the other hand, there are many voices saying that one of the important purposes of the public education system is to engender conformity, which is very rarely conducive to a truly out of the ordinary life. Noam Chomsky would be one of the most credible, and he has often credited his own career to his non-standard early schooling.
I've got two kids, and so I can attest from limited anecdotal evidence, that it's completely unpredictable how each kid will relate to homework.
wikipedia said he went to a regular private school then went to Harvard which sounds pretty conventional. Yeah he didn't finish, but worry about that later.
Been there myself.
Most school systems have been developed to produce workers, not scholars and researchers.
Hence, rota memorization in prioritized.
Getting people to hate learning is not seen as a serious failure of the school system.
Also, students are never encouraged to question the information they receive or the decision around what is important to learn and what is not.
non scholae sed vitae discimus.
FWIW the UK Education Act actually enshrines this concept (parents are responsible for a child being educated), though this has been eroded considerably by Tory policy in the last decade or so. We did tell our primary school "don't expect homework from $child[0]" one year, they didn't complain. However we also wanted to do flexschooling and the head refused it, which I'm still smarting over and consider to have been a significant detriment.
That same primary school ditched regular style homework in favour of projects, which $child[1] wants to do, and we can decide easily to curtail the time used doing it if we want. It's a great way of doing homework IMO; very child-centric.
I'm sympathetic to unschooling, but I've also seen it go wrong. If you decide to unschool, don't be dogmatic about it. Feel free to be more or less unschooly if that seems to be needed. Don't get sucked into a community where unschooling "purity" is highly valued.
As someone who participated in many academic competitions as a child, the winners were consistently weighted 9:1 private:public.
> https://doi.org/10.1126/science.1238041
That article doesn't seem relevant. 'Poverty impedes cognitive function' is entirely compatible with 'IQ is highly correlated with wealth and is also highly heritable', and with the claim that private school can't make a genius out of a kid who lacks an extremely high IQ.
edit: maybe I misunderstood, and you only were only pointing out that some potential-genius kids have that potential stifled by poverty. But that doesn't seem to be the original point of disagreement.
https://www.researchgate.net/publication/259207757_Comment_o...
On the first page, the breakdown is 23 kids from public schools, 7 kids from private schools, 4 kids listing some sort of math circle / whatever Arateem is, 5 kids listing universities (home schooled/taking uni courses? No idea). Possibly a few of the private school kids are on scholarship. The 7 kids go to 4 schools.
I cofounded a technical screening startup, and despite our efforts to educate our customers that enter their questions on our platform, 95% of their custom questions are bad questions. They tend to ask trivia questions, that can quickly be googled, instead of asking work-sample questions (which we at TestDome.com prefer). I think they learned that from years of paper tests in school, it is very hard to unlearn.
Just to give you an idea, for testing web programmers we suggest questions where you need to find bugs in HTML code (https://www.testdome.com/questions/html-css/inspector/17629), while our customers would ask questions like "What does CSS stand for?"
is it the difference between understanding a language and not speaking it, perhaps? following a conversation, but not being able to participate in it?
An applying level question could be, "Find the running time of algorithm Y (which was never discussed in class)."
Applying is writing python code with no reference book/manual.
In both cases doing it properly means you are reasonably fast while being confident about the correctness (either that you correctly predicted the program's behavior or that you wrote bug-free software).
The advise against level 5 questions is interesting. I found them to be very useful for answering if a candidate has really applied in the field before, especially comparing two big fields against each other. For example asking if they prefer sql or nosql. Anyone who doesn't answer some variant of "it depends" has probably not had enough hand on with either of them. But regardless if the candidate has a strong opinion or not, and regardless if it's the same opinion as yours, the answer is going to give you endless amount of openers for dialogue and follow-up questions on how they reached that conclusion and in which projects they used such techniques and how it worked out.
As the articles notes, the questions can easily be misperceived as checking if the candidate has the same opinions as yourself, or you becoming outsmarted by eloquence. Since you are yourself asking and evaluating the questions you have to ensure this doesn't become the case, by putting opinions aside and only asking about fields which you yourself deeply understand. Not sure if it might be off-putting for a candidate to get this type of question though, if they don't understand the deeper purpose of the question? Should you give a disclaimer that you are not actually not interested in their final opinion?
After the discovery of dozens of new trans-neptunian "planets", categorizing Pluto as a "real planet", while it probably had more in common with them became problematic.
The best definition that we had to come up with to cleanly separate "real" planets from the others was therefore quite unlikely to include Pluto!
I think the reason founders want introductions to influential people is not because they think that this is what you need to be successful. It’s because they have seen many cases where influential people talked up startups that seemed to be pretty mediocre, and it helped them succeed. And they think “well my startup is at least as good as those other ones, so publicity by famous people will accelerate our path to success.”
They’ve seen press coverage of lousy startups propel them into fundraising successes and growing revenues, even if they never became profitable. And they think “I’d succeed faster if my legit startup had that kind of exposure.”
It’s not that they think these things are a substitute for having a good product. They think (correctly) that having these things will accelerate their growth and somewhat lower the bar to success (particularly if there are network effects involved).
Incorrectly, unless you define success as a local maxima. You usually can’t influence your way to broad user adoption
I know that a few fortunate tweets my startup received drove tons of traffic our way, and likely resulted in the press coverage we received soon after. And that press coverage brought more users, and also the proof points of having been covered by major media. That makes fundraising and partnerships easier, etc.
If pg believes what he's saying in this essay, I have two questions:
1. why did he design a testing system that was easily hacked?
2. why does YCombinator itself focus so much on hacking the venture capital system, as opposed to making great products? [1]
I think the answer to both is that the ideas presented in this essay don't scale. If you can't create a 200 person per year startup incubator that focuses on true learning instead of process hacking, then how could someone do it for a 20,000 person university? Things get boiled down to metrics like grades and valuations because they are the only way to measure/manage the productive output of a large number of people.
The insight that I've had is that as long as large civilizations and organizations exist, there will be easily hackable systems. There is a direct relationship between the number of humans involved in a thing, and the number of opportunities to hack that thing. Understanding and exploiting this seems like a valuable ability.
[1] I was personally struck by how much of YCombinator was (in 2008) oriented around learning what magic words to say to investors and exactly when to time techcrunch launch articles, etc. There seemed to be no time given to "how to make a great product", likely because this is something that can't be taught
If it were simply build the best product then there wouldn't be a need for YC. If it were simply build the best product there wouldn't be a need for marketing or sales or business degrees.
The hacks involved in marking and sales became so important to any companies success that they've over time become entire fields of their own. Same thing with an MBA - essentially a degree to hack business. Regardless of what you think of it, it's clear based on the financial returns of it that it works.
One of the biggest lessons constantly mentioned of startups is how naive engineers are to the business side of things and there is more than building a great product. In this essay PG takes all of that and throws it out the window. Is that suddenly no longer true? My read of it is that he (rightfully) likes YC and as a result doesn't see how it has the same flaws as the systems he describes. Essentially like every parent who only sees their kid as beautiful.
As a university professor, I can at least try. Not at scale -- certainly not for my entire university -- but for my own classes as an individual.
There are several constraints, of course. I have to give grades, and students care about grades, a lot. These grades should measure (to some extent) how prepared students are to proceed to whatever is next. Students have a number of conflicting priorities, and it is natural to concentrate on whatever is most urgent. And students come with wildly differing backgrounds and interest levels (and amounts of available time), and I have to be consistent and fair to everyone.
But, when I grade, I have a lot of leeway to decide what I believe is worth measuring, and measure that. My aim -- which I hope I at least partially realize -- is to ensure that the best way to "hack" my classes is to learn what I hope the students will learn.
I want the obvious response to the above to be "okay, just raise the bar for teaching professors" but, unfortunately, those decisions are usually made by people who are completely ignorant to the process, and who rely on broken systems of teaching evaluation. Those who would be good at making those decisions are busy doing the teaching, and doing it well. Also, I don't personally think we want a system that so micro-manages our professors' pedagogical styles. I get fairly positive student feedback, but so do the professors who offer endless exam re-submissions and whose students have basically zero material retention after they pass the class. The optimal (but impossible, partially due to sheer laziness) way would be for students to self-select themselves into classes with professors that they believe will ultimately teach them more and better material, without having the extra pressure of the effect on their GPA.
(edited with some last-minute thoughts).
Who is fighting at the organizational level for the students?
The problem is systemic. All of the solutions I see, here and elsewhere, describe a small piece of the problem. At best — a few courses in a curriculum.
Plenty of people. Sometimes at cross purposes, rarely with little effect.
Unlike Tolstoy's happy families, good teachers are not all alike. For the most part, I can't envision what a productive organizational-level fight would even look like.
I know professors who have changed the answers during the re-submission time, so students brute-force their way to a failing grade...
It can be taught, but the general principles are weak (it’s basically “how to be an effective futurist”), with most of the useful knowledge being domain-specific.
And the thing with the specific domains is, either they’re not understood yet—in which case everyone is flailing around—or they are well-understood—in which case the market has already consolidated around the companies who can best operationalize the set of techniques required to create great products for that market, and the best you’ll hope to do is to get bought out by one of them, not to compete or replace them.
Because, in the end, having a great product is a multiplier, but so is execution; and execution provides compound interest on its gains, such that corporations inevitably execute exponentially better as they age.
In other words, the only time where you can set out to win a market, and actually have a hope of doing so, is precisely when we don’t yet know how to operationalize the production of great products for that domain.
> YCombinator was (in 2008) oriented around learning what magic words to say to investors and exactly when to time techcrunch launch articles, etc. There seemed to be no time given to "how to make a great product”
https://www.startupschool.org/library
I've watched many of the videos on YouTube. I would say that each speaker consistently emphasizes the need to "build something users want" before making a big press launch, before applying growth hacking techniques, before talking to investors, etc.
They always say that in the first 5 minutes of the talk, and then go onto more tactical stuff.
They do teach you about hacking investors, or maybe more charitably seeing it from their side, and seeing their motivations.
I think it's true that you can't teach the "build something users want" part, so a lot of the advice is lower level than that.
Advice about building things tends to sound very vague and general, and for every rule there's an exception. Probably the most memorable advice is "live in the future and build what's missing", which I think is true, but hard to implement if you're not the right person.
Whereas the hacking tactics are more specific and actionable (although more likely to be out of date).
“Startup = Growth” is pure actionable advice and probably the most useful document i’ve ever seen on starting companies (I say that as someone who has founded 3 venture funded startups and two bootstrapped before that)
To add to that point, take a urine test: people have been hacking it for years. The purpose of the urine test is to see what residual byproducts from certain drugs are in my system. I can hack it by using someone else's urine or imbibe on drugs that have a short lifetime in my bloodstream. Otherwise, to pass the test I have to not consume drugs.
His complaint is that people who pass the tests really shouldn't have. It's like a principal assuring parents that a strict urine exam is given to all teachers only to find out all the teachers who work for you are drug addicts (drug addicts who can hack the pee test).
His reference to private schools teaching students how to hack tests resonates with me as social workers teaching drug users how to hack their pee tests. I take that to mean that if teachers are doing their best but resort to teaching how to hack tests then their needs to be more work done outside of the teacher-student system.
[This is where disparities due to parent's wealth and income come in:] Students who have access to tutors can get the help they in a way having a teacher can't. A teacher can be a tutor, but cannot be a tutor for everyone (an issue with scale).
Having a great product, building things people want (i.e. the "non-hackable" parts of building a startup) are necessary but not sufficient requirements for building a successful startup. What YC tries to teach is all the other parts (how to deal with VCs, what metrics to focus on, how to manage your time, etc.) that are also necessary, but not the core of what makes a good company. These are the "hackable" parts if you will. But you're very likely to be unsuccessful if you only have those hackable parts right, but still don't have a product people find useful and want to buy.
Since you're not posting anonymously, I'd be curious if you could offer more information on what parts of YC you think you hacked, and how your startup turned out.
When he says, "Make a good product", I think he's trying to convey a core and fundamental principle of business that gets obfuscated by all the things you have to do be successful in a startup. That's necessary when teaching beginners.
But he's traditionally had a hard time conveying this seemingly simple concept, and he's always been puzzled by it. You can see it in the previous essays he referenced in this one--the high school one, and how founders over-complicate things.
Though he's talking about this topic as if creating the company is the goal, I'm not sure to what degree he realizes how YC contributes to the effect he's seeing.
Even after getting into YC, I was moderately surprised by how you're still being judged, sometimes explicitly, and other time implicitly. I had batchmates that didn't want to go into office hours, for fear of appearing weak and hence wouldn't get a follow-on recommendation to investors from a YC partner.
Of course, I think PG would say that's stupid. If you go into office hours and fix your problem, then by virtue of being a better startup, you'll be a more attractive investment. Trying to do it the other way is putting the cart before the horse.
That said, the effect is real. Like the Observer Effect in quantum mechanics, by virtue of being a conduit to further funding, founders will turn their attention to the tests. Because if you don't, it's easy to be written off and not be able to get the help that you need.
But so much of YC felt (and probably does more so now) like you're being judged all the time. Which makes total sense to me, because seeking prestige is so fundamental to human nature. That co-evolved with social cooperation in a virtuous cycle.
If pg really believed what he wrote in this essay, YC would be structured very differently. The moment pg allowed some companies/founders to have better access to him than others, or the moment he talked about how some founders are better than others, he implicitly set up a criteria for judging founders and their companies. And people will always naturally feel envy and try to hack that criteria. But a precondition for doing good original work is not feeling judged. So he should have gone on a crusade to completely eliminate all forms of favoritism if he wanted to maximize the odds of success for a modal company. But that's not what he actually ended up doing. At all.
Now the question is, is the essay wrong or is the YC structure/culture wrong. If I were a betting man, I'd put all my money on the former every single time, and I think I'd never come lose that bet once.
I think that's a false dichotomy.
> The moment pg allowed some companies/founders to have better access to him than others, or the moment he talked about how some founders are better than others, he implicitly set up a criteria for judging founders and their companies.
It's correct that this creates a hackable test, but the losses incurred via that might be less than "going on a crusade to completely eliminate all forms of favoritism". This does not imply that hackable tests aren't bad, meaning the the essay is wrong, just that the pendulum can swing too far in the other direction too. Which isn't that striking of a realization.
Yes, I agree. I can't tell if he's ever thought about it. I would have thought Jessica, being the social radar, would have picked up on it. Maybe it just wasn't a high order bit on the list of priorities.
I've come to think of PG as the disapproving father I never had.
It reminds me of a quote from The Elephant in the Brain. I don't remember the wording exactly, but it's something like "if there is a behavior in a large group of people that doesn't make sense to you, not only is it by design, but it's also probably the whole point."
Incidentally, I was also struck by how much of YC was (in 2009) about hacking venture capital. But lots of people from that era now say this behavior is unethical disavow all knowledge that it was ever actively promoted. It's like we lived through different realities.
If I were to steelman The Lesson to Unlearn I think I'd say this. On an individual level there is enormous benefit to learning how to operate without hacking a test. That's how you do great work; that's what original thinking is. On a societal level this seems unshakable. But societies have made dramatic shifts in behavior and social organization before. From hunter gatherers to farmers to city dwellers, for example. So while this is hard to imagine it might just be that-- lack of imagination. The social return to abandoning this system might indeed be enormous.
There's a snake eating its tail situation here.
But if one were to find oneself in a situation where there are no gatekeeper tests, then how would you go about doing great work? And by being able to see it as an idealism, at least we have a target to shoot for, even if we might fall short in practice.
That's not to say "hack the system", but it is to say look deeper. There's value in learning. There's value in the stated objectives; after all, there's a reason they're being called out. But there often are unstated, important systems and objectives to understand and address, too.
I'd love this dream world PG describes where everything aligns, the stated priorities are the only ones, and the systems and cultures and tests optimize only for them. But that's not the world we live in.
Most big tech companies don't care about how good you have been at delivering some value through creating software: they want to see you deliver a very specific type of performance at a whiteboard. Interviewers are given specific math puzzle questions to ask. Interviewees are explicitly told by the same companies' hiring departments that they should aim to hack the system by studying books like "Cracking the Code Interview".
This is an industry that prides itself on supposedly making data-driven decisions through A/B testing. When it comes to hiring people to make those decisions, everybody just plays along to a decades-old script.
I will say though, the problem is one of “standardization” across an organization where it’s too big for everyone to fit in a room.
Suppose you give each team high autonomy to hire whoever they like using whatever “good” process they come up with. 90% of the time this results in good hires. But as you grow, that ten percent of underperforming people becomes large in absolute numbers, and is very painful to deal with.
It becomes a real problem when relatively lower performing people end up concentrated on a team, and then start being the hiring gatekeepers for that team, thus multiplying the number of lower performing hires.
Later you start having institutional problems when everyone starts to perceive that the engineers in Department A are generally better than the engineers in Department B. Engineers in Department A are more likely to leave if they perceive the company is getting worse at engineering - it becomes a self-fulfilling prophecy.
Then you get enormous pressure to come up with standardized testing - aka algorithms on the whiteboard, or some other academic inspired exercise - imposed by higher level leadership that wants to address a genuine problem (skill disparity across the org) but does not know any better way to do it.
I think, as PG points out, there may be a real opportunity to innovate here, and probably a big financial opportunity if anyone can figure out how to productize and scale a solution.
I struggle to see an easy answer, though. In a utopian universe (for a hiring manager) I’d do something like pay candidates to come on site and work for a week, then make a hire/no-hire decision based on that. But I think that is far too onerous for candidates (and a big company) to have legs.
To be honest, if a large company has multiple departments that essentially govern themselves, perhaps they could just give them proper names, as if they were subsidiary companies? Because it seems to me that the mechanism behind the "best performers leave when they feel the average performance of workers in the company goes down" is really bounded by the company name. I.e. people in department of frobnification leave because of the people in department of foobing bars. But people in Frobnificator perhaps wouldn't leave because of people in FooBars.io, even if legally/administratively the two situations were equivalent.
Some places attempt to solve this by saying that you can use whatever vendors you need to solve the problem but often the best place to solve it is the other department since the solution is infeasible without them.
The problem is that often the solution is infeasible with them because the guys who made them good have left and now you only have the lameos.
As to why they can't operate as independent business unit? Well, there's the Theory of the Firm to tell you why that doesn't work often.
To ensure this, someone experienced with interviewing from outside the team/organization, so called bar raiser, is always added to the interviewing team. This person is there to ensure high quality hires and has hiring veto power.
The rest of the interviewers are from within the team, likely under pressure to compromise on quality when they're desperate enough to hire someone, but they can't do it unless the bar raiser agreed.
That's just one data point, and maybe what you're describing is for more senior hires?
At this point it's Kool-aid. In practice bar raisers have so much more interview experience that you can very easily pick them out, but so much of their behavior results in simply weeding out the bad.
I have received LOTS of offers from Amazon over more than a decade (only one accepted at one point), and my friend used to be a bar raiser. In his first interview at a new company he was both rude to an interviewee and flat-out wrong technically on multiple points along with it.
You can't raise the bar if you can't pick the bar raisers.
If this is raising the bar, no fucking thank you.
Actually I did exactly this as a candidate at one place for an afternoon. I left after 2.5 months. :-) In the end I felt a bit sorry for them since they put a lot of effort into the process, being a really small company. On the other hand I really didn't want to stay there. It was technically really interesting and I could contribute but the business was run really badly IMHO
I think you could reduce the 10% to 1% simply by actively firing bad employees, and the easiest way to do that is with a probationary hiring period (let's say 6 months, maybe a year).
I am fully aware of how systemic rot can be, and how hard it can be to remove. The real problem is when you hire a bad manager, because they will not only let the rot fester but encourage it to grow.
I think you could wildly relax hiring standards for ICs and be incredibly successful. What we need are better practices around hiring managers more than anything.
The effect on morale and real productivity of getting rid of the toxic and worst performers is amazing.
I can't name that place without doxing myself, but it sounds like Netflix has a similar culture. Some people love it. Some people hate it. But I don't think many people would say they have an issue with employing less-than-stellar employees.
Of course this is only "solving" (it's certainly not perfect) the problem of hiring for engineers, which most of us are on HN (at least I assume, but certainly more engineers than engineering managers). It does nothing to address the issue of hiring bad managers. I have no solution for that, other than to make sure the first manager is excellent. A players hire A players because they want to do the best work they can, and will go out of their way to hire people better than themselves. B players hire C players because they just want to make themselves look good. Once you have a bad manager, the assumption is everyone beneath him will be as bad or worse (obviously there are exceptions).
Leadership, from the top down, is absolutely responsible for the mentoring, training, and environment created by the management. None of the systemic problems with & induced by management is new. It comes down to: does leadership actually care enough to do anything about it? Companies institute all sorts of formal stuff like internal surveys and NPS scoring and so forth but it always comes down to trust & communication and then follow through. And the follow through is the clearest way to create trust. All of the things that even well intentioned leaders spout, if not followed through, quickly erodes and ultimately destroys trust. So, just like with kids, don't make promises you won't keep.
Toxic systems are always the responsibility of the managers and that's from the top down. The systemically perverse issues created by people "managing up" is one of the most critical problems that people higher up the food chain need to correct for.
It has a bad reputation as it's often used year on year - it's meant to be done just once or twice to get back on track after years of over-hiring and reluctant firing.
How often do you actually boot someone though? For juniors we have pretty low expectations so they never fail this.
This was not the practice where I worked, but you could have different probation periods for different roles. So someone who is primarily focused on technical debt might have a longer probation period than someone focused on rapid feature development.
Like I said, it's not for everyone. A bad manager can ruin your experience anywhere and fire you regardless. That place was just very forthcoming about the fact that you need to demonstrate value if you want to keep your job.
- More headcount is always better for the manager's prestige and career prospects, as long as those people contribute to project success at least a little bit (and most low performers do, just less than you'd hope). Managing infinitely many people each producing epsilon above breakeven is much better than having 1 rockstar.
- Headcount availability ebbs and flows with the business cycle. It's important to take everything you can while the money is flowing in case you might need it during a contraction. It's important to have nonessential staff on hand during a contraction so you can satisfy your X% cut requirement without compromising effectiveness too badly.
- As a manager of a high-performing division at a growing company, you will be given an integer multiple of your current headcount, and expected to deliver proportional results with it. If your strategy was "employ only high performers" you will fail, someone will be brought in immediately above or immediately below you to "help out," and within six months you'll be "spending more time with your family."
- The hiring process is hard to mess with. Recruiters put your ICs onto the panels directly; systematically leaning on them to change their votes will probably get the ethics hotline involved. But the decision to retain year over year is entirely yours.
I feel that beyond hiring, our entire society has a problem with tails of distributions. There is this strong movement behind standardization of everything everywhere to minimize the variance, but that not only eliminates both tails, it also lowers the mean. So e.g. here, in order to avoid exceptionally bad hires, companies not only forgo the chance of getting exceptionally good hires, they also lower the quality of a typical hire.
I'm mediocre, at best, at these tests. I don't come from a traditional CS background (started as an EE). I tend to take unusual, hybrid approaches to solving problems; often incorporating elements of new-fangled tech with patterns that have been around for thirty years, and I've found that people get VERY uncomfortable with "thinking outside the box."
I also tend to take some time arriving at the final release, going through iterations of improvement. My first bash is usually fairly naive and clumsy. It works, but not so well. I make each iteration improve on it, maintaining that working software throughout; so there's always something working and testable.
Despite all the aspiration to "disruption," it seems that people don't like to step out of their comfort zones.
Making software that SHIPS is a learned and earned skill, and one that I believe, can be most effectively demonstrated with portfolios.
When a designer goes for a job, they bring with them a large black case. It's filled with their designs and working drafts. Much of the interview consists of the designer going through this portfolio with the hiring manager; discussing each example, and talking about why they did this, or why they didn't do that, etc.
No design manager in their right mind would ignore that portfolio; instead, throwing a matchbook on the interview table, and asking the applicant to "Draw Spunky."
I'm actually shocked that there is so little importance placed on software portfolios. A portfolio represents fairly substantial work product. That's why I find open-source contribution work to be so attractive.
This is actually frustrating me quite a bit because as I look for new jobs people like to ask, in detail, what I worked on, and I can’t really tell them.
That was a big reason that I learned to draw people out when they interviewed. I asked them to "tell me stories" about projects they worked on, without getting detailed about specifics. In my experience, they were always able to demonstrate plenty of enthusiasm and creativity without revealing the family jewels. A half-hour whiteboard test would have been completely useless. I was usually able to establish a fairly comfortable level of tech knowledge fairly quickly, and the bulk of the interview was really about how well I thought they'd fit the team (NOTE TO SELF: Don't go for homogeneity).
However, this isn't a problem that's unique to our industry. My father was in the CIA, and I never found out until just a few years before his death. Lawyers have legal cases they can't discuss, doctors have medical cases they can't discuss in detail, without violation of HIPAA, etc.
Somehow or another, these folks are able to interview for jobs that often have a far greater risk than a line programmer on a major initiative.
True, some are screw-ups, and that doesn't become apparent until after they are hired, but the same goes for an engineer that can ace every problem on HackerRank, but goes all to pieces, when presented with 100 KLoC of spaghetti code, and told to fix a problem (exactly what happened to me, on one of my first programming jobs. 100 KLoC of 1970s-vintage FORTRAN -I fixed the bug, but had to hold my nose, while doing it. I later learned that this was what everyone did).
What bothers me most, is that a chief reason that I'm given for these tests, is that companies want to find people that can come up with innovative and unique solutions, yet they seem to actually have the opposite effect; filtering for people that only come up with standard solutions.
I remember once doing a "take home" test that asked me to apply a third-party library. I did what I always do with dependencies; I encapsulated it, and that seemed to totally freak out the interviewer. To this day, I have no idea why they lost their bottle so badly over it. My solution worked extremely well, and it also gave the problem tremendous "future-proofing." That's why I encapsulate dependencies.
I take that as part of my job in interviewing the the company/interviewers. I will very deliberately give a more radical, non-standard but correct answer precisely to see if they fit into that broken/fixated mindset and see how they deal with being challenged. The dynamic of how they deal with that shows everything about not just how clever they are but how intellectually honest, curious, and emotionally mature they really are.
Then I read "Soul of a New Machine" by Tracy Kidder, a book that documents the design process of a minicomputer in the late '70s. The idolized manager of that project wants to hire young engineers fresh out of school so that he can mold them according to his wishes and have them working all-nighters. If that's your hiring principle, you're not going to get anyone with portfolios.
So I suspect it's an industry sickness that's been turned into a virtue over time.
Maybe I won't become a specifically-software person.
But everyone can do a whiteboard interview!
After a couple of these fiascos, I just gave up, and decided to start my own gig. I'm fairly good at what I do, and it really is too bad that it doesn't show in half-hour theory tests.
I do have a pretty massive portfolio, though. Comes from doing open-source software for over twenty years. I was paid to manage; not develop, so I needed to keep my tech chops up in other projects.
I also realize that this is not a typical thing. It's just my journey.
I thought it was funny that I mentioned this once, and someone insisted that I could have faked the portfolio.
If someone can fake ten years' worth of commit history, in dozens of repos, and tens of thousands of lines of code, as well as hundreds of pages of documentation, and dozens of articles, you should hire them immediately, because they are a genius.
I don't doubt that you did, but I doubt that everyone at your company places the same effort as you do. At my current company, 90% of interviewers half ass their interviews. They don't have much incentive to care, so you want interview questions with consistent standards and produce some signal with minimal effort. Algo questions fit this bill perfectly.
Good interviewing is difficult, it may not necessarily be because they don’t care.
Some of it I would like to share (and some of it not!), but it really isn't my choice.
(Also - because I am fairly cross-disciplinary, the code for a controller to stand up a 100 ton rig, is not the same as the code to monitor the growth/feed/heat/light for algea in vertical tubes. And neither is likely to be asked of me for projects having to do with creating controllers for missions related to diabetic control or scooping up space debris. The ability to analygize only goes so far.)
Of course, difficult whiteboard interviews that require studying to pass have the same problems probably and give less useful information. So, :shrug:.
I just feel that it's rather self-destructive to ignore those rare instances where they exist.
I agree a full blown project over 24 hours would be better, but it's more costly to create questions and score them (if you want to do it in a way where cheating is hard/impossible). I've seen startups do this, and it works well for them, but might not be scalable for companies that hire thousands of employees each year.
People would look-up solutions to that exact problem; it's very hard to come up with a unique problem that was never asked before, and questions leak pretty quickly. (especially at larger companies)
The goal of a test is to evaluate whether you'd be a good employee; I agree algorithmic questions are not representative of day-to-day work, but evaluating your friend's ability to help you is out of scope. Companies want to hire someone that has decent programming skills; you can't rely on other people to solve all your problems, you need to have a minimum level of skill.
[again, playing devil's advocate here; we all agree the process is suboptimal, but let's not ignore the negatives of some of the alternatives suggested here]
Note I agree we have the same goals to hire people that can do the work not people whose friends can do the work but I'm not convinced that the typical interview puzzles (find the longest segment of an array that contain 3 values) has anything to do with actual work skills. Maybe it would be better with more typical tasks? If the person claims to be back end engineer ask them to write a query to select all people between the ages of 30 and 35 in Nebraska? Maybe if they are front end ask them to implement a select/option UI?
The only difference between these people and those who pass normally is that the latter group looked up the answers to the problem before you even asked them. Nobody comes up with things like mods of dfs or n-pointer problems on the spot. These problems were the subjects of doctoral dissertations the first times they were solved. We pass them because we've seen problems just like it. The system has been gamed and doesn't work.
I have had interviews where the company has done that. I would rate all of those interviews to be very high quality. No nonsense trick/leading questions. Just simple: can this person actually program? What does he understand? What doesn't he understand? I felt like it was obvious what I could and could not do...there was nowhere to hide.
Btw, in terms of investment, this seemed far cheaper for the companies too. What is cheaper? Arranging an hour-long interview where you go through a simple problem that is focused and will clearly identify knowledge. Against an hour-long interview where you probe someone randomly about their life and projects that the interviewer has no idea about (funnily enough, no matter how good you are communicating ideas...interviewers almost always get it wrong).
I think you've got a lot of this right (disclaimer: we've built the product I think you're describing)
The don't think the most important problem is standardisation though, it's observability/instrumentation ie. if you don't measure what's working, you can't improve things.
The very best tech companies measure quite a lot, and often look back at their hiring processes in the event of a mis-hire to figure out what went wrong and how they can avoid the same happening in future... but even then they only do that in exceptional cases because it's done fairly manually. That means they have low statistical significance and a stuttering cycle of learning.
I believe they should be constantly looking at what's working well, for every hire. So that's what we built.
Once your hiring pipeline is trivially visible, a lot of these questions go away. You can see what's working well and try new things in safety, you can optimise with your eyes wide open.
One thing we did straight away was to deprioritise CVs and replace them with written scenario-based questions relevant to the job. If managed properly that takes your sift stage from a predictive power around r=0.3 to a performance we find typically above r=0.6. Far fewer early false negatives makes your hiring funnel (a) less leaky, (b) more open to pools of talent previously ruled out by clumsy CV sifting, and (c) potentially shorter as the improved sift accuracy allows companies to consider dropping their phone interview stage(s)
Our NPS rating for HR teams is currently running at 85, and MRR churn is under 1% so there's clearly some value to the approach.
* conditions apply.
I worked at a FAANG company and saw several people get let go for underperforming. I have no idea if they could have been productive doing something else but they were clearly not productive at the tasks assigned them or that they chose to do. I also saw several other people I thought were a candidate for going down the same path before I left. Whether they got moved to something they were more productive on or whether they were showing value in other ways I have no idea.
The point being the hiring system didn't reject these people. My guess is they either got lucky like me, easy questions, or they are good at answering the questions but not good in actual production.
For example one person that was let go was clearly smart and could make the code but they would refactor forever looking for perfection and end up taking 4x - 8x longer than others around them. Maybe in some place that's a plus but for our team we needed to ship and this person was not able to prioritize shipping and after several attempts they were let go.
Another had a task that they were taking a long time on. Others on the team gave them the benefit of the doubt that it was harder than it looked but when they were finally let go it turned out it was not harder than it looked and was finished rather quickly.
Another wrote bloated obfusticated code that seemed to be 6x the number of lines it needed to be. Maybe they could have been helped via code review.
Yet another was one I interviewed and they passed the "can write code test" but in the entire time I was there I don't think I saw them submit a single PR. I have no idea what was filling their time.
I don't know if that ratio of unproductive candidates is better or worse than anywhere else I've worked on average. Pretty much every place I worked before FAANG was 100 people or less and my team 30 people or less and those teams were 10 engineers or less. There were always 1 or 2 engineers in the company known for being slow or writing bad designs. Sounds about the same but I didn't measure.
Beyond that, in my opinion, nearly anyone with average programming experience can hack a typical programming/whiteboard interview, so these kinds of interviews are definitely not the best way to find the best fit for particular roles. It's a good way to filter out the inexperienced, for sure, but it's a pretty low bar to set, especially if it's a company looking for top tier engineers, people with the ability to see the bigger picture.
It's also likely that a great engineer would pass the test with ease, but because their skillset and abilities have been inaccurately assessed due to the poor interview process, the company could have initially assigned them a much more effective role where both the company and the employee could have benefited to a much greater extent, but didn't. Overall, relatively speaking, the "hackable test" approach wastes time and hurts everyone in the long run, both employers and employees. A little investment up front with more personalized, specific interviews can go a long way.
I think we should start refusing these impractical interview processes, or at the very least, from an interviewee standpoint, turn down jobs (if you can) and let the potential employer know that their interview process is the reason.
I don't know the solution to this problem. Eliminating tests whatsoever and just talking with the candidate about his experience and probing his knowledge on different topics is not efficient either. There are a lot of talking heads around who when given a simple task fail miserably.
We as a tech community need to come up with better ways to assess other people's competency while also making sure those people fit within our company's culture, work efficiently with others and after all create value. This is a hard problem. So, we try to simplify the problem by imposing the "proven" way of finding such people -- give them arbitrary tests and hope they pass them.
Can you give me an example of a forward thinking company according to you?
Can you list all 'forward thinking' companies for the benefit of all of us?
I agree with your conclusion that all it does is filter out inexperienced/incompetent candidates. This is valuable, but to me it's the "first question" - can they code?
I don't think there's really much value between an "Amazing coder" and a "can get the job done" coder. Very rapidly you're trying to answer much softer questions like "does this person work well others?" or "how fast can they learn new systems?" and there isn't a good way to do that.
The interview question I usually give is algorithmically very simple. There's no linked lists, no graph theory, none of that. What I actually look for is mostly how they interact with me and how they go about solving it.
There's a pretty straightforward gotcha that more or less everyone hits. If the candidate hits the gotcha and then keeps piling on special cases without ever taking a step back, that's a big minus. If they keep doing it even after I suggest there's a simpler solution, that's worse still. I much prefer somebody who takes a step back but can't quite figure it out without a little more help than one who just remains confident in their solution no matter what.
Yeah it's inherently shitty that I have 45 minutes to figure out if this person will benefit the org for years to come, but there's a lot more signal there than just "Did they open their CS 201 algorithms textbook in the last few week?"
I think the opposite is the case. Every industry has some sort of test to get into it. But the other industries are the most centralized and hackable ones. The tech industry is doing the best job of any of the top industries of having an application process that is open to skilled people who don't match the "centralized" standard.
Your grades in college, the experience on your resume, those are "centralized" tests. You have one resume and you send it out to everyone.
When companies give programming tests to applicants, each company is free to measure the applicants' skill in any way they see fit. One might ask math puzzle questions, but another company might give take-home Rails projects. And companies are free to just do a "soft" interview and look at your resume if they want.
Facebook hires far more software engineers with no software engineering education than Chevron hires chemical engineers with no chemical engineering education.
Plus, starting a company is open to anyone with a credit card to open an AWS account. No interview required. Just make something people want.
Today, unless you are doing things at scale (tiny fraction of startups), you don't need to know how to make things run in the most optimized way possible.
"Your initial Facebook interview is coming up and we want you to ace it!
Here are some tips and resources so you know what to expect and can prepare adequately. Preparing is key.
Our initial interview determines whether to continue with a full series of onsite interviews. This initial interview is primarily a coding interview that will take place between you and a Facebook engineer.
It's important for any engineer to brush up on their interview skills, coding skills and algorithms. Practice coding on a whiteboard or with pen and paper, and time yourself. Preparing increases your odds significantly! Below are two helpful links to a Facebook Interview Prep Course led by Gayle Laakmann McDowell, author of “Cracking the Coding Interview”. Use password FB_IPS to access the videos. Cracking the Facebook Coding Interview - The Approach Cracking the Facebook Coding Interview - Problem Walk-Through This article provides much more advice about how to prepare: Preparing for your Software Engineering Interview at Facebook
Here's a sample problem to get started.
Sample Problem Write a function to return if two words are exactly "one edit" away, where an edit is: Inserting one character anywhere in the word (including at the beginning and end) Removing one character Replacing exactly one character Most importantly, you can view this and other Facebook sample interview questions and solutions here.
Want more sample questions? Try HackerRank, LeetCode, and CodeLab. Be sure to practice questions in a variety of subjects and difficulty levels."
(Once in a while maybe a complete genius takes the test and reinvents the good algorithm on the spot, but only gets the same grade as the diligent student, for his effort.)
Too many basic coding screens seem to rely on some algorithmic insight. I would rather give a candidate a complicated-but-straightforward problem (like a super-fizzbuzz that takes half a page to describe,with a bunch of complicated rules and exceptions to those rules) because that’s closer to what a real world programming task looks like. No room for a clever insight, just translate the problem statement, carefully, into code, and test whether it runs correctly.
To make it even closer, you'd have to have people coming in at various points with contradictory updates to the task, impossible requests, an overbearing managerial diktat, etc.
I've been writing code my entire life, and I've been employed as a software developer for longer than I'd care to admit. I don't have a quick answer to that problem. I know there's an algorithm for computing the "distance" between two strings, so this is probably about that. I have no idea how to write it, and it would never come up in my job, and if it did I'm sure I'd import a library to do it.
After searching it, this is what they're talking about https://en.m.wikipedia.org/wiki/Levenshtein_distance
That means my only chance of getting this problem correct was either:
1) I prepared for it by studying the bank of questions and memorized the formula for Levenshtein distance.
or
2) I considered this problem for the first time during the interview, was immediately struck by the same insight that mathematician Vladimir Levenshtein had in 1965, and developed the algorithm on a white board in real-time.
At least they're transparent that if I do lots of HackerRank, LeetCode, and CodeLab then I'll be able to pass their test. I suppose that's better than only those "in the know" realizing how to be employed at FB.
I really hope I never have to do another job interview again.
Yes, this is hackable, but relying on resumes like every other industry (which is implicitly relying on hoop-jumping and college prestige and GPA) is a much worse form of hackable. This levels the playing field and is a signal for (1) how smart someone is and (2) how much they prepared for the interview, both of which are weakly related to the job. Further, people who excel at algorithm are very unlikely to be bad at the kind of work at big tech.
It's not perfect, but finding something better at scale is hard.
It only requires knowledge of looping, conditionals, and how to work with strings in your language (e.g, indexing, finding length, maybe substrings, etc.).
This is like a less insulting version of fizzbuzz.
People are way overcomplicating this. Any professional programmer could solve this.
abc , adc
a - 1
b - 1
c - 1
after second word loop
b - 1, d
Regardless, I was also thinking Levenshtein is sub-optimal, and that you can probably solve it in O(n+m).
Even if you go for Levenshtein first, you should identify a simple modification to stop calculating the matrix early if the distance is necessarily greater than 1. Get a bit fancier and you can just 'go down the diagonal' and greatly optimize the algorithm.
In this case, all we need at each step is to look at one character plus one lookahead character from each string. We’ll call them a and b for string 1 and c and d for string 2.
If a and c are equal, advance.
If a and d are equal, delete c and advance.
If b and c are equal, delete a and advance.
If b and d are equal, change one of a or c so they match, then advance.
Once you use up an edit, the algo changes to “a and c must match” for the rest of the string.
Edit: terminate the algo when one of a or c is a NULL char. If they are both NULL, the string passes. If only one is NULL, then your one edit allowance was not enough to make the strings equal.
Also, rather than “deleting” a char, it is sufficient to simply advance that pointer an extra step.
> Please solve it without division
This is just an absurd constraint…
> write a depth first search and explain its time complexity.
Part of our test goes "here's a function, in plain English what does it do?", and then "there's a few bugs in this implementation, what are they and how can they be fixed?"
I think there's a lot of alternatives to traditional algorithmic questions that just don't usually get considered. This one I liked because it involves things you'd actually do day-to-day: Reading other people's code, figuring out intent, and finding/fixing bugs.
I'm pretty sure that didn't come up until at least my 2nd or 3rd year - which was 25+ years ago (and I've never had to implement it by hand since.) You seem to be biasing your selection criteria towards "new/very recent graduates"?
My experience with former BigCo engineers I've had to work with is that 1% of interviewees can actually confidently design a relational data model from scratch for a weekend MVP of reasonably any major web service you could think of. If they can't even do that (which is basically among the first things you do as an engineer standing up a service from scratch), how do you trust that they have the skills to design and implement even more complicated systems?
See, I just took what you said and replaced it with a couple of key phrases. But the truth is, when it comes to designing and implementing even more complicated systems, my passage (while reductive and still arguably wrong) makes a lot more sense than yours. I'm not saying this to pick on you, just to say that you probably want to revise what you say at the very end. Instead of "how do you trust they that they have the skills to design and implement even more complicated systems?" you should really be phrasing what you're looking for as "How do you trust that they have the skills to maintain and build efficient modules at Google scale?" because there are plenty of "even more complicated systems" The truth is that the architectures that consist of the grunt work of what you do scaling a company from $0B to $100M or $100M to $1B or $1B to $10B that are by and far more important than a depth first search. The fact that you leapt to do that rather than something more widely applicable shows exactly what's wrong with this mentality. It's not that you're incorrect; you're not even wrong. You never got to the point of knowing what the right question is.
So far I've been very fortunate that recruiters reach out on LinkedIn and I rarely have to do such time wasting activities as they vouch for me.
Recently dealt with a company who pays parity (everyone makes the same .. no woman, man, etc can negotiate their worth), had five to 6 interviews and 2 to 3 coding/design challenges. WoW I guess they are looking only for a subset of talent who would do all that. People might do that for a FAANg company or when jobs in the field are scarce, but this was no FAANg company and thankfully there is still a good amount of demand!
The frustrating thing, is how easy it is to hack the test. After one particular interview, I remember talking to a friend about how hard the technical interview was. He told me that he had already seen the problem and knew it would come up because he bought leetcode premium for the interview. Kind of frustrating to spend hours and hours learning data structures and algorithms when the real key to success is getting lucky and memorizing the problem before hand
Contrast that with our industry where a bright kid out of nowhere can study for the (imperfect, hackable, dehumanizing - I agree!) interview and have a realistic shot at that job at FAANG. At least I wish it still works this way, although of course it helps to be a Stanford graduate!
PG specifically points out that success within a large tech company is predicated on bogus hackable tests. So it makes sense that their selection criteria should also be a hackable test. The better you do on that artificial selection criterion for admission, the better you're likely to do on the artificial selection criteria of internal career progression. So in this particular case, the hackable admission test is a good proxy.
I don't know how it was at Stanford, but at Berkeley in Physics there was a series of big honking tests ("Qualifying Exams") that dominated our attention the first two years. You were expected to know all of undergraduate Physics. Fail it, and you're out.
Grad students don't know any better. "How do we hire the best people? I know - they'll have to pass a big hard test, just like we did."
Then everyone else goes, "How do we hire the best people? I know - we'll do it like Google does, they seem pretty successful."
Ugh.
No, I don't know that it really went like that. I just suspect it.
I know big companies do a lot of AB testing with interviews and change their approach over time (Google stooped asking why are manholes round and looking at AST scores because data showed them those things had almost zero correlation with performance).
That realization / confirmation freed us four engineering cofounders from worrying about some “growth hack” we never learnt, and helped us focus on prioritizing our product development through talking to users. We then used Sean Ellis’ “very disappointed” survey methodology (which Rahul @ Superhman recently wrote about) and used it as our North Star to find the few features to focus on polishing.
We got acquired by Slack in 2015, and built out Slack Calls. I left last year, and I’m now in the final stages of launching a new product around super-charged meetings for remote teams (picking up where Screenhero / Slack Calls / Zoom left off), and am using the same principles again. No hacking of tests, just building a product that people want, and using their feedback as the only valuable bit of data in determining priorities. Thanks PG and YC (and Sean Ellis!) for startup principles and methodologies that have stood the test of time.
So people who are good at tests get ahead. We get that you're not interested in grades. We all know people who play the game and win, not sincerely engage with their job or education or whatever.
But I'd like to hear from pg / other people in YC how they think they are "hacked". What do their successful applicants actually optimise for when "delight" and even "growth" are just as subjective as "learning"? After all they fund lots and lots of companies and founders, many of whom make no returns. How do founders keep their funding "success" long past the point it was deserved?
So there's still a question what companies should do until they reach this growth possibility.
First of all, once you're 55, it's easy to say stuff like that, because you won't be tested anymore. And I think there is a good chunk of survivorship bias: "Hey, I made it without worrying about tests, so you can, too!" Although that is probably not true for a big chunk of the population, especially once we look at non CS people.
Then I think, he is fundamentally right. Grades shouldn't matter so much, it is about what you learn. And I like to approach things that way too, but in the end, I always have to study for the tests as well.
The problem is exactly how deeply it is ingrained in everything. If I just stop caring, it doesn't really help. I will just end up in a worse position, since everyone around me still cares about test results. The people that need to change their mind are the people that use tests as measures of qualities that they are not a good measure for. I think a lot of people are falling thought the cracks because they don't fit the expectation of a HR person close enough.
In the end, we will always have to have tests. A completely individualized assessment of peoples qualities or fit for certain roles just doesn't scale. It would be great though to come up with new ways of testing that are maybe more in line with actual learning. For example, it would be nice to have tests at university where I can google things, just like in the real world.
I am lucky enough to be able to work in the booming tech sector where there are so many jobs that it is fine if I don't work towards a test.
You can you use "test" as a proxy for the impression you want to make on your boss and peers. Maybe even friends and family.
It's up to you to redefine that test (or create a complementary one) based on your own goals and your own timeline.
One of the best professors I had at university for a Linux course always said: “If you don’t know something, google it!”
He backed up his words by allowing googling on the actual coding part of his exams. You had to understand the underlying concepts but you never had to memorize syntax. One time, he had us write a small program in C that involved threading during a proctored timed exam. Everyone ran out of time for that part.
His course and exams were _rigorous_. You could never cram for his class and expect to pass and indeed I know of one student repeatedly failing his course.
This was one of the courses in university where I gained the bulk of my useful knowledge and practical skills I use to this day. I can count the courses that were of similar quality I took at university on one(1) hand. The rest of my degree involved hackable tests, apathetic professors and were massive time-wasters. I learned nothing from them, obviously.
The good professors made me realize how bad the rest of my degree was and how much of a racket higher education could be.
I look back at my undergrad days and wonder how much more I would have learned if I had not focused so much on grades. I’m still actively trying to unschool myself.
When I studied law, all our exams were open book. Bring whatever you want. I brought casebooks and printouts of legislation and important cases. Others also brought headnotes and volumes of Halsbury's (a legal encyclopaedia).
None of this helped if you didn't know the material well enough to identify the legal issues in each question.
Which is why I failed Torts.
My machine learning professor who was from CMU would allow us to put anything we wanted on both sides of one piece of printer paper to bring to exams.
No amount of notes helped if you had no understanding of the algorithms and formulas.
I almost failed that course too, but it was also one of the cornerstones of my university career. It made me realize how much I had been optimizing for the wrong thing: grades instead of understanding.
To be fair he did say "For me, as for most students, the measurement of what I was learning completely dominated actual learning in college. I was fairly earnest; I was genuinely interested in most of the classes I took, and I worked hard. And yet I worked by far the hardest when I was studying for a test."
>I think a lot of people are falling thought the cracks because they don't fit the expectation of a HR person close enough.
In retrospect I'm sure this was the case for me. It's probably why I ended up starting my own business.
So you had to read and understand each of the upcoming 3 papers because there was a random chance you would have to present one of the three.
The result is that you eventually read all of the papers well enough that you could give a 20 minute presentation.
There were no grades. It was pass/fail.
It was one of the best classes I ever took.
Easily the most stressful final I've ever taken.
Two important things here: * Grade inflation makes for happier students and counterbalances the value of test hacking. Today there are very few schools that don’t do grade inflation. Without considering grade inflation, much of this essay is simply Graham entertaining his own nostalgia. (I’m not necessarily a proponent of inflation but it’s a key phenomena missing from the essay).
* Good teachers hold lots of office hours and give good feedback through those offerings. Email support is also a lot more popular today. You can also play games like test corrections, rough drafts, etc. Students don’t learn well without feedback, and most teachers who don’t invest in it won’t succeed today. This focus towards feedback in modern teaching is at odds with the education community profiled in this essay.
If there’s one lesson to un-learn, it’s to forget about giving attention to non-constructive papers that lack evidence. While this essay gives a thorough criticism of tests, it offers no concrete alternative. And it offers no grounding for the claimed ‘good’ student who ‘focuses on valuable learning.’ Graham, like any other VC, seeks to upwell sentiment in the interest of controlling it (e.g. creating an investment asset out of it). Reading this essay doesn’t teach you anything about hacking unless you recognize that the author is trying to hack you himself.
Just because you made waves somehow or filled your pockets with money doesn’t mean your time was ‘better spent’ than trying to make A’s in organized education. School can very well teach you how to make better outcomes.
But, yes, school steers youthful energy away from making change... err making money for VCs.
We had a really interesting conversation. I've been a middle and high school math teacher most of my life, so this is quite familiar ground for me. Over dinner I asked him how he would go about measuring whether a kid is "above grade level". He got really animated and started to tell me what he thought second graders should know and be able to do, and how third graders should expand on that, and what he thought he'd start to learn in fourth and fifth grade.
I told him I could start to teach him some of the operations and skills he'll learn in the coming years, but that it probably wouldn't mean a whole lot. Instead I asked him what ideas he's already familiar with that other kids his age don't know about, because we talk about math almost every day. "I know what phi is, and pi!" Yes, and we've also talked about limits, calculus, negative numbers, fractions, decimals, and he's looked at pages of math that's well beyond my understanding as well.
I told him that when he gets to middle and high school classes, he's going to see higher level math and it's going to sound familiar because we've already been talking about the ideas for years. He's going to watch other kids fail and drop out of those classes because the ideas and the skills will be too new and too difficult for them to take in. I said we're probably much better off just continuing to talk about fun and interesting areas of math for the time being instead of worrying about "grade level".
That is wrong – or, at the very least, incomplete. "Build it and they will come" is the dream and misconception of every programmer who's talented at software development but not at sales. Hell, this is the number one advice of every business how-to book ever: You can't rely on people finding out about your great product by chance, you need to put in the work and sell it!
> Then people will not only use it but recommend it to their friends [...]
That's not how it works for B2B software, and only rarely for B2C. Usually, if a product is successful, then as a result of good marketing (a good product ist somewhat necessary, but not sufficient).
Or, in other words, a lot of successful business - including startups - is exactly the "test hacking" PG urges founders to unlearn.
I'm honestly surprised by this essay arguing that "hacking the test" is the wrong approach in startups/business. "Hacking the test" is essentially what your marketing is supposed to do. These "non-authoritarian" tests like selling things are just as hackable as school tests; you just have to discover how the system really operates.
The whole authoritarian/non-authoritarian split doesn't carve reality at the joints, IMO. Football match is really an authoritarian test - it tests who wins under the game rules, which are given from the top. It may be hard to hack, but that's because passing that test is usually synonymous with the terminal goals of the test takers - i.e. they want to win the game. You'll note however that, once money gets involved, people are sometimes made to deliberately lose games; in these scenarios, there are usually other people involved who are absolutely hacking the test.
The best way I've found to carve the reality at the joints is to talk about terminal and instrumental goals. Learning useful things isn't the terminal goal for most students, getting a good career is (and/or not pissing off parents by getting bad grades).
So I'm ultimately surprised that PG argues that building good things, not hacking the test, is how you win the startup game - I would think most of the startups have products as instrumental goals; the exit is the terminal goal. And building a great product isn't the best way to achieve that goal.
One might say that, by analogy, this is like saying you cheated on a test by studying. “Learning how the system really operates” isn’t hacking; it’s the thing you’re formally setting out to do as a business!
In general yes, the product has to meet some minimum bar of decency. How high? Depends on a lot of factors. Note the popularity of business models involving some kind of lock-in - e.g. network effect, holding data hostage. Those are all attempts at hacking around the need for a decent product.
Depressing, if true.
I would hope that most people have a product that they want to build, and more money is an instrumental goal that lets them build it better. For example, I'd expect someone who loves to code to have software they want to write, and want enough money to allow them to do so—as opposed the executive who loves money and views writing software as the best way to get it. I'm not convinced that any great work was ever crafted by someone who was primarily crafting for instrumental reasons.
But you're probably right, and that's probably a big part of what's so toxic about startup culture these days.
(1) The scripting language Rexx.
(2) The text editor KEdit with macro language KEXX much like Rexx.
(1) and (2) are far and away my two most important tools in computing, from IBM mainframes to PCs, DOS, OS/2, Windows, and my startup on Windows 7 64 bit Professional on an AMD FX-8350 64 bit 8 core processor at 4.0 GHz with Microsoft's .NET.
E.g., I typed in the 100,000 lines of text for the software for my startup using KEdit and never an IDE (integrated development environment) -- KEDIT worked FINE!
(3) LINPACK.
(4) Microsoft's PhotoDraw of about year 2000.
(5) Robocopy.
(6) Western Digital's Data Lifeguard Tools and their backup and recovery software.
(7) D. Knuth's TeX (not LaTeX, just TeX).
Ha! We use robocopy at work because IT refuses to fix our internal network. We can reliably reproduce an NFS mount dropping off the network around the 12th GB of a large file and the only answer is "stop bugging us and use robocopy". All it does is turn a 30-40 minute process into a 3-4 hour process.
I don't get ms. There's a good kind of copy and a bad kind of copy. Why is the bad kind the default?
In every modern OS, the system packages—and most installable first-party packages in a package ecosystem—are there to make the (legacy) software you might install “just work”, not to provide the best experience for users using those packages directly.
Users, for their own needs, are expected to install their own tools and runtimes, or to consume collections of them through “toolchain environments” provided by software like IDEs.
I'm not sure what is available on Linux but maybe the same thing.
Rexx can be used as a macro language for nearly any software, but I use it just for command line scripting. I have 200+ Rexx scripts I regard as serious and write more for one-shot applications a few times a week.
E.g., I have several Rexx scripts for running Robocopy, and each script has VERY carefully worked out, tested, etc. options.
One Rexx script I leave running in each Windows command line window as a shell: It's useful but dirt simple, just reads the command line, does what it can do and otherwise passes to Windows. In particular with that shell I have the cutest little file system tree walking, jumping, etc. commands.
One of my favorite Rexx scripts I invoke with an icon in the UL corner of my screen. This script moves the windows, maintaining the current Z-order and X position order, so that the UL corners of the windows are on a line from the UR of the screen to the middle left. This way at least a little of each window is visible and can be made fully visible without moving any of the other windows and still leaving at least a part of all the other windows visible. To me THAT is how to arrange screen windows. So, yes, Rexx has a little API to some of the Windows API.
In any command line window, Rexx can do good things with the environment variables of that window. So, I initialize each command line window with environment variables like
mark.hn=C:\Users\user1\DATA05\TOPICS\MEDIA\HACKER_NEWS\
So, that environment variable has the file system directory I use for HN posts, etc. Then via my little shell, I can type
g hn
(go to hn), and my little shell looks up environment variable MARK.HN and makes that tree name the current drive/directory.
Apparently when using an hierarchical file system to implement a taxonomic hierarchy of subject matter, some of the leaves of the hierarchy are wanted much more often than the directories on the path to that leaf. For such leaves, say, x, my little environment variable MARK.x work nicely.
It's common for me to have a file system directory tree name in KEdit and then want to open a window with that tree name the current drive/directory. Okay. In KEdit I put the tree name on the Windows system clipboard, and then my little Rexx shell command
g
makes the tree name on the clipboard current. Dirt simple to implement and darned handy.
There's lots more in my little command shell, and it's easy to implement more.
There's a book on Rexx by Mike Cowlishaw, and it's nicely done, but Open Object Rexx comes with some nice PDF files that might be documentation enough.
Rexx is elegant. E.g., it's a total sweetheart for its string manipulation facilities. The only data type is a string. If the content of a string is the decimal digits of a number, then can do arithmetic on it. Yes, can ask if the content is a number.
Built-in quite nicely are stem variables, essentially collection classes or, if you will, AVL trees. So, for variable i can have
array.i
where array is the collection and i is the key. If have strings i, j, k then can have a three dimensional key
array.i.j.k
Powerful. Easy. Elegant!
Yes, there is a statement interpret where can enter a string and have it executed immediately. So, e.g., could have some code that constructed a string that was the source code for a do-loop, give that string to the interpret statement, and have the loop executed.
Yes there are some plenty good enough trace and debugging facilities.
Rexx is no toy: For some years Rexx on 3000+ IBM mainframes connected with VNET (at the user level much like the Internet) was the basis of administrative computing in IBM.
Can get add-ons for sorting, TCP/IP, manipulating windows on Windows, scientific functions, etc.
I suspect that on Windows PowerShell is more powerful but harder to use for when Rexx will work.
In time I'll have to get okay facility with PowerShell, but I'll likely never give up on Rexx.
Rexx used to be the macro language for KEdit but eventually KEdit wrote KEXX, their own version of Rexx. KEXX is nice, sometimes works better with KEdit than an external macro language could, and has built-in documentation that is likely good enough, especially if had a start with Rexx.
Rexx and KEdit let me have just simple text as the basis of nearly all my computing and essentially all my typing. Rexx and KEdit are for text like a chef's knife and cutting board are for a cook. AFAIK, all source code is still just simple text!
Long live the kings, text, TeX, Rexx, Kexx!!!
Presumably the 10% weekly growth that YC and PG advocate as the default growth metric isn't expected to be in the absence of sales but the contrary, sales plus anything that works that is ethical and not some ponzi scheme.
He provided examples via the link to http://paulgraham.com/ds.html where he describes stripe founders engaging in zealous activities many would call sales despite being the type of startup that could have leaned back and had ample demand.
It certainly doesn't have the same ring to it, but it's an extremely successful business model.
Well, the amount of marketing required strongly depends on the quality of and demand for your product.
I built a software product and launched it with very very minimal marketing (I emailed a blogger and published it on a mailing list).
People liked my app so much that I had a 1000 downloads within a few weeks (which I consider decent for a somewhat niche app).
Working on that app has since become my full time job, and I do very minimal marketing.
Maybe I could make more revenue if I focussed more on marketing, but to be honest I'm pretty amazed how well word of mouth advertising works.
That sounds like a very cut-throat business.
In my target audience, most people are happy to share how they work and which tools they use. I even occasionally see random people recommending my product online (which makes me very happy)
The real world is very much about optimizing for and beating tests.
It might be producing a specific CV or preparing for an interview whiteboard session to land a job.
It might be socializing and networking in a specific way in order to land funding.
The real world does not commonly reward just being really good at arbitrary things. It's almost always focused on meeting a need that someone else has defined, much like a test.
I could dare say that this mentality flowed from academia to the greater economy. Academic funding has been generally formalistic and test-driven for decades.
It's not the real world - its the human world. The real world is very different, and getting more so imho. Skills to survive the human world are usually useless in the real physical world. It's a distinction I read recently from a philosopher that I lost track of, maybe someone here knows. Her point was (I think) that when someone says the real world, with relation to education, it's not really - it's the world of human systems that are very much open to degradation and corruption, and they can be changed.
is that a hard fact or wishful thinking?
In the context of this discussion, the rewards do not come from "the physical world" but from other humans (money) , so generic mountain-survival skills are irrelevant
When a startup prioritizes feature A over feature B, they are making a decision on what is going to be tested by customers. The startup then optimizes for passing the anticipated test.
If your ambition is being successful on your own, as an academic, an entrepreneur or as an artist, there are no standardized tests to beat.
thank you.
it's awesome for someone like PG to bring the subject upfront. but what you are saying resonates with me.
genuine interest is missing in our field, and to some extent many others. "beating tests", as PG puts it, is at all time high. i am at a point where i don't know who or what is right.
the numbers are telling an important story! those who learned to hack the tests or the system are popular online or offline. and they are "succeeding" in life. they seem to have outnumbered those who put genuine interest first.
for instance, as many of my peers, i wanna learn machine learning and AI. but it is hard for me to find materials that resonate with me. materials that teach from first principles like those that got me hooked back in the days. they are lacking because we are so good at shortcuts and hacking. maybe we don't know how to do it anymore?
but who am i to blame anyone? there are so many, many, ways to hack and beat the tests and the system. and seemingly you can get ahead of many others in life by doing so. #AceYourFirstInterviewAfterBootcamp #TensorflowPyTorch #BeSureToLikeBelow #ThanksToMyPatreons #SubForChatAndEmojis #InstagramFacebookDown
i am very thankful for PG for starting the discussion. and for this comment.
However for this essay in particular, had I been among the set of draft readers, I would have suggested toning down the “I hadn't realized X until now, so now I think other people haven't realized X either” message.
Back in college I remember noticing choices of “better learning vs better grade” and deliberately choosing the former with a proudly “screw the authorities” attitude. I'm sure plenty of other Tara Ploughmans could say the same.
Yet, most interesting questions in life don’t yet have defined answers. Thus you need to have and test a hypothesis, which will very often be “incorrect” the first time around. But that doesn’t actually matter - the mere act of thinking about and defining what an answer could be sets you up to iterate and test. Eventually, you may find product market fit (a - not the - right answer).
You have to be willing to be wrong at first to learn...and win.
In real life you don’t know the question, there isn’t a single canonical answer, and there is no score-keeper to tell you if you are passing.
> And at elite universities, that means nearly everyone, since someone who didn't care about getting good grades probably wouldn't be there in the first place. The result is that students compete to maximize the difference between learning and getting good grades.
> When I started advising startup founders at Y Combinator, especially young ones, I was puzzled by the way they always seemed to make things overcomplicated.
I suspect part of the problem (of early founders being focused on hacking tests) was related to how heavily YC funded people from elite universities—which select people who do exactly that.
I love reading his tweets! They're all so down to earth and illuminating. That said, I'm not a fan of his blog writing as much. I find it full of overly-sparse nuggets of wisdom. It needs to be more concise, to the point, with fewer clever metaphors. This article could've been compressed 70% and still have been human readable.
"Omit needless words. Vigorous writing is concise. A sentence should contain no unnecessary words, a paragraph no unnecessary sentences, for the same reason that a drawing should have no unnecessary lines and a machine no unnecessary parts. This requires not that the writer make all sentences short, or avoid all detail and treat subjects only in outline, but that every word tell."
- - -
https://www.harpercollins.com/9780060932237/essays-of-e-b-wh...
I can take breaks from truly hard problems until I have enough clarity of mind to approach the problem with a new perspective. I can't do that with a deadline.
I don't care if the answers "are out there" meaning the professor has returned graded tests and posted the answers; I do the work because I want to learn. That paradox of students wanting to review graded homeworks and students who want to learn by doing the homework is resolved practically by those that are more vocal about their wants.
I'm happy to take my time, doing my best (with the understanding that my best takes time) and observing my grade afterwards. Sometimes my learning takes me down paths that harder and more rewarding with the consequence that I take longer than others: I am a slow learner.
I'd posit the YC application in and of itself shares some of the facets that the article is critiquing in a "test". The fact there are paid services popping up to review YC applications reminds me of SAT Prep services.
The key here is the “in theory” part. It is very hard to design exams that actually test your understanding of some topic as opposed to your memorization skills.
As somebody who recently got his bachelor’s CS degree (currently working on my master’s degree), and who also puts more emphasis on actually learning rather that getting good grades, I can tell you that understanding a subject might be enough to pass, but is seldom enough to get good grades.
All those domains are testable. But what is the practical work product of a deep knowledge of medieval history? It's extremely difficult to test the past, so it's certainly not predictions about what already happened. Nobody needs knowledge of medieval history for any practical purpose in the modern day. There is no possible way to test for this knowledge in a practical scenario because there is no practical outlet for the knowledge. Literary criticism is the same, along with much that is called "liberal arts" today. So for subjects like this, exams and essays are the only possible work product, and it's true that at that point you're going to have to use contrived tests, since real tests don't exist.
This strikes me as a really sad way to see the world.
Would you then agree that say, archaeology or palaeontology are things that nobody needs knowledge of for any practical purpose in the modern day? If you disagree, why?
But, I also think that probably more people are studying them than need to be. If that is a student's interest, then well enough. Otherwise, I'm not sure that, e.g., a middle school medieval history class is a good use of a student's time.
In my opinion, the issue in liberal arts education is with grade inflation, along with the scalability of assessing learning discussed in the essay/these comments.
I suspect that really rigorous liberal arts education is much harder to do at scale than really rigorous science/engineering education because to be blunt, there is less room for argument in scientific concepts.
For example, properly grading student essays and research is more difficult at scale in the liberal arts and the level of the whole thing gets dumbed down as a result.
The bridge test can be passed by memorizing one bridge.
The history course from American university required students to write few paragraphs as answers to around 5 questions that covered topics from lectures. Which is pretty close to what actual historians do.
Practical usage of medieval history: know when people are bullshiting for ideological reasons when they blable about how people in history behaved, what values they had and origins of certain traditions. Pretty much all "in the past people would" is a fantasy.
It's no coincidence that some of the largest hackathons are run and attended by students from the top universities pg mentions in the essay.
Which is why YC's hackathon was so satisfying. It was great seeing people pitch the normal hackathon-y ideas, then getting simple but direct questions such as "Why would someone use this app?" or "Why wouldn't I use <alternative> instead?".
The question I have is how do you hack the system such that you help those who are struggling to survive?
Luckily startups themselves are indeed not a test. So you can avoid any test by making enough revenue to not need external funding, but the second best strategy is take the VC's test without trying to hack it as your own test of the investor. Assuming you're a good investment, you'll beat those trying to hack the test unless the investor's test is susceptible to hacking. If you're not a good investment (which is by definition not obvious in the early days), maybe you should just try to learn to hack the test? Ironically, one of YC's biggest value adds is that it helps you hack those tests by lending you their name.
What has been driving me to startups is the fact that creating a profitable business has no place for hacks.
Fundraising is a game that could be easily hacked though. I thought it was only possible to hack the fundraising game for a couple of rounds but Adam Neumann et al. changed my mind that hacks could carry a business even to post-IPO.
Not sure that's true. Marketing is a hack, for one. If you have the best product but nobody knows about it, it's no use.
More generally, there's a reason that the tech career ladder is hackable: performance reviews are both important and also too expensive to do right. If you want a real evaluation of what I've done over the past year, what you really need is someone following me 40 hours a week, noting how I contributed (positively or negatively) at small meetings, keeping track of whether my projects are late because I helped someone with something truly important or I spent my time on HN, etc. But you can't assign one reviewer per employee, so you make an approximate process where employees self-report what they did and managers report the fraction they've seen. You also can't get rid of the process, because a simple profit motive demands you incentivize employees for actually delivering business value. So you have a process that's vulnerable to exploits like flashy launches that will wither in a year.
If you as a business owner aren't evaluating your employees, you won't be profitable. If you're watching each moment of your employees, you're wasting time. If you spend your time developing a fairer review process, you're not working on your actual business. If you don't hire employees and just put yourself out there in the market, your customers certainly aren't evaluating you fairly. (And of course your employees and potential hires are evaluating you on partial data, and it's in your interest to hire and retain good employees.) So whatever you do, it's hackable, and if you don't play the game you'll forfeit it.
I feel that this hack/not hack distinction comes from confusion of goals. If your end goal really is creating a profitable business then hack/not hack is not a useful distinction. Everything you do either gets you closer to your goal (then it is good, hack or not) or not (then it is bad, even if it "feels right"). But suppose your real goal is "doing most good for society" or whatever and you think that you can achieve this goal by building a profitable business. Then if you e.g. create some bullshit product and market the hell of it (thus achieving your proxy goal), it will feel like a hack that doesn't get you closer to your real goal.
TLDR: hack/not hack distinction arises when there are two goals: a proxy and the real one. A hack is something that gets you closer to your proxy goal but not the real one.
It is equally naive to think some idea of "quality of mind" or what someone has already learned in their first ~17 years is what they should go by. This is just shifting from one bias to another.
Evaluating a human being, reducing them to a number or a boolean, is hard, and people tasked with doing this, no matter how confident or how much thought and intelligence they put into it, will get it wrong, often with bad consequences. So perhaps we should take such systems lightly, stay humble, and not get overconfident in our judgements.
Possibly most of one's life is spent (over?) fitting to some "test" in a system, or "expectation" someone else has, instead of doing what they truly want: trying to pass a test at school, trying to land that job (you think) you want, trying to be socially acceptable in the schoolyard, at a party, or in the office, trying to please a parent, a friend, a partner, trying to conform to some unconsciously shared vision of "success", trying to conform to your own vision of "success", etc.
The list is long.
Fairer, unhackable systems would definitely help in reducing the amount of such "tests" we artificially created as a society. We should strive for this.
However is it possible to solve human nature? To solve "affinity bias", "confirmation bias", "psychological needs", etc.? Is it possible to create a society in which everyone is truly free? Or at least as free as possible from all this? Is it possible to solve freedom & wisdom? :-)
When a peacock is being judged for his feathers or song or mating dance, these are all being used as proxies for the actually important thing, the quality of his genes. And it's not just a male thing; women do all kinds of work to appeal to arbitrary male preferences.
On the other hand, survival of the fittest is an honest judge, so maybe these peacock tests work better than we think?
When I was in school, I did all kinds of hacks. I didn't graduate high school in the US. I took the GED to go to college. I did challenge tests to skip classes and save money. I got to know people who collected old tests. They trade old tests with me for homework help. Everyone graduated, got jobs, and continued their life hacks.
The focus on learning and doing is important. As long as we're delivering, we can hack some stuff. The philosophy debate between Kant and Locke will never end. There's a balance for each person. We need to discover it ourselves.
Instead of "hacking is bad", the essay was saying "incentivizing young people with a single easily-hacked methodology is bad".
It is the professor's job to decide which parts of medieval history are the most important to learn. Therefore, those are what I need to study.
Maybe not all professors do this effectively—but as long as we're talking about ideals, this is how it should work.
My college also did not have any old tests floating around, at least as far as I was aware.
This stuff helps you learn important stuff and makes it all less dry, but it absolutely does not make sense to penalize students for not learning it actively.
It does not mean that you did not learned or that grade does not measure learning. It just means the topic is not whole medieval history (which would be ridiculously broad) but only selected parts of it.
It is kind of like complaining that algorithms 101 are allowing students to pick any algorithms to learn and require everyone to learn the same set of algorithms.
Autodidacts often end up reading a bunch of stuff from here and there with sub-par results. One of the main benefits of a (university) course is focus and guidance. That the prof or teacher selects specific topics they deem important for that stage of education. As a student it can be difficult to judge that yourself.
The thing that had always left me a bit uneasy after my rocky road through education was just how damned certain everyone is about everything. Our education system, at least up through undergraduate unless you dip your toes into undergraduate research (and even sometimes still then!), teaches you things we “know” about the world we live in. Humans have discovered an _incredible_ amount about the world around us, but we have a _tonne_ left to figure out. We don’t lean into the unknown with our education system until late stage. The masses miss out on the beauty of the unknown.
PG starting to lose touch with regular joe.
Half way through the essay I was thinking, surely this can't be the first time PG has realized the idea of "playing the game", which is what I've known it as all my life. But then at the end of the essay he uses that exact same phrase to hit home the point that "playing the game" is antiquated now if one desires to become rich.
While I'd agree there are more opportunities for individuals or small groups to get rich now than in 1960, for the vast majority of us, the best bet to become rich is to "play the game".
I'm not being normally someone to comment on such things ( why bother with the effort?). I am making concrete efforts to get noticed by identifying groups/people who might align with my own thoughts/beliefs for my own vested interests.
Meanwhile, the other 7B population with substantially less power, we have to deal with each other. Politics are, and always will be, a thing. As long as people exist.
I've worked in tech for quite a while now. I used to buy into the meritocracy cant but as the years wore on, I realized how much bullshit it was.
I think the biggest moment came when I started to think about how to explain to people in my new home, South Africa, that working in tech would free you economically. There's some truth to that. But that's not the whole truth.
The truth is that all those years ago when I got my first coding job in financial services, I ticked 3 important boxes for the interview: white, male, and university-educated.
For me when younger this was about Ivan Illich (deschooling society), Feynman (the Brazilian light polarization story) etc. Now I'm older it's reading the anger in 'Excellent Sheep', watching the pain in low-trust workplaces, and telling my kids I value real learning over test-passing (they listen, but huge pressure to conform).
It seems true that a new field is more intellectually stimulating and liberating than a codified one (I've talked to older female analyst programmers from the 70s/80s who had a blast with computers and left when it got rigid and boring, I worked in Gurgaon early 00s when those interested in IT were self-educating geeks doing it for love, who later got moved aside + down when the money arrived). Maybe this is why IT people chase the new shiny, as a defensive strategy against codification, a red queen game to keep the game interesting ( though inefficient).
Anyway, glad Paul is publishing these viewpoints, even if the actions to be taken are not clear. Also, society is over-geeked currently. It's not obvious that hackers ( or intellectuals or logistics experts or technocrats) should have too much power instead of History students, or politicians or sports-stars or whoever. Life is both a game and not a game, short term gaming has long-term consequences. History and Biology and Literature teach us this clearly.
As a physics student, I can say we definitely pulled 2am-ers 2 weeks into the semester. Our Condensed Matter professor gave us some of the most random questions that no one could anticipate. To be honest, I was one of those "diligent" students who got good grades. I'll tell you right now his quizzes were not hackable. The following held true:
|quiz subject matter| >> |lecture matter|
On the other hand, I've also had professors who give exams pulled from the internet and allow use of the internet while taking those exams. I hacked that immediately during the exam, then almost got in trouble for it.
> ...if the professor tells you that there were three underlying causes of the Schism of 13... you'd better know them.
A professor like my Condensed Matter Physics professor would go on and on about 3 fundamental causes and then quiz students about the 5 fundamental causes, giving a zero to anyone who couldn't think of all the base 3, then grading the remaining students who thought creatively of 2 additional causes against each other on a bell curve. :)
To me, this sounds like a plausible argument against such tests.
The issue of creatively grading responses to an ill-defined question often pops up in discussion here about interview practices. In those discussions, typically someone will say “I don’t do a generic whiteboard interview. Instead I do [idiosyncratic thing x]. It really gives me amazing insight into the candidate.”
Then someone else says “Yeah right, a weird test with unclear metrics just gives you a big empty space to fill in with all your biases and pick someone who answers the way you would”.
Of course, both sides are exaggerated here. But it’s not clear to me that “creative” tests are necessarily any better.
For example, you could work on analog computers (look it up). The early AI systems were all analog, using motors on "volume knobs" to adjust the weights.
Analog is MUCH faster than digital. You could probably train an analog neural net "at movie speeds" because it all happens in parallel. Analog weights are trivial (capacitors), backprop is just differentiation which analog does trivially. Function inversion just involves putting the function in the feedback loop of an op-amp. A board with 100x100 mini-caps would be dirt cheap and would have 10,000 "weights".
Besides being "up to speed on AI NNs" you would also be breaking new ground (well, really old ground but nobody remembers anything before last year).
Imagine talking the department into an "analog AI lab" in cooperation with the EE dept and you get to be "the person". Heck, now you've nearly "joined the department" rather than being a student.
Think of college as a big company that lets you do ANYTHING. Profs have to prove themselves to get tensure. Students have to "get good grades" and "learn to hack the FAANG interviews so they can be a 'good cultural fit (aka grade hackers)". But YOU are at college as a "free agent" and can use all the resources to excel at what you love. Imagine having 4 years at a company with the resources of a college that lets you do ANYTHING.
You will never have it this good again. Forget the grade.
Every assignment and test is just marked pass or fail. Realistically tempered by what percentage you're allowed to fail for the class, e.g. 40% for university calculus - no different than grading on a curve today.
Fail any N assignments/tests and fail the course. Maybe some are weighted to count extra, e.g. Final project, midterm, final exam.
This is really not much different than the current system, but if you stop caring about the difference between C and A, students can either slack off or focus more on learning (if the subject is interesting).
Nobody cares about your grades after your first job anyway. I had nearly a perfect GPA in college, and it was a terrible waste of time in hindsight. If I'd spent more time socializing maybe I would have made some lifelong contacts that would have benefited my career or become co-founders with me.
I've always thought to tell my hypothetical kids to not sweat the grades and remember that the benefit of university is really the friends (and sometimes life partner) who you meet there. It's total shit when it comes to learning stuff. A self motivated person can learn more in a year than a 4 year degree. Somewhat depends on the subject though - I wouldn't want a self taught doctor who didn't do the practical parts of the education.
Testing and grades in general are a society level filtering mechanism and is a measuring proxy for how the work force filters for candidates at the entry level. As most know, once you've got about 10 years, most of that is largely ignored at the bottom of your resume.
Entrepreneurs, in the eyes of this system, are deviants of sorts. Hacks who couldn't/didn't want to make it in the system, but turns out that system is not the only way. A non-western example of this are a lot of the families that came up in money in China the past few decades. They didn't make it in the rigor of Chinese school system, but when they said fuck-it, I'll survive by starting something, they were wildly successful.
Controversially there's evidence pointing to education doesn't help people at scale[1]. So that whole structure, debatably, has diminishing returns.
So, I agree with Paul Graham 100%, schooling isn't the only way, and I mean that by - if my children decide they do not want to attend school anymore because they have a plan that, while unconventional, can do great things; Support all the way.
Unfortunately education cannot test that, because tests are by definition testing only a fragment of a system. Knowing and understanding one thing end-to-end is meaningful knowledge, and qualitatively better than knowing fragments of 100 things. Perhaps you can assess the former by grading a diploma thesis on a very specific subject. Tests may have an important role in motivating students to delve more deeply in subjects but they don't go beyond that.
There's similarly more value to knowing some technology "end-to-end" rather than specializing in only one level and never understanding what's below.
It's also interesting to read this:
> There are now ways to get rich by doing good work, and that's part of the reason people are so much more excited about getting rich than they used to be.
Which may reflect how things were 15 years ago. There were 2 threads in "Ask HN" last week which basically concluded that these days you're much better off (money wise) working for an uncreative role in FANG.
I feel like the university doesn't allow for time to stroll from the beaten path and explore topics that you're interested in. Most people I talked to have a very stressful studying experience. You know people care about your grade in the end, and you pay a lot of money for your degree. So it makes sense to also spent the majority of your time to learn the course material in and out.
Unfortunately, during job interviews many companies ask for GPA on graduation. I always had to explain “I learned a lot! I just pushed myself to the limit and when you do that you get B’s”
It’s always an interesting discussion and occasionally will lose my potential interviews. The truth is that school is really about certifying what you know, not learning. Personally, I think that should change, because building the skill of curiosity and The ability to research and understand a topic is far more valuable. But to learn to be curious you also have to be able to explore - can’t do that craming for test.
I've read most of PG's letters but I think this is now my favorite because while I've noticed each of these issues separately I've never connected them together through this common flaw in our education system. I find it really profound.
IRL most interesting and important questions are not anywhere close to that.
The real question Graham poses here is: what is the point of schooling? At the moment, it teaches and conditions people to "hack tests" as Graham puts it. So people continue to do that throughout their lives.
It also conditions people to think there is a "correct" answer to problems. Which in the open-ended nature of the real questions we have to grapple with in life is a complete lie. A teacher will rarely say: "It depends."
What passes for "education" today is really just "training" and very far removed from real learning. People only learn about things that interest them. And you learn by asking questions, which most of the time will lead to further questions to which you will only ever find tentative, temporary answers. Not that different from how scientists treat their "solutions" to big questions as tentative.
What can we do to change that? Change schools. It's the one place where we have kids kept for 15 years by mandate! It's the biggest lever we have to change perceptions and attitudes. Remove tests and grading. Learning will free up when that happens. It's remarkable what happens when at the beginning of the year the teacher gives everyone an A. Let go of the idea that permeates American society that everything needs to be measured. Some things do, but things like intelligence can't and shouldn't.
I know the immediate response is that this is unpractical. And to that I ask why? Because we need a way to control? Control what? What district is "performing" better? When was a score on a math test a true measure of human potentiality?
We need to go back to core questions as to what is the point of schools and what is worth teaching and learning in today's world. The world has changed immensely, but our "curriculums" are the same. The only difference is we use tablets and Microsoft Word to hand in assignments.
I agree with Graham's conclusion. When we liberate ourselves from these useless idealogical metaphors we will free people and society up to stop hacking tests and start living.
> I suspect many people implicitly assume that doing working in a field with bad tests is the price of making lots of money.
> Now you can make lots of money by making cool things.
An analogous set of incentive structures, which compels people in elite universities to play the tests applies to intra & inter-company interactions as well: companies who play for the tests will enjoy structural advantages over companies which do not; and this lever allows winner-takes-most mechanisms via buyouts, consolidations, private equity.
While founders, individually, may make out of this wealthy, companies, and society as a whole suffers, and playing for the tests, as a behavior, saturates.
Well I never had this problem in school.
I remember a high school calculus exam where I got the answer to the derivatives of the fundamental functions wrong (simple memorization - boring!) but I got the delta epsilon part of the test perfect.
I got a C- on the exam, but the teacher wrote a comment that always stuck with me: "You understood the most complicated part , as usual!"
Same thing in University, I never did homework in groups. I wasn't looking for the answer to the questions on my problem sets, I wanted a unique answer, or a more elegant answer.
I would outdo the professors sometimes. One time I saved a ton of busywork on a problem by setting up a homomorphism with another vector space, proof done in 5 steps!
However, for the student, it'd be better if they'd focus on the intersection of what they're interested in and what is societally relevant and not to be graded at all.
I know a friend of mine who went to a school for 1 year where he didn't get graded, all he learned was soft business skills. I can clearly see that he grew a lot from it. But I can also see that it is tough for anyone else to see that he learned anything at all. Partially this is because he isn't graded and the behaviors he picked up are now natural to him and it just looks like it was always there.
Our year's 2-hour exam was the following. The professor gave us a small cup of water, a piece of paper, and a piece of toilet paper. There was only 2 questions - place one end of the paper in the water. Describe what happens. Explain - place one end of the toilet paper in the water. Describe what happens. Explain
The year before us was with a small piece of glass and a small piece of potatoe - breathe on the glass. Describe. Explain - rub the potatoe on the glass. Breathe on it. Describe. Explain
You REALLY needed to understand the class material to succeed in these seemingly simple tests
I find it especially relevant as my first job was starting my own SAT tutoring company. Anecdotally I found it interesting that immigrant parents & most of the wealthiest US-born parents shared a recognition that this was a game. They were direct and explicit to their kids that this was a system that they must participate in, so they might as well win at it.
I think that is the key: make sure your kids can recognize when something is a bs system they need to hack (like selling enterprise software) and when something is actually about craft and the best will win (like building great products that end-users want to use).
One test I had was a counterexample to the hackable tests that Paul talked about. And it was the only one of that kind that was of that nature. It was in the EE Static Fields course. The final exam was one question, and you could hear the groans as the test was passed out:
Derive Maxwell's Equations
This test was a great indicator of how well you got the basic facts of the course.
It wasn't hard, but it was the fifth most failed course in the university because the tests were extremely difficult to "hack" and the projects were big enough that a student could not rush them at the last minute. There was no homework or attendance to pad grades.
It was one of the few times in my college career that I felt like a professor actually cared whether or not students actually learned about the subject.
When you learn through projects your ability to apply concepts is tested by the results you generate, connecting to our intrinsic motivations.
Standardized testing is, functionally a disease. Check out “DIY University” to see the effect it has on society long-term.
If you think projects can’t be the path to teaching highly abstract disciplines, such as physics, check out “Surely You’re Joking Mr. Feynman.”
“Most Likely to Succeed” is probably the best book written on this topic.
I spent 10 years as an edtech entrepreneur. It is hard to overstate the degree to which we have built a Testing system at the expense of an educational one.
Less extreme variant of it would be - OK, you went to college - but show me what you did on your own (personal projects, small attempts at startups etc.) during that time. That's what the game dev industry is doing. If the answer to this question is "nothing", then you're probably facing a person who's happy to take directions and land himself a cushy job in some large organization, where he will use his intelligence to navigate towards maximum pay and minimum workload.
Maybe I'm overly cynical, but I see another possibility: They want to create the appearance of running a hot startup without actually having to do the work. Why make something people want when you can just dazzle some investors and get them to dump a million dollars in your lap, no strings attached?
I wonder if that also might explain why such a tiny fraction of startups generate essentially all the wealth -- most of them aren't really trying.
I'd rather not waste 30% of my life on that nonsense. Better to use uni for what it is meant for: Learn whatever seems interesting and don't mind the grades.
So in some perverse sense school does prepare most of us well for working life, as most of the time doing well in BigCo isn't to actually perform well or to actually make a difference, it's to leave the impression of it at the right time with the people that matter for your next promotion.
This is what happens when the teacher's hidden goal is to make marking/grading easier.
At secondary school level, it's no longer an issue. Parents just change schools. A few months ago, an uber unserious secondary girl altered her end of term results. The father brought the slip to school to complain about the unprofessional look of the slip due to multiple alterations.
Found out what his daughter did and change schools.
Training/Certification Firewall
The Linux Foundation has two separate training divisions:
Course Delivery and Certification. These two divisions are
separated by a firewall.
The curriculum development and maintenance division of the
Linux Foundation Training department has no direct role in
developing, administering, or grading certification exams.
Enforcing this self-imposed firewall ensures that
independent organizations and companies can develop third
party training material, geared towards helping test
takers pass their certification exams.
Furthermore, it ensures that there are no secret "tips"
(or secrets in general) that one needs to be familiar with
in order to succeed.
It also permits the Linux Foundation to develop a very
robust set of courses that do far more than teach the
test, but rather equip attendees with a broad knowledge
of the many areas they may be required to master to have a
successful career in open source system administration.
If there's one thing I learned in politics, the biggest hack is in having a great narrative. Ultimately, brands, companies, and politicians are characters in our consciousness that only achieve meaning through the stories they tell.
Essentially, the story must be believable and ring true to the audience being targeted according to their past experience. For example, evangelicals love a good redemption and hard work story. Each group has their own highest morals. I'd recommend "Don't Think of An Elephant" by George Lakoff, which explains how politicians hack the human mind by crafting an entire language that plays into their hopes and fears.
I'd love to have every test in life be genuine and unhackable. But when tests are clearly so hackable, it doesn't help to not call out how the hack works. Call it responsible cultural exploit disclosure.
Bingo! The real problem is the using incorrect metrics. If you judge a fish by its ability to climb a tree, it will spend its entire life thinking it is stupid!
Disclaimer: I work there now and my opinions don't necessarily represent my employer's etc.
You can claim non-artificial tests are "better," but humans (VC's) still hold the purse-strings so to speak. If you can convince them you're winning the non-artificial tests say of the market, then you can get money and do coke and party for a while, whilst sure your company will crash and burn but you escape unscathed with millions of sweet free money. My point is the problem is deeper than just "artificial" vs. "non-artificial", although it is an important aspect.
Companies and other big institutions need very strong assurance that supplicants will indeed "work out" by comporting themselves to the culture and making valuable contributions to their goals. And the reason such absurdly strong signals of assurance are needed is the ridiculous level of friction inherent in admission to and ejection from these institutions and their hierarchy of stations.
So, to get into Big Company X, you need a prestigious degree, and to advance once you're there, you must identify and play their internal signalling games. Such is the attachment to signalling, that you'd often do much better to lead the holiday decoration committee than to create actual value by solving some important problem.
The relative lack of friction in open source is why, given the ability and inclination, even a dog can contribute at a high level.
IMHO, a radical innovation in education is one that can address at least two of the following flaws and failures in the current education system, at any growable scale:
–Duration and cost –Content –Delivery
If I were to satisfy a VC, I could also add a 4th bullet point called "Supply/Demand", but since I'm not writing for a VC, then I'll write what actually matters. In the case of education, that is, it's okay to ignore the existing supply and demand, and focus on the demand that must exist that doesn't yet.
Wrote a memo on these and elaborated more https://www.linkedin.com/pulse/q4-2019-what-i-see-current-st...
P.S: I think I got lucky that my mom was a teacher, so I grew up having a definition of right and wrong different from what was expected of me, and that stayed with me throughout my education. I didn't do what was asked, but what I believe was right for my growth.
In theory, tests are merely what their name implies:
tests of what you've learned in the class. In theory
you shouldn't have to prepare for a test in a class any
more than you have to prepare for a blood test. In
theory you learn from taking the class, from going to
the lectures and doing the reading and/or assignments,
and the test that comes afterward merely measures how
well you learned.
I was constantly amazed at how many of my peers, who were paying good money to learn stuff at school, were utterly shocked by that philosophy. I really wish more students understood it.
Growth is like cancer. Eventually a company will get big enough to bully others for the sake of growth.
Last night I had dinner with a friend who is building a (yet another) language learning app. Simple fact about language learning, or learning in general: You must practice almost every single day. Even if it is only 5-10 minutes.
PG himself said, if you don't die (aka give up), you will eventually succeed: http://www.paulgraham.com/die.html
To me, the best, most sensical way for learning applications to measure success is to delete user accounts with > 3 days of inactivity. This way you can measure the absolute effectiveness of your program. Students who stick through with your application should 100% become fluent. Target the users who actually want to learn, and measure your success by how many users you made successful.
"NO no no", he says. You'll have no users if you do that.
> At this point I've told the founders something you'd think would be completely obvious: that they should make a good company by making a good product. And yet their reaction would be something like the reaction many physicists must have had when they first heard about the theory of relativity: a mixture of astonishment at its apparent genius, combined with a suspicion that anything so weird couldn't possibly be right.
The essay context is students and startup founders, but it turns out most multi-billion dollar enterprises have forgotten this as well.
This notion does not come embedded in the heads of most “senior management” from either world.
> No, I would explain, that is not how to get lots of users. The way you get lots of users is to make the product really great. Then people will not only use it but recommend it to their friends, so your growth will be exponential once you get it started.
Ironically enough, if there is one thing I've heard from HN, it is to not believe in the "if you build it, they will come" myth. That no matter how good a product you build as an engineer, it is all pointless if you aren't willing to hit the ground and start aggressively marketing and selling it. That a mediocre product with great marketing/sales, will win over a better product with poor marketing/sales.
I've heard this directly from YC partners themselves, when they keep telling early-stage founders to market themselves via "do things that don't scale". At Reddit, one of YC's early successes, the founders were literally spending their time spamming their own site with fake posts and comments using sock-puppets, in order to create the illusion of activity. As someone who loves building products, I hate the idea of spending time and energy on tactics like that. I'd love to spend all that time and energy on making my product better. But I've learnt grudgingly from YC that product development is pointless unless I'm using guerilla tactics to hawk my product in front of users.
Maybe I'm in the minority, but I've actually had the opposite experience as what PG describes in his essay. Back in school and University, I did study very hard to ace tests. But the way I studied was to genuinely learn and understand the material as well as possible. Not to "hack" it in some way. Whereas once I graduated, and especially once I tried out entrepreneurship, I realized that just building a great product was insufficient. I now had to "hustle" and use "street smarts" and "growth hacking" in order to get people to notice what I'm doing. The lesson I had to unlearn from school was that the quality of your work will speak for itself, and will win the day. I never needed marketing and sales in school, but it seems indispensable in the real world.
You have pretty much summarized another one of his talks : http://paulgraham.com/ds.html
(Or even two : http://paulgraham.com/schlep.html )
What is worse (and although it is related to grades it is not the same), you only do what you are told to do. Your goals are set for you - read this chapter, solve this problem, answer this question. Thus your inherent creativity is slowly but surely gets sapped away until you become a "professional" - someone who solves problems of others without questioning them much.
Not sure what to do with this though - every way to overcome this problem seems inherently not scalable. Also not sure if this is really a bad thing - a society where everybody suddenly becomes more creative can suddenly collapse because of all the excess creativity!
Software is outrageously accessible. The threshold cost for make-lots-of-money-by-making-cool-things has fallen to a used laptop and a wifi password. The prerequisite talent is mostly the ability to focus relentlessly while illegitimi non carborundum.
But since those are rare talents the laptop can only elevate rare people. It's a path to relative wealth for only a pareto distribution of people, like pretty much anything else. But it's a path to absolute wealth for most of the rest of us. Cool things that make money tend to make our lives better.
Make-lots-of-money-by-making-cool-things is a recipe for both more absolute and unequal wealth.
The best that can be done is to try to keep the correlation between the "test" and what it's trying to measure add high as possible and to teach everyone how to hack it, that way everyone is on a level play field. If everyone hacks it then the difference in scores will be mostly about what is being measured and not about who hacked it and who didn't.
That's why big companies actually advice interview candidates to practice on leetcode and give them example interview questions.
But I agree with the main point that being good only at "hacking tests" in life instead of being good at your profession is very bad.
Maybe there aren’t a lot of fine artists starting YC companies, but how about designers who went to art school? Do they treat these problems differently? Do they have less to unlearn, or different bad habits?
There's no point looking at one of their idiosyncrasies in isolation.
But then Paul Graham plays himself because all the startups are looking to get or softbanked. Absorbed into the monopolies or put in permanent luxery lift support. What a joke. Playing the VCs is more important than playing the increasingly poorer consumer hoards of the decreasing yield suburban consumer farms. We're years deep into the startup fad, and blinded test takers appealing to their blinded test taker money holders is all that's left.
He doesn't admit any ways to actually hack market growth (as if there were none), which is weird. Seems like Use Dark Patterns, Pad Your Numbers, Fake It 'Til You Make It, Regulatory Capture, &c would all make the list.
I agree with the general idea, though: I think the least hackable tests are either Wimbledon/tennis or GSL/StarCraft tournaments; you simply cannot schmooze your way to even the lower brackets.
[0] like most things people put in 2 categories
It would have been an interesting contrast.
I wonder is that because almost all people who do YC are university educated.
Perhaps Paul graham doesn’t have much experience of entrepreneurs who are not university people.
Sure, ok. How do you make a good product? That definitely is not "completely obvious"
Making a good product (with whatever shortcuts/hacks come with it) is necessary for doing well. Getting exposure and launching on a Tuesday are neither necessary nor sufficient, but they will give a marginal edge.
Both the former and the latter are 'hacks', the only difference is degree of relevance. pg throws the baby out with the bathwater a bit by saying don't do both types of hacks.
But if you just given someone a “B” and send them up a year at school/university, then presumably they find it harder and harder to do subsequent courses. And over time they learn they’re a B student. Which is demoralising, and possibly also insulting as it ascribes failure of the learning process to the student and not the teacher.
Most of the time, our grades were based on the work we produced, and our participation in discussions. Even in Art History, when the job was to study and memorize slides and identify them, the test was not to just say what they were, but to write a paragraph about how they fit into the cultural context from which they came.
That said, the website is a pain to use on mobile safari and difficult to properly capture when sharing it to Notion or Bear. Notion doesn’t capture it at all, and Bear can’t capture the hyphens or line/paragraph breaks. It’s probably because of the read more button. It also has some jaggy scrolling issue where if you switch between apps it scrolls up and sometimes collapses the article or jumps to the bottom of the page.
> Anyone who cares about getting good grades has to play this game, or they'll be surpassed by those who do. And at elite universities, that means nearly everyone, since someone who didn't care about getting good grades probably wouldn't be there in the first place. The result is that students compete to maximize the difference between learning and getting good grades.
It's a race to the bottom.
Perhaps if we move away from wage work then school will change too.
In academia, having good grades is sometimes helpful to climb the ladder. Not all people can study medicine. Not all people can get a master degree.
Often these gigs are given to the best students only, but sometimes you can circumvent the grading issue by waiting longer, going to the military, be friends with a prof. or working for free.
I think you are falling for the fallacy that the essay is trying to expose?
Surely those that pass medicine are those that are good at medicine exams. Plenty of people would make great doctors that don’t due to the exam system, not due to a lack of capability.
Figuring out the tricks to pass tests is not hacking, it's going with the flow. Or at best, it's hacking in the "script kiddie" fashion.
https://www.amazon.com/Why-Greatness-Cannot-Planned-Objectiv...
But PG is wrong. Startups can, and do, game the system. By showing unsustainable, artificial growth with no real value behind it, they can fool investors long enough to make a lucrative exit by IPO or acquisition.
> I was genuinely interested in most of the classes I took, and I worked hard. And yet I worked by far the hardest when I was studying for a test.
This is a great sharing of internal mental processes. The same process evolves around investment reports. "Oh look! I got a good grade!"
Oh, no no no, it was too cold always
(Still the dead one lay moaning)
I was much too far out all my life
And not waving but drowning.
[1] https://seths.blog/2018/09/education-needs-to-be-inconvenien...
Various quotes: https://www.goodreads.com/author/quotes/41319.John_Taylor_Ga...
So TLDR is, I dont think the situation is so bad, because grades are not only a function of tests but also assignments and projects. I think the deeper issue is that people learn to be told how to learn (which assignments or projects to do), and not taught how to learn by themselves. There are exceptions (at Stanford a ton of CS classes have projects where the students map out the problem and solution themselves), but I think this is pretty important too.
---
Phaedrus' argument for the abolition of the degree and grading system produced a nonplussed or negative reaction in all but a few students at first, since it seemed, on first judgment, to destroy the whole University system. One student laid it wide open when she said with complete candor, "Of course you can't eliminate the degree and grading system. After all, that's what we're here for."
She spoke the complete truth. The idea that the majority of students attend a university for an education independent of the degree and grades is a little hypocrisy everyone is happier not to expose. Occasionally some students do arrive for an education but rote and the mechanical nature of the institution soon converts them to a less idealistic attitude.
The demonstrator was an argument that elimination of grades and degrees would destroy this hypocrisy. Rather than deal with generalities it dealt with the specific career of an imaginary student who more or less typified what was found in the classroom, a student completely conditioned to work for a grade rather than the knowledge the grade was supposed to represent.
Such a student, the demonstrator hypothesized, would go to his first class, get his assignment and probably do it out of habit. He might go to his second and third as well. But eventually the novelty of the course would wear off and, because his academic life was not his only life, the pressure of other obligations or desires would create circumstances in where he just would not be able to get an assignment in.
Since there was no degree or grading system he would incur no penalty for this. Subsequent lectures which presumed he'd completed the assignment might be a little more difficult to understand, however, and this difficulty, in turn, might weaken his interest to a point where the next assignment, which he would find quite hard, would also be dropped. Again no penalty.
In time his weaker and weaker understanding of what the lectures were about would make it more and more difficult for him to pay attention in class. Eventually he would see that he wasn't learning much; and facing the continual pressure of outside obligations, he would stop studying, feel guilty about this and stop attending class. Again, no penalty would be attached.
But what had happened? The student, with no hard feelings on anybody's part, would have flunked himself out. Good! This is what should have happened. He wasn't there for a real education in the first place and he had no real business there at all. A large amount of money and effort had been saved and there would be no stigma of failure and ruin to haunt him the rest of his life. No bridges had been burned.
The student's biggest problem was a slave mentality which had been built into him by years of carrot-and-whip grading, a mule mentality which said, "If you won't whip me, I won't work." He didn't get whipped. He didn't work. And the cart of civilization, which he supposedly was being trained to pull, was just going to have to creak along a little slower without him.
This is a tragedy, however, only if you presume that the cart of civilization, "the system," is pulled by mules. This is a common, vocational, "location" point of view, but it's not the Church attitude.
The Church attitude is that civilization, or "the system" or "society" or whatever you want to call it, is best served not by mules but by free men. The purpose of abolishing grades and degrees is not to punish mules or get rid of them but to provide an environment in which that mule can turn into a free man.
The hypothetical student, still a mule, would drift around for a while. He would get another kind of education quite as valuable as the one he'd abandoned, in what used to be called the "school of hard knocks." Instead of wasting money and time as a high-status mule, he would now have to get a job as a low-status mule, maybe as a mechanic. Actually his real status would go up. He would be making a contribution for a change. Maybe that's what he would do for the rest of his life. Maybe he'd found his level. But don't count on it.
In time - six months; five years, perhaps - a change could easily begin to take place. He would become less and less satisfied with a kind of dumb, day-to-day shop-work. His creative intelligence, stifled by too much theory and too many grades in college, would now become reawakened by the boredom of the shop. Thousands of hours of frustrating mechanical problems would have made him more interested in machine design. He would like to design machinery himself. He'd think he could do a better job. He would try modifying a few engines, meet with success, look for more success, but feel blocked because he didn't have the theoretical information. He would discover that when before he felt stupid because of his lack of interest in theoretical information, he'd now find a brand of theoretical information which he'd have a lot of respect for, namely, mechanical engineering.
So he would come back to our degreeless and gradeless school, but with a difference. He'd no longer be a grade-motivated person. He'd be a knowledge motivated person. He would need no external pushing to learn. His push would come from inside. He'd be a free man. He wouldn't need a lot of discipline to shape him up. In fact, if the instructors assigned him were slacking on the job he would be likely to shape them up by asking rude questions. He'd be there to learn something, would be paying to learn something and they'd better come up with it.
Motivation of this sort, once it catches hold, is a ferocious force, and in the gradeless, degreeless institution where our student would find himself, he wouldn't stop with rote engineering information. Physics and mathematics would come within his sphere of interest because he'd see he needed them. Metallurgy and electrical engineering would come up for attention. And, in the process of intellectual maturing that these abstract studies gave him, he would be likely to branch out into other theoretical areas that weren't directly related to machines but had become part of a larger goal. This larger goal wouldn't be the imitation of an education in Universities today, glossed over and concealed by grades and degrees that gave the appearance of something happening when, in fact, almost nothing is going on. It would be the real thing.
Open up HN again this afternoon for a quick work break...
And find my eyes IMMEDIATELY glancing at my karma score in the top right before anything else.
Perhaps PG should practice what he preaches and rip that out of the UI?
I would encourage it... damnit... I just looked again.
td:nth-child(3) > span.pagetop { color: transparent; }
Hides karma, leaves user and logout intact (as they're links)
It is still true. A FAANG job is the easiest path for a engineer to millionaire status in about 5 - 10 years.
The interview process is eminently hackable.
Or perhaps you want to do a startup, get funding. You build a this for that .com. Do some social media, find a few users who love it. Read pg's advice, pitch deck advice, go to some demo days.
Hack the test. Get funding. The incubators, the vc fiends are themselves, oligopolic. A handful of firms with common expectations. Mmr. Hockey stick user growth. It's the team! Here's your A good chap.
Those are the tests here. But there are other ways to get an A+. You don't have to start a business the way software people start a business. You can do it like the cabinet maker, or the electrician, or the store owner.
You can buy one increasingly easier today because the baby boomers who started than decades ago want to retire. Imagine how you can take that business into the future with the software you can use to automate it.
Business brokers abound. Go talk to one.
Stop thinking go big or go home.
Go talk to a business broker. Go ask about owner absent business options. Residual income and you didn't even have to be there. Use automation to make the employees lives better and more prodictive. And your customers' lives.
Package that software you write into a SaaS and sell it to you competitors. Profit more.
You are your own case study.
edit: and now I got to a point in the article where that's pretty much already said. Well, too late to delete this comment now.
TLDR opinion. The hackable c corp structure Y pushes has payed off well for Ycombinator yet it is the very thing that can distract from building a good product.
Then again I'm sure someone might make the claim that c corps are also a way to easily share equity with your employees.
Does pg argue that they simply should ignore this test grade?
Yes and the job interview.
(My favorite from this one: "Learning is the naive algorithm for getting good grades.")
In my view schools/universities are institutions for mass indoctrination. Noam Chomsky has said something along similar lines. I borrowed the words from him.
A bunch of my other observations:
https://realminority.wordpress.com/observations-of-the-world...
In D. Knuth's The TeXBook is:
"The traditional way is to put off all creative aspects until the last part of graduate school. For seventeen or more years, a student is taught examsmanship, then suddenly after passing enough exams in graduate school he's told to do something original."
In particular, there can be some comeuppance to too much course hacking and not enough learning: Without enough actual learning, may have too much trouble doing "something original".
On the course hacking Paul describes:
(1) Dad had his Master's in education and to me emphasized just learning and never grades. So, I emphasized learning. He was a bit extreme in ignoring my grades: A few times my neglect of grades cost me, never much but a little.
And for Paul's remarks on studying for tests, that's what I did: I never studied for tests. Essentially as Paul explained, all along in the course I'd been trying to learn the material so no way would just cramming and losing sleep help me.
Broadly Dad was correct: Most of what I got out of education that actually helped me was just the learning. And one of the best things I did learn was how to learn, i.e., pick good subjects, topics, and sources and learn independently from the sources.
E.g., the college I went to as a freshman was selected because it was nearly for free and I could walk to it. But the college was so much like just a grade 13 that they didn't let me start with calculus. But I'd been a math major at a relatively good college prep high school, e.g., that the year before me had sent three students to Princeton and my year sent one to MIT. And in my year, of #1, #2, #3 on the Math SAT, the MIT guy was #3 and I was #2. Similarly on the CEEB Math knowledge test. Sooooo, I SHOULD have been in calculus, and the course they had me in was 90% lower than what I'd already learned in high school and for the other 10% I just taught that to myself in a few days. But I wanted to be a college math major and not fall behind so got a good calculus book and dug in and did okay. The next year I went to a college with a quite good math department, continued with their sophomore calculus from the same text Harvard was using, pretty good text, made As, and did fine. So, I never took freshman calculus; my ability to teach myself saved my tail feathers.
On Paul's remarks on grades, there was a cute story: When the SAT scores came back, one of the teachers, I'd had in the sixth grade, was appointed the counselor for giving the scores to the students; so, she knew me well and knew my reputation among the teachers. She read my Verbal SAT score and had a nice smile, said it was "Very good". It wasn't so good, but she was happy because the score was MUCH better than my reputation! Then she read my Math SAT score and became afraid, upset, confused, speechless. When she recovered she said "There must be some mistake." Yup, there had been, hers and the other teachers for 12 painful years.
E.g., there was one really good day for learning: In plane geometry, my favorite subject, the teacher was competent but really nasty. No way did I want her to have any credit for my learning; so no way would I ever admit to doing any homework. Each day she assigned three not very difficult exercises. I outlined solutions in the margin of the book for the most difficult exercises, maybe some of hers, and then turned to the more difficult exercises in the back and did them ALL, never once missing one. One of them took me from Friday afternoon to Sunday evening. In class on Monday, she worked at the board an easy exercise with the same figure I'd worked on all weekend. So, I woke up from having my head down resting as usual and for the first and last time spoke in class: "There's an exercise in the back with the same figure." She took the bait! About 15 minutes later, with frustration, she was exhorting the class "Think, class, think. Think about the given ....". Heck, I didn't know that I was the only one doing ALL the homework and that even the teacher had not done all the homework! I didn't want to be accused of ruining the class so blurted out "Why don't we ..." and in a few more words, with great anger, she interrupted me. NO WAY did she want my solution! With anger she said "You knew how to do it all along!". Of COURSE I did: No way would I ask HER for help in math! This story might make a good scene in a movie!
There was one more with her: On the state test, the last problem was how to inscribe a square in a semi-circle. After the test I kept thinking about a good solution so asked to see her after school. She didn't like that. I mentioned the problem, and she asked "Why are you working on that?". Well, it was curious, and I'd already solved everything else in sight! So, I went off on the side and constructed a square, circumscribed the semi-circle, constructed a fourth proportional, so had the crucial length in the original figure and did that construction. She said it was not legal to construct the other figure. In college, a student in a geometry for teachers class showed me, given triangle ABC, construct D on AB and E on BC so that the lengths of AD = DE = EC. So, off on the side I started with the angle BAC, picked any length for the AD for that figure, finished constructing the figure on the side similar to the given figure, found the crucial fourth proportional, and completed the construction in the original figure. She said, "How'd you know to do that? That is smilimitude, an advanced technique we are studying." I said "I rediscovered it in high school, and the teacher said can't do that.".
(2) I started my career in applied math, computing, and physics for mostly problems in US national security near DC. There no one cared what course grades I'd made but cared a LOT what I could DO that was important and based on what I'd learned or could learn quickly. And I was learning like drinking from a fire hose. Net, what I knew and learned got me a good career, at one time an annual salary six times what a new high end Camaro cost.
(3) I did well in freshman physics, led the class. One of the guys I'd beaten got some respect for me so contacted me when FedEx was struggling with scheduling the fleet. I'd been teaching computer science at Georgetown University (had never taken a course in computing but had been learning from that fire hose) so joined FedEx, typed software furiously, got a schedule for the fleet, thrilled the Founder Smith, pleased the BoD, enabled some crucial funding, and saved the company, all based on what I'd learned, in physics class and in computing. Later did a little calculus for some revenue projections, pleased two crucial BoD Members, and saved the company again.
(4) In grad school, what I'd learned before entering helped a LOT: On the five Ph.D. qualifying exams, I did the best in the class on four of the exams, and for three of them what I did was all from what I'd learned in class or taught myself before entering. Later my interest in learning and neglect of formalities torqued off my department Chair. So, in a course in optimization, I saw a question with no answer in the course. I found no answer in the library. I got some rough ideas for how to get an answer so signed up for a reading course to address the problem, not necessarily solve it. In two weeks I had a nice solution with a nice new general theorem and also a nice solution to a related problem stated but not solved in the famous paper of Arrow, Hurwicz, and Uzawa in mathematical economics. My work looked publishable (it was; I did publish it later in JOTA). Word of my solution spread in the department. Then I had a halo: No one cared about formalities and let me just LEARN and, then, do my Ph.D. research, with no faculty direction. Again, what I had LEARNED saved my tail feathers. And for Knuth's remark about something "original", the learning let me do that. I did find that the thinking techniques I'd used to do the more difficult exercises in Rudin, Royden, Neveu, Loeve, Halmos, Fleming, etc. were sufficient to do publishable original research.
So, net, just learn and do that by doing the more difficult exercises! E.g., back to high school a challenging test question got me to reinvent similitude!
I still have no idea why this isn't used more generally in life. Want to see if someone can do this job...hm, rather than actually seeing them do the job...have you ever thought about how many windows there are in New York? All you are testing is whether someone has either been asked that kind of question before or, more likely, whether they went online and looked at what you would ask them.
It is truly stunning how stupid people are about this. And, tbh, I think it largely reflects what most students about the world at university where certain professors seem to revel in the utter banality of pointless tests.
A: Because you and them don’t have the same circumstances, and thus don’t have the same utility function. You have different values.
The whole (long) thing is a mess of obvious misses. Too bad.
So FPGA solutions that "live on the wire" and can be "plugged into the LAN port" are a really useful area you can hack in your dorm room.
I worked in the security area. One of the key problems is "exfiltrations" where someone tries to copy valuable files off a server. I created a startup to attack this "on the wire" with hardware so it can't be hacked. (It failed because nobody knew what an FPGA was, nor how TCP packets worked. Sigh).
The idea is to create crypto-hashes of the valuable files. Hand-install the hashes into an FPGA using a read-only micro card. The FPGA sits on the wire, hashing files being sent. If the hash matches, scream.
THe same idea of "unhackable" malware virus scans can be done on an FPGA. Only someone with physical access to the FPGA card can modify it. It operates "at wire speeds".
The hard part is finding people who can spell FPGA and know what malware means. Be that person.
You could do this in your dorm room.
The side effects are that you learn a LOT very quickly, it is grounded in real code, and you could actually convince YC that your "homework" is worth a passing grade.
Besides, FPGAs are just plain fun. Make your own hardware Neural Nets. Hack your bike with a camera and an NN to warn of approaching cars.