Hacker News Reader

Comment by wolframhempel | Hacker News Reader

This has been eluded to by a few other comments already, but I think it's worth stressing: Maybe the problem is that our definition of "cheating" doesn't make sense anymore.

Our methods of teaching come from a worldview in which the information stored in your head is your only resource. That wasn't true in the past where you could consult colleagues, look things up in books or call experts - and it's certainly not true now that everyone carries the entire knowledge of humankind around in their pocket.

So what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life? How about instead, we create (whatever the plural of curriculum is) that teach you how to use these resources efficiently and test you on how well you perform using all the tools that will be at your disposal in your actual job as well?

View on HN

Being able to look things up, even in seconds, is not equivalent to knowing them. When someone suggests "let's do X", you cannot reply with "but that would cause Y", if Y is buried in some pdf file on your computer, or one Bing search away - how do you know you should search for Y, when there are infinite other things you could search for.

That's one of the main points of The Shallows by Nicholas Carr and I think that I agree.

Long term subconscious memory is where a lot of the connections happen. If your long term memory only knows where to find things without knowing them then that ability is stunted.

The way I used to write/set-up tests when I was teaching is that you could use a limited set of online resources to answer, but if you have to search every question you won't have enough time to do it. So it also tested your ability to quickly locate the information you need should you not know it right away.

Back at university one of the first things we were told about being an a good engineer was: "You don't need to know everything, but you do need to know where to look it up".

It's a statement that I agree with. There is just too much knowledge to know it all. These days there is a very real skill in knowing how to search i.e. a second order version of the above statement "you need to know where to find out how to look this up".

There's a HUGE difference between looking something up to reactivate sleepy memories and learning something new for the first time.

There are some really technical things ive done in the pasy that I couldn't do without a reference right now, but once I've got the reference I don't need to be reminded about low level details or edge cases or gotchas - it's all still there in my head I just need to see the road map and it will all come back to me.

Yep, my dad used to sum this up as "All Engineering school does is teach you where to shit." I guess I may have gone with "where to eat" but he had a flair for the dramatic.

But still, there's "minimal required knowledge".

Like how to use google

Nope.

I wouldn't want to go to doc that has to Google basic stuff.

Conversely, I wouldn't want to go to a doc who doesn't know how to Google basic stuff.

Underestimating the amount of knowledge required or of importance to know (yes, even when google exists) is just another form of the infamous Dunning-Kruger.

Google’s search mechanisms and rankings have changed a lot over my lifetime. I don’t usually trust it to find the right answers to problems in my field unless it’s a very well supported technology. I don’t agree that “knowing how to Google” is a skill in its own right for this reason. Hell they even claimed not long ago they would fix quote searches but they seem worse than ever.

It depends. I haven't yet in my career come across something that I couldn't research online because I hadn't been taught it. Usually the first step is to discover which words to use, reading around the domain. Once I have the words I can usually find out everything I need to do my job.

Knowledge is so easily acquired these days that, while useful, it is not essential. I hire people with zero domain knowledge for that reason.

I see people struggling to find the correct initial keywords to search on all the time; and I have the same problem trying to google domains I don’t know. But this can usually be solved with sufficient effort, though I don’t know many people who are willing to put in that effort.

But more problematic is when people don’t know there’s a problem to google in the first place — it seems to do the job — which is usually resolved by both education and experience, but is ultimately a result if they don’t know what they don’t know.

The knowledge is generally out there, but you need to know you’re looking for it, and it’s only in the common/trivial problems that it’s so obvious that the compiler/tooling/customer just flat-out tells you you’ve missed something

Oh, rly?

Explain this to me by looking it up. https://imgur.com/Bj24AAR

It is a lot faster to have things stored in your head than to look them up. You can just put a time limit on the test and tell people that they can either spend a year before the test starts learning the material and getting it into their heads, of they can try to learn biochemistry from scratch using Google after the 40-minute timer for the test starts.

The thing is, if you search for something enough you will end up learning it. It's a natural way to learn things.

This is how people become software engineers these days. They _constantly_ look stuff up, watch videos, and learn by doing that. Yes, you can still go to college and take tests that require you to "know" things, but honestly the vast majority of what I "knew" in college turned out to be short-term knowledge stored for tests.

By searching and learning, you

A person can certainly become a software engineer without a college education, but I worry that constantly learning things just in time for using them will give them disconnected snippets of knowledge in a lot of areas rather than a good concept of how the underlying system functions. I guess it depends on if they are studying computer architecture, complexity theory, and whatnot in those videos or if they are just copying snippets of code from SO.

I learned by doing, initially, because I started learning programming with minimal resources before going to college and had to unlearn or restructure a lot of the knowledge I had gained up to that point once I did. I would probably have gotten there in the end, but it would have taken me a lot longer. The world was different then, though. Instead of easy access to info before going to college I was lucky to find a book somewhere that someone threw out that talked about things in a peripheral way that I needed to know, or ran into someone I could discuss things with that were as info-deprived as I was.

That's where my skepticism of this approach comes from, I guess. But I didn't live in a world where knowledge was at my fingertips from a very early age, so who knows?

When someone suggests "let's do X", you cannot reply with "but that would cause Y"

This is reasonable. E.g. sure a person can google how to build a house, but also sure that at planning phase they’ll mess and conflict with a half of good rules of thumb. Or forget them right after reading, because there is no habit nor knowledge in their head, and all that info is pretty transient. Pro is not the one who has read an article, but the one who can tell a couple of their own notes about it.

Otoh, gp is also correct that tests should check how you are doing in a real life situation. So maybe use both ways of testing?

Right but you shouldn't be tested on things that can just be "looked up". You should be tested on your ability to apply what you know, and teachers should be taking the time to write globally unique problems for every single test (gasp)

in the past where you could consult colleagues

Perhaps tests are to make sure that you are a going to be colleague worth consulting then? Yes, pure memorization and recall of facts is less valuable now than before, but knowing how to apply the facts available to you is still a skill worth having, and similarly worth testing. Those skills can also be assessed using written tests, and the potential for cheating is just as on memory tests.

what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life

Let's take language learning as an example: do you consider a foreigner with a dictionary to be just as expressive as a native speaker? Perhaps there is value in testing the immediate knowledge of the student rather than their lookup skills?

I want to test their ability to perform a task - rather than how well they memorized things. Which is not to say that memorization is not important, but it's just one piece of the Puzzle. So when it comes to language tests, rather than having the candidate fill in prepositions into blanks, let's have an actual conversation with them in which they apply their combined skills.

I actually adhere to this philosophy when hiring candidates as well. Rather than having them explain how to reverse binary trees on a whiteboard, I have them sit down and do a mini version of the actual task they will have to do in their job later. And, unsurprisingly, knowing how well they do the actual task is a much better indicator of how well they perform at doing the actual task later.

I understand what the halting problem implies, what complexity is, what Turing machine means and why they are an abstraction for algorithms, and the basic common memes of programming such as using a stack and a heap, IEEE 754 floating point numbers, 2s complement binary numbering, and so on and so forth.

Yet I don't know a lick of Go. If I were to open up a Hello World tutorial for Go during the interview, would I honestly still have a shot?

I think people who are really into "do the job for an hour" interviews don't understand this. Some skills (particularly programming languages) are just really easy to learn. I learned go in a week for a consulting job. If I had to do an interview in go, I would not have gotten the job.

Then how about you spent the week learning Go and then come interview?

Because there's a big difference between doing a week of unpaid work hoping to get a job and doing a week of paid work after you already have it.

I think this is a good philosophy for hiring employees, but when it comes to education, we wish to go beyond training employees, and what your capabilities are is not the only thing that matters. We should test practical capabilities, but we should test more abstract skills like how many facts you've ingested as well.

When I tell people a story about the history of the french revolution or something, I don't have the opportunity to go research it beforehand. I've already done that research, and I've already thought about those events for months or years, which gave me the opportunity to synthesize an insight or opinion I wish to share.

You don't get that if you don't know things in your head.

> Let's take language learning as an example: do you consider a foreigner with a dictionary to be just as expressive as a native speaker? Perhaps there is value in testing the immediate knowledge of the student rather than their lookup skills?

You may be well aware of this, but you've essentially converged on the classical Chinese Room thought experiment [0]. It's usually applied to the philosophical question as to what does and does not constitute intelligence in the context of AI, but it's interesting how it is also applicable to what does and does not constitute knowledge in the context of learning.

[0] https://en.wikipedia.org/wiki/Chinese_room

Language learning is actually a good example.

Words are rarely restricted to their denotative (dictionary) meanings. There are often abstract, connotative meanings attached to them by speakers. For example, consider the following list of words.

Hello. Hi. Hey.

These words all have the same basic denotative meaning. They're all a greeting. Yet, there is a subtle connotation of formality which is rarely described in a dictionary.

These types of connotation are not often understood by beginning language learners. It's something they acquire with time and experience in using the language. While a teacher or search could share about the formalities, the language learner is still going to require time in applying those words in different contexts to fully understand these subtleties.

> So what is the point in artificially creating a test situation where the student is isolated from resources she'll have at her disposal in real life?

I can envision many scenarios where a person needs to be quick on their feet and does not have time to look things up. Drivers, pilots, doctors, etc. Certification provides a lot of value to society.

The other point of testing is to separate wheat from chaff. To see if someone is disciplined enough to learn the material or see a task through. There are issues of going beyond the ideal amount of testing here, but not totally useless in my opinion.

True - and there's certainly a line to be drawn between core knowledge and wider knowledge. A programmer, looking up what an if-statement is each time won't do well, regardless of the resources at their disposal.

But I'd argue that realistic test conditions correct for this automatically.

If your test is "write a program that does X within two hours" the programmer lacking core knowledge will fail.

If your test is "recite 20 programming paradigms from memory" you will have to check for "cheating"

Those scenarios can be better tested through simulations. E.g. to pass a driving exam you are presented with a road situation, and given only a few seconds to respond — this completely eliminates the possibility of looking things up on the internet. A similar approach can be used for other disciplines.

> separate wheat from chaff

Wheat can demonstrate/use the knowledge, chaff can't. Why does it matter whether the knowledge is internal or external, assuming results are the same?

Because you do not know if the results are the same before making the decision to go with person A or person B. There are time costs in figuring out if someone can do something or is capable of learning how to do something, so test(s) of various kinds are used as a proxy.

Even if all tests would be open-book, there's still a need to make sure that the person who is being tested is actually the one taking the test.

This could be done with tests, by having open book tests.

The point of a test is to check if you are capable on your own, and not piggybacking on someone elses intellect or hard work.

It is true, that in the real world, the resources available to use are infinitely larger than a classroom or a teaching setting. But your idea springs from lack of understanding about the true purpose of education, i.e, to groom children to think.

Alas, modern education has become a cesspool of rote learning, atleast in my country, India.

Well, it depends.

A big part of the valuable experience comes from having memories of similar situations, and thus a shortcut to the solution. In IT it can be similar to “indexing”, like I may remember reading a relevant stackoverflow question and I know how to find it again, but I don’t remember the exact solution.

But the catch is, to index something you first have to crawl through it. In medicine for example, there may not be much point in holding every single extreme-rare disease in memory, but having memorized them once and mostly forgotten them will lead a good doctor to know about it enough so that he/she can look it up on the odd chance it gets relevant. It will be only possible when he/she memorized it once back then.

My point is, there is a point in memorization (especially that it improves memory all-around - one will remember facts much easily after extensive learning), what should be better distinguished is facts that are to be learnt for life vs those that are only needed for a given test. It’s like unknowns vs unknown unknowns - the latter can be much worse and you don’t want someone like that as a colleague.

> [...] memories of similar situations [...]

I'm going to claim that memory is only one part, maybe 1/3, of what I would like to examine for. Certainly in a job interview, but I also like university exams that include the parts below.

There's another huge part, *problem solving skills*. Working long enough to figure out which pieces of knowledge that you have apply to the situation.

There's another part, *making it work*. Software engineers deal with failures in practice (debugging) as well as theory (don't know how to do it). Sadly but interestingly, sometimes this is the point of failure for a student/examinee when they possess plenty of knowledge but don't know how to use it.

This works for concrete knowledge (declarative, procedural) where simply knowing a set of facts or a pre-existing process is sufficient to perform the job. But when you most into more abstract, creative work, this idea falls apart quickly.

You can easily find an answer to "how do I enable file sharing on Windows." You cannot easily find an answer to "how do I design a website for an engineering firm which wants to increase it's inbound leads by 25%."

The former is a simple search away. The latter requires the integration of numerous abstract concepts which the shallows of a search engine cannot return.

Further, such an approach also limits our ability to create new knowledge.

I believe the issue has more to do with the accreditation of learning institutions. “Untimed power tests” are the gold standard for testing, and in the modern world this includes own internet access. Everyone acknowledges that these are better at assessing knowledge, critical thinking, etc.. The problem is that these tests are very difficult to make and logistical nightmares concerning cheating. A timed, closely proctored, closed book/internet exam is much easy to create and administer, and it halfway (better than nothing) assesses whether or not the student has learned the material.

I agree. I dropped out of school in kindergarten. Why do I have to know my multiplication tables by heart? Why do I even have to know how to add two numbers together? I don't even know the alphabet and how to read and write - I'm just talking into a microphone and this is being written. I don't even know what country I live in or who the polical leader is, why should I? I can look it up online, I just don't have any desire to.

As you can see, your argument is silly.

It's important to know things, in your head.

Of course I realize that you don't mean people shouldn't learn the alphabet or learn math, but where does one draw the line, where they don't have to learn stuff in their head?

But there are always holes in what one is even able to look up. Maybe not for everyone, but for most.

I just and a conversation the other day about this with someone and I was flabbergasted. The topic we discussed is that anyone can create a SQL database - anyone can. But do they normalize the database to 3rd or 4th normal form? He worked with a lot of corporations, and he said most companies don't have normalized databases. In university, that was a basic thing that they taught a semester course on that. An entire semester. Not to memorize anything, but just the general concepts.

Someone can learn how to create an incorrectly structured database easily, but they don't even know about normalization, so how can you even look it up in the first place? And even if you do, somehow, learn what it is and look it up, I took the class for an entire semester. How is someone going to learn it in 15 minutes by looking it up? I looked up stuff in the book, too, and got info from the professor, and it took me an entire semester, and everyone else in the class, too. Unless you're calling us all dummies.

There's all kinds of stuff like that. You don't know how to look up something if you don't even know it exists to look up, and it is not obvious.

That sounds nice in theory, but how would you write a test for, say, basic physics, when every type of question can be Googled? Even if you make brand new questions, it's still just generally available categories of problems, like inclined planes and stuff. It's even worse with algebra, because there's software that can both solve it for you and show the steps (which you can then copy onto a test).

Unfortunately, in order to progress to more advanced topics, you need to understand the basics yourself. It's not sufficient to just be able to research it online. This is true in every field I can think of. That's why you can't just give people online access for every kind of test.

I had a basic physics teacher that allowed notes and gave us a chest sheet of his own which literally listed equations, described variables and explained the situation you might want the equation. People still failed. In a basic thermodynamics class the teacher said it was open everything but the person next to you: laptops, internet, the book etc. For fun I took the test without either book or laptop and did just fine. Others found it hard. In both cases the problems required thinking about the situation posed and which basic principles should be used to evaluate the problem.

In my opinion, cheating from internet access or outside resources is not the problem, cheating by gogling exact problems is.

Most if not all of my upper classes in school were open book. Even my beginning CS classes for Java was open book. There was no way you had enough time to finish the tests if you look up every answer. You would have to learn topics on the spot if you didn’t study, it was near impossible to do with the time constraints.

Most of my upper classes were heavily weighted towards group projects anyway.

I agree. We had open notes / open internet (no communication) exams at engineering school and they wouldn't help you pass the test if you didn't know the material. Maybe you could spend all exam period solving one of the questions if you weren't prepared but you'd be lucky to get a D.

In online tests, students cheat by asking the answers to other people online (eg. chatrooms with other students in the same class) or by having someone else do the test.

Teachers don't care about students looking into their books/pdf, what they want to prevent is them getting the answers from a snapchat group.

As a physics student in college, almost all of my (physics) exams were open book to some degree. Some were even take home tests where I had a few days to complete it, and was allowed to use software like Mathematica and Matlab.

I definitely think that is a better way to test.

It’s faster to look stuff up in the brain if it’s already there . Also having knowledge in the brain let’s it make connections you wouldn’t expect

Curricula

Tests are supposed to test your knowledge of a concept. If you have to look it up then you don't know it.

Reply on news.ycombinator.com