Teaching Students to Sift Mountains of Data | Hacker News Reader

Teaching Students to Sift Mountains of Data

nytimes.com

34 pointstokenadult8 comments

I am reminded of the early-90's complaints from IT organizations -- "You [Univeristy X] don't teach your students how to do anything useful like Visual Basic 3, etc. They come to us, and we have to teach them these basic skills."

While techniques for dealing with large volumes of data are certainly more general (and intellectual) than VB3, these complaints are similar in nature. I don't hear Google complaining about deficiencies in top-ranked undergraduate CS programs. It's not as if a basic knowledge of algorithms and data structures is not still required (if anything, it's more valuable with large datasets).

For years, universities have thought their mission was to educate, and businesses thought their goal ought to be training. Interestingly, those businesses with reputations for hiring top talent don't seem to complain about specific coursework. It's not like the mid-90's Microsoft complained that its recruits didn't know VB! And, Google certainly has a penchant for hiring smart people without regard to specific technical knowledge.

Interestingly, those businesses with reputations for hiring top talent don't seem to complain about specific coursework.

Yes, because they assume they can train you. If you're a smart bastard, you'll learn fast, and almost anybody will learn faster by applying knowledge than by reading a textbook.

"And, Google certainly has a penchant for hiring smart people without regard to specific technical knowledge."

Again, yes. What does this suggest? Google (and like companies) are hiring for intelligence, not specific skills. What does this suggest combined with the lack of concern about specific coursework?

They're hiring for intelligence, they're not that bothered about what you know, and they mostly recruit from top schools. This suggests that the top schools mostly act as a filter; anyone who got into School X is a smart bastard, so we should give strong consideration to hiring them just because they're smart.

So Google could cut out the middleman and hire based on SATs and cut out the middleman. Why not? They like some skills, and getting through college signals a certain degree of conscientiousness, which is good to have in employees.

The news article is about the joint NSF-Google-IBM Cluster Exploratory program (http://en.wikipedia.org/wiki/Cluster_Exploratory), which grew out of a Google project to bring MapReduce into undergraduate education (see, e.g., http://www.businessweek.com/magazine/content/07_52/b40640489...). The goal of that program: train the next generation of Google engineers. Based on the evidence, I think your point is exactly wrong.

I was lucky enough to write software for government supercomputers and work with government scientists. From these experiences, I've come to the realization that "science has turned into data management" is completely the wrong conclusion. In 99% of scientific supercomputing studies, if you do not know what you are looking for (or at least have a very good guess) before you start looking, then you will not find anything in your data -- you only have time to code and wait for a few analyses.

Unfortunately, the systems that scientists study with supercomputing are often beyond our understanding. Even with the supercomputers, we can not form good hypotheses and therefore we learn very little.

If finding useful pieces of information from huge databases was really hard search engines would advertise smaller databases. But, fear of large numbers makes a far better story.

PS: The only way a scientist is overwhelmed with information is when they need to do something by hand. It's hard to guess how a scientist would like to collect less information. Worst case you ignore it because you can't process it yet.

>If finding useful pieces of information from huge databases was really hard search engines would advertise smaller databases.

Come again?

> PS: The only way a scientist is overwhelmed with information is when they need to do something by hand. It's hard to guess how a scientist would like to collect less information. Worst case you ignore it because you can't process it yet.

I am still trying to parse that statement. 12 years ago we needed a 250 node cluster to get anything done with all the data that was presented to us and that's a fraction of what's being generated today

>> If finding useful pieces of information from huge databases was really hard search engines would advertise smaller databases.

Some do. See http://alltop.com for example. But the bigger point (and see Google as the case study for this) -- the fact that it's hard is what makes it valuable.

Reply on news.ycombinator.com