Hacker News Reader

Comment by w_t_payne | Hacker News Reader

Hooray! Some sense at last.

I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth.

All of them wanted to feel like they were doing something special.

The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were handling.

Hadoop was required in exactly zero of these cases.

Funnily enough, after I left one of them, they started to build lots of Hadoop-based systems for reasons which, as far as I could fathom, had more to do with the resumes of the engineers involved than the actual technical merits of the case.

Sad, but 'tis the way of the world.

View on HN

There is principal-agent problems in a lot of companies.

CEOs want to run bigger companies, even if it's not best for shareholders.

Managers want to run large mini-empires, even if it's not best for the corporation.

Engineers like to use the latest tools, even when they're not best for the problems at hand.

Solving this requires well respected leadership that knows enough about the details, and which battles are worth fighting.

Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external.

The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for just over £1,000,000 supplied with one full time UNIX monkey to keep the plates spinning.

Turns out they had only 10 CAD/engineering staff and about 8Gb of data in an Oracle DB which a single Sun Ultra desktop (<£10,000) could have handled without a problem at the time.

When in the 90's? I'd expect it would have taken the full 2x 42U racks to keep the whole dataset in memory at one time. Not to say it's required, but that doesn't sound totally crazy for the time period.

Also, having an order of magnitude more hardware than you ought to need is totally understandable when hardware capability is increasing by an order of magnitude faster than hardware replacement cycles. It's when you're off by several orders of magnitude that it starts to become suspect.

1998. It had 4Gb of RAM in each node which wasn't that scary. Most of it was UPS and disk.

I had a Sun 1000E [1] for 3 years as a workstation from 1996. 8x 50MHz CPU, 2Gb of RAM, 16Gb disk array. It was made in'93 and was approx 12U for the unit and array. Things weren't that big in the 90's. 80's yes...

[1] http://en.m.wikipedia.org/wiki/Sun4d

Yeah... it looks like my employer is going down the same route with an EMC Isilon NAS system.

We really just want high throughput for linear scans; possibly with a small amount of parallelism, which would argue for a fairly simple solution with as little cacheing as possible (ideally none whatsoever) ... but I cannot even tell (from the website & the literature that I can access) how these things are supposed to operate -- which is a worrying sign in itself.

Previously, I got good performance out of a SAN using a handful of NexSan SataBeasts -- which I would have recommended to use again, if anybody had bothered to ask.

:-)

Isilon is nice - expensive (it is EMC), but it gives good performance and is reliable.

After I moved from academia/gov't labs to private industry, I was absolutely flabbergasted at the waste. Many companies literally just buy whatever their vendors throw at them to solve a problem, and if a reasonably intelligent techie spent just an hour or two thinking through the problem, they'd realize that the solution was either far less complicated than the proposed kit or far more; either way, what's proposed is almost never right.

Another thing I haven't seen much of is acceptance testing. This is actually more surprising to me than the reliance on vendors for problem solving. It's an absolute no-brainer to say "I have problem X, which will be solved by solution Y, and the success will be measured by the set of tests Z. Unless the above happens, we will not pay the bill and you will take the hardware back." The vendors looked at me like I was from Mars when I started suggesting these.

I have not since complained about government waste in research computing budgets.

Outsourcing your own teams means there are no in-house experts to perform the oversight role you've described. Thus the principal-agent is amplified.

Agreed, but I'm not optimistic that the view will become wide-spread.

There was a phrase I learned long ago: "if it fits into cache it's not a supercomputing problem."

This essay promotes the related view: "If it fits on a desktop, it's not Big Data."

BTW, you might like Greg Wilson's 2008 essay "HPC Considered Harmful". I can't find the slides anymore, but there's a lovely interview with him about this at http://itc.conversationsnetwork.org/shows/detail3682.html .

Just because it's easy to lose track, I just checked, and "Stereotypical Grandma walks into Best Buy and buys their best desktop because she doesn't know any better" "(and probably got bamboozled by the sales guy)" ends up with Grandma having 16GB of RAM for $900.

Tricky catch 22. You want the kind of employees that are interested in what they do enough to want to learn new skills and try new exciting technologies. But, you want them to actually use something boring.

There's nothing boring about pandas

It's even more exciting mixing pandas with the IPython notebook.

heh. Until you actually have to repeat something! It's quite useful for iterative refinements like tweaking pyplot graphics, but it's all to easy to lose track of what versions of which blocks executed to create the current state.

Maybe I just need more discipline..

Walking on a highwire is also exciting. :)

Until you run out of memory.

that hardly makes it boring... it's just inappropriate for some kinds of task

That's when the fun starts!

Reply on news.ycombinator.com