Comment by maxxxxx | Hacker News Reader

Comment by maxxxxx | Hacker News Reader

Parent Full thread

I can see this in my company. IT has for some reason decided to put some data into Hadoop. Now the users need to write spec documents for each little thing and every trivial report takes months to get done. Before they had Excel and could create whatever they needed in minutes. Us programmers don't always make life easier for end users. Often we create a level of bureaucracy.

EDIT: I like a lot of the responses here. They clearly illustrate the disconnect between end users and IT/devs in many companies.

If you're putting Excel-sized data in Hadoop, something has gone seriously wrong in your IT department's decision-making process. It does make for a nice parable about the dangers of shiny-new-thing-syndrome though.

Yep. this is going on everywhere. How many solid but boring Java applications have been replaced by a microservice, cloud, NoSQL, NoieJS monster?

http://www.zdnet.com/article/disgruntled-developer-breaks-th...

Not really. They could be trying to build a company-wide data lake. Some files in the lake will be very small, others will be very large. Because it's in HDFS it won't matter if the very small ones suddenly become very large.

End users groan because people groan every time anything changes and they have to learn something new.

If users were previously empowered to do a task themselves, and now need to request another team do that task on their behalf, that's a very valid reason to be groaning IMO.

You have to understand that enterprise class companies don’t want Access databases under people desks. It might be valid reason for the sales or ordering guy who loses control to complain. For the IT professionals at the company it allows them to create a scalable solution that can be used at an enterprise level.

And suddenly every addition that could be done in 5 minutes now takes months or gets rejected by the devs.

Hadoop isn't Oracle. If you need to write a file to the data lake, you write it. If you need to read something out you do that too. There's no schema, it's actually a file system with actual files in it, like your shared drive. Except that it's distributed, can handle arbitrarily large files and lets you perform queries on those files instead of downloading all 2TB to access a single row.

I mean, end users also groan because the user experience for spreadsheets is far better than writing map reduce jobs. One is a skill that a large percentage of BS graduates have, another is a skill that a small percentage of CS graduates have.

You don't need to write map reduce jobs to access data in Hadoop. In fact, you can do it from excel:

https://support.office.com/en-us/article/Connect-to-a-Hadoop...

They're practicing RDD -> "Résumé Driven Development"

Agreed. The problem is that RDD works extremely well if you want to get ahead.

you laugh, but a guy who knows Hadoop is worth more than a guy who doesn't all else being equal

One must disagree with this. It doesn't matter what the technology is, if one has the appropriate mindset and enough years under the belt, it will be picked up.

What matters in terms of better value is whether or not one can understand the problems space and find an appropriate solution for it. All sorts of solution space solutions can be used, some of which are better than others. The point is to try and keep it simple enough for the user, irrespective of how difficult it might be for the IT guy.

The knowledge of Hadoop in and of itself is not as valuable as one would suggest it is.

[Edit fix wording]

Unfortunately that's not how the job market works in most cases.

That's only true if the management level can't see that they are screwing themselves over, either willingly or ignorantly.

Hiring a hadoop guy for what used to work in excel is a bad decision in most ceases...

Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense. Also putting the data into a secured multi user environment with redundancy and also business continuity.

as opposed to fecking sharepoint

Sharepoint is where good information goes to die. It's a write only wiki - you put stuff in and you can never find it again.

Mervellous. Maybe this is true for a lot of DMSes, as well. But marvellous. :)

Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense

Yes but https://aadrake.com/command-line-tools-can-be-235x-faster-th...

Sometimes, but it depends on what you are doing with a data set - even if it's 1.75Gb : from the link...

"Since the problem is basically just to look at the result lines of each file and aggregate the different results" which is why this works on this data for this problem. Many problems are not efficient to solve in this way, for example searches through large products and expansions, especially where the evaluation function is costly.

Some problems are data bound - you are handling 100'sTb of data, hadoop can be a good fit.

Some problems are CPU bound, and embarrassingly parallel - you require massive parallelism to get through the core seconds that you need to solve the problem. If you need 1 million core seconds and you have 1000 cores, and a way of getting the 1000 cores to do 1/1000th of the work... all is good.

Some problems are memory bound. You can build single space machines with many TB memory now, also you can cache with SSD's and optanes more than in the past, but Hadoop does offer a utility for these problems.

But in a corporate all problems are admin and discovery bound. If your dataset is what I need and I don't know about it, can't find it and anyway if I did then you won't give it to me and I can't unlock it legitimately; I am screwed. If it's a proper corporate asset then I've got a fighting chance.

> Us programmers don't always make life easier for end users.

I'd go so far and say: We programmers often make life more difficult for end users.

One would agree with this. Most IT groups I have come across seem to go out of their way to make life difficult for the end user.

This has been a source of work for me over the years, providing a means to bypass the IT department to get end user work done.

Us programmers also think about things like failure scenarios and disaster recovery. It may be trendy to criticize us for overcomplicating designs--and many do--but non-tech people using Excel are often doing more than just getting the job done...they're creating a disaster waiting to happen.

I wouldn't generalize too much. I once took a job where a $50 million in revenue a year business was running on a single AWS server that included everything (web, database, cache, etc.) with no backup of the data and the server was hand-maintained.

It was truly a disaster waiting to happen.

I have been on both sides of the divide and my view is that the overcomplicated designs are more hazardous. If one understands the end user and their requirements, one is generally able to steer them in the appropriate direction.

However, the general view of IT departments is that they just dictate what they want and not help with what is needed. policy is set and then demanded to be followed with little or no justification to the end user why and how it will benefit the end user.

The number of times that I as IT professional (designer, developer, etc) have had to fight the IT department over some problem to get it solved is far too many. Instead of fixing or changing something that will allow end users to actually get on with their word, IT departments are seen as dictatorial time wasters who don't get back to you to sort out what is needed.

Part of our job is to think of failure scenarios and disaster recovery. This should not get in the road of those using the systems being put in place. The end user has a job to do, we as professionals should be providing good solution to enhance as well as protecting them from any disasters.

Anything that can be done in minutes with Excel could be done in relatively short order via an actual programming language.

By accountants and financial analysts?

With Excel any reasonably capable business analyst can do sophisticated calculations quickly without needing anything else. With a web app, SQL database and whatever the ramp up cost is huge in comparison.

The ramp up cost isn't months vs minutes. That is just pure nonsense.

I disagree. I am working on something right now at my employer where we are taking spreadsheets that have been used for quite a long time (and were created fairly quickly) and they are converting all the business logic to code. Each workbook is taking weeks for them to code and test and then send back to us to sign off on.

I'm a coder DOING some of these conversions. I see both sides of the coin here. One additional point I wanted to mention is that once we have done one or two, or a dozen of these conversions, we start to see common themes.

More often than expected, there are mistakes and misunderstandings in some of the spreadsheets. There are inconsistencies in the spreadsheets. In IT we also get calls that a spreadsheet is "locked by someone." Or that someone deleted a spreadsheet.

With code, and databases, we can ensure consistency between operators. We can also leverage the data across other applications since it lives in a database now. We can pull reports for management more easily, and with more automation.

As some other comments mentioned, spreadsheets are great for prototyping, but also have their drawbacks.

It certainly could be if these tasks now need to be delegated to a centralized software dev team who is fitting them into their project queue.

With Excel I can do things fast, at the beginning, but then I find that it get a bit more tricky, I get lost, I find out that things are wrong, I do more, I want to move to R. I can't move to R. It will take me a week. I have a day. Then I want to move to R, it will take me a month, I have a week, then I want to move to R, it will take me a year, I have a month. Then, I want to drink bleach.

Adding a new screen to capture information? With a nice graph to display for output? Does it conform to your UI style guide? Does it have passing unit tests? Is it integration tested? Is it code reviewed? Any migrations necessary? Any business continuity ramifications? Etc.

Engineering has overhead otherwise it's just a bunch of cowboys that are no better than Excel hackers - and probably worse, because they have a lot more rope to hang themselves.

Does your '=vlookup(x, ...)' have tests surrounding it? How about code review? How about source control?

I think you missed the point. The comment you replied to is stating that excel does not have these things. Proper software engineering will have these. Excel is fast, but also easy to repeat a mistake. With unit tests, software is much less likely to repeat a mistake. Once the mistake is made, found, corrected, unit tested, making the mistake again is likely to be caught by the unit test.

In my experience, the fact there is a human being staring at the output all day goes a long way towards mitigating the lack of automatic tests.

As programmers, we write tests because we can't see what's happening inside the program while it's at work, so we have to give it a thorough exam before letting it do that. Spreadsheet jockeys can take a radically different approach to validation.

There's also a bevy of issues that impact programmed systems that (in theory!), don't apply to Excel.

I can't guarantee the a programs output will be the same after every VM patch, component update, re-deployment, or even just re-compile. Generally we have multiple devs touching the same system, or same code. Generally we want that code to be robust in the face of reuse in other components and contexts. Generally we have no ability to see production errors unless they are properly logged and monitored, etc.

Worst of all I, a subject expert in nerdy things, am implementing a model in a domain I am an amateur in and have no fundamental context for assessing.

A spreadsheet jockey inverts all those problems. They know what they want, and are intimately connected to its operation.

Actually, in my experience "sanity checks" were very often built into financial models.

Does your balance sheet balance? That's a pretty basic "unit test".

Do your calculated values match the financial statements values? No? Throw an error.

Divide by zero? Reference a blank cell (similar to an unset variable)? Using a value as a number? Referencing outside of a range? All immediately throw an error that will flow through your model.

And code review often does take place - in banking at least, models written by analysts are checked by associates and then VPs. People who understand what the numbers should look like can spot mistakes pretty quickly using a TI calculator.

Source control is a problem, but many models are one-offs for specific projects - no company (or deal) is structured identically.

You're identifying problems that have, for the most part, already been solved. Not perfectly, but in ways that work well within the structures that have been built around the use of Excel as a core tool.

Erm, they would also need a programmer at that point. More to the point, excel is more of a thinking tool that allows for lots of tinkering with real time feedback. Programming languages require lots of thinking up front and are not meant for thinking without significant augmentation (REPL, Jypiter notebooks).

Excel is a garbage environment and its pretty trivial to find environments that provide both a capable environment AND immediate feedback.

Please tell me where every user can create his own gui within 5 minutes with shapes, activex controls, pictures, keyboard shortcuts, combine those with code which was created by macro-recording (and as such needs the most basic knowledge of programming if any), that can be opened by almost every PC in the business world (as 99% has excel installed), which comes with 100's functions and formulas allowing to create anything and everything. On top of that, please no higher costs than ~50$/user.

Delphi, before Embarcadero shat the bed

It might take a time till coder has time for it. And coder may misunderstand requirements and needs time to get them.

if it's important then coder has time, coder will understand, and coder will spend the time.

You ever tried to implement a generic pivot table builder? Good luck. See you in 2019

I have! Twice :)

https://pivottable.js.org/ and https://react-pivottable.js.org/

Reply on news.ycombinator.com