Hacker News Reader

Comment by ams6110 | Hacker News Reader

Really? I took a stab at R about a year ago and found it utterly confusing.

It is. It's convoluted and idiosyncratic for no reason and all together a puzzled together mess. R is not (much) better than Excel. It's better in the same way that getting stabbed in the eye with a hot poker twice is better than getting stabbed with it three times.

I’ve taught R (and Python) to mostly non-programmer scientists as part of multi-day Software Carpentry workshops. Especially with R, I feel like most attendees after a couple of days were able to do things that would be decidedly non-trivial in Excel.

EDIT: emphasis on my original point of “past a low complexity bar.” If someone is looking to manually enter data and maybe sum a row or two, Excel is probably easier. I’d say at the point where someone needs to reshape data or do any kind of non-trivial formula, R becomes easier—certainly easier for me to imagine teaching.

As a non-programmer (my job is in business intelligence so the most “programming” I do is in sql) I found R really easy to understand and use. Within hours of starting with it I was already doing part of my job in it, especially thanks to Tidyverse libraries.

I did once try to learn Java, which I did found incredibly confusing and impossible to do anything useful with.

do you mind if I derail the thread for a little bit?

[if yes] I currently work at a small-medium sized company, and we have one BI guy. Our ops team is ~5 people, and 1-2 of the more junior guys are tasked with doing everything necessary to support the BI guy. which basically means making sure the BI software is installed properly on the container, configured properly, has access to the db, and it comes up properly etc

BI is a critical business function, so it's a pretty high priority, but the process is way less automated and skillfully crafted than the other parts of the system, mainly cause everyone hates it and only barely understands what the BI system (in this case, pentaho) actually does. it's no coincidence that the most junior engineers often end up doing it (with occasional but pretty infrequent help from the seniors when blocked)

what i've been wondering is, what exactly does "a BI guy" need their system to do? I read a bit about ETL systems, but haven't done much research. I assume the job of an ETL system is to pull data from the production database(s), clean it up as necessary, and use that data to calculate whatever business metrics are required? is the software more about making it easy for the BI team to use an interface to craft their queries/dataflow/etc?

basically my motivation is I've mulled over the idea of having the ETL system be written in-house. the benefits being potentially smoother and more native operation and in a way that's more seamless to integrate with monitoring, alerting, automated jobs etc. my guess is that such a project would end up being a waste of time in 98% of cases, especially because you need a usable frontend for the BI guy.

so, if I may, what exactly is business intelligence / what do you do on a day-to-day basis? I'm pretty experienced with traditional finance/investing jobs, eg reading 10-q's, 10-k's, balance sheets etc, and using the data to calculate cash flow, opex, gross margin, and every other metric you could conceivably want. But I'd never heard of "business intelligence" until starting an internship several months ago.

thanks, and no worries if you don't have time to respond / if this is too off-topic.

The definition of what is BI varies by company and is one of the 'squishier' terms in IT becaise the needs and roles vary quite a bit. In a small company (which it sounds like yours is) one BI person does everything out of necessity. In a Fortune 500 company, a BI person would likely be on a reporting team and spend their days doing nothing but generating reports and performing analysis... there would be another team(s) to deal with ETL, and another to manage the Data Mart(s)/Data Warehouse(s), possibly another for Data Science etc etc. Your IT group sounds a bit dysfunctional, though not atypical: you admit that it's business critical but that the most junior people support it. I'm sure the business loves that (not to mention it's a bad career move for the senior technical people.)

In your company, it sounds like BI is considered to be everything from ETL (which you've got the rough idea of) to providing the final deliverables (reports, analysis, dashboards etc.) to the business customers. Essentially what a BI person would want in that environment are tools with a high degree of leverage for working with the data, probably integrated (seamless workflow, unified metadata), and more geared towards individual productivity rather than group collaboration. Basically, if your BI guy stops showing up for work, the business would be flying blind. He's the person that managers and above go to in order to get the wider view of the business. The requirements fed to him would drive many IT people nuts: 'you know that one-time report you gave us last quarter? Well, we need that same report again this quarter... with these changes... and add the data from this spreadsheet to it. Oh, yeah, there are some gaps in my spreadsheet where we couldn't figure out where to get the numbers... can you fix that?'

Regarding rolling your own ETL tool. Here's an easy way figure out if you really want to (hint: probably not) do this: offer to have your team take over some of the ETL workload from the BI person (i.e. pick a small project or two, learn how to use his tool, see how it works/what it does for him.) If he's good and busy, he'll jump at the offer. That's exactly what you'd be taking on anyway if you decided to roll your own and it's the best vantage point to collect requirements from. I think what you'll find, if you really have ETL/BI needs in your company, is that the requirements are changing far faster than your group would be able to handle. Good ETL/BI tools are generally around the midpoint of custom code (task specific, more structured, managed) and Excel (general purpose, rapid prototyping, free-form, flexible.) Fast turnaround time is the name of the game.

At the end of it you'll probably find you would no more want to write your own ETL tools than you'd want to write your own DBMS. You might event find that, if you have good tools, you want your team to start using them to eliminate some of their custom code. I've been on all sides of the data business (I've rolled my own ETL and reporting tools and used most of the major commercial ones) and when given the option of a decent tool, the tool wins over custom code every time.

That was an awesome response - that helped a lot. Thanks for sharing your knowledge.

I think the point is that formulas and macros are more utterly confusing.

Yeah, at least the R code only goes up and down, line by line. Most non-trivial excel code will have different bits of code going right and left (formulas sit in a row,column pair). One you get into extra tabs/sheets that's yet another dimension.

The problem with debugging formula code is that it's hidden in the default view, and you have to know which cells to click on to find each bit of code.

In simple spreadsheets, the proper cell to click on is obvious, because the formula only updates the one cell where the formula is. The problem comes with multi-cell functions like Excel's VLOOKUP or Google Sheets' QUERY, where a cell that looks like it contains literal text might contain the complex formula call, and then all of those around it might look like plain text, but they're also the call's results. Editing any of this wrecks the sheet.

Despite these dangers, I recently perpetrated such a thing in Google Sheets, because that's the only platform I could get to easily work across many sites, some of which have users that are often offline. The next step up from that would be to stand up a custom web app, and the project simply did not demand that amount of effort. The next-best idea was to keep passing around .xls files via email; barf.

That's the seductive nature of these one-off spreadsheet "apps": they're quick and easy to stand up, yet difficult to extend and debug by the time that they grow to the point where they would justify development time for a proper application. By then, they've also gained a big enough user base and feature set that rebuilding the app properly also means a big effort in all axes: development, testing, deployment, and retraining.

Reply on news.ycombinator.com