mastratton3

I agree with the fact about cutting out meetings but I don't know if I agree with everything here.

While its easy to say "Just leave at 5pm and come back refreshed" is easy to say, I found personally to have learned to most on the days when I've worked until 2am really digging into a problem and getting it fixed. I also have found that the best people I've worked with have a similar get it done attitude. Obviously this isn't sustainable but I think required from time to time.

I also think this lacks the notion of "how much do you want to move up vs stay in your current role". If you don't have any desire for more responsibilities than this applies, but I think if you want to move up/learn new skills, reducing the hours works against you.

I guess I follow the mantra of, "The first 40 should be productive and for the employer in the role I was hired for but I should spend additional time on top of that learning new skills"

I've just finished my first pass through the book (kind of... have to go back through some stuff.) and I concur that its a great read. I do wish there was a companion that had some code examples of some of this stuff in the real world.

For instance, intuitively state traversals or using product monoids for folding lists seem to have tons of real world applications that I've been looking for something to solve, but I find it still takes lots of exploration.

Agreed, I almost use base r exclusively during exploratory work and move to ggplot only when I want to present a finding... I guess you could say its in-efficient but it seems to work.

Awesome, I looked into immigration processes a couple years ago and it seemed almost impossible.

So I did find it useful for doing additional exploratory aggregations once the data was already cleaned and denormalized. My comment was more directed at the upfront initial data processing (In our case, extracting time series data out of a large amount of files).

I did hit issues w/ multiple joins and shuffling though. Have you not hit issues w/ shuffling?

I was using Spark 1.5.1 for the record.

I've been personally very impressed with Spark's RDD api for easily parallelizing tasks that are "embarrassingly parallel". However, I have found the data frames API to not always work as advertised and thus I'm very skeptical of the benchmarks.

I think a prime example of this is I was using some very basic windowing functions and due to the data shuffling (The data wasn't naturally partitioned) it seemed to be very buggy and not very clear why stuff was failing. I ended up rewriting the same section of code using hive and it had both better performance and didn't seem to have any odd failures. I realize this stuff will improve but I'm still skeptical.

Did RadioShack have any customers?

In terms of "How will you make money", are you interested in the market for the initial product/use case or what we think the product can eventually grow into?

ie: We think our total initial market has the potential to be around $30M/yr but can grow to take on other use cases.