frak_your_couch

I'm the author of the blog post, so if you have questions or comments to me then you can reply here. :)

So, you guys forked gemfire, spark and spark-jobserver. Are you planning on going back into mainline or will you be maintaining the forks to stay current at some interval?

Hey, sorry, got busy yesterday and forgot to respond to this.

Well, I work for a Hadoop distribution, so I may have some biases showing through in my setup. I like to use spark in conjunction with Hadoop; I've never actually used it stand-alone before, honestly. For relational data, I'll ingest into Hive as that allows me to pivot to the right tool for the kind of analysis that I need to do whether it be simple SQL via Hive, something more suited for Pig via HCatalog or Spark via SparkSQL. I'll often do my analysis on a Hortonworks Sandbox for small data like this.

For larger data and a more professional setting, I like to do prototyping/ad hoc investigation/etc in python with pyspark inside of jupyter. Generally that transitions either to Java or just plain python (depending on the degree to which it's difficult to transition).

Anyway, hope that helps! Happy to answer any other questions you might have too. :)

Shameless plug for a series of blog posts that I wrote in 2014 about this dataset. The aim was to demonstrate how to analyze data with Spark and Python, but probably the most interesting two sections turned out to be Outlier analysis (http://blog.caseystella.com/pyspark-openpayments-analysis-pa...) and Benford's Law analysis (http://blog.caseystella.com/pyspark-openpayments-analysis-pa...)

Great idea! I signed up, but I worry that there's no geographical adjustment for salaries. For instance, I make $200k and live in Cleveland. Even 20% more in NYC or SF is a very different lifestyle.

If you are interested in this, you might be interested in my (warning: shameless plug) 5 part blog series located at http://blog.caseystella.com/pyspark-openpayments-analysis.ht.... I'm using the python bindings for Spark to illustrate doing data analysis on healthcare financial data on Hadoop.

I agree, I was super skeptical when I started with my current employer due to this policy. I cynically assumed it was intended to somehow deter me from taking any vacation. Then I took quite a bit of vacation and got absolutely no pushback.

I think that our relationship with vacation has to change to one where we consider it a vital part of being a professional. When that attitude shifted for me, I became a pretty big fan of the unlimited vacation policy.

I figure that the worst case scenario, I'll get denied the vacation and I can choose if I want to move on or stay..no harm done.

So does Hortonworks as of 2.1

http://hortonworks.com/blog/announcing-hdp-2-1-tech-preview-...

Just wanted to add that I'm the author. If anyone has any questions, I'm happy to comment here.

I've been the engineer in that scenario a number of times in my life. Almost every time there were management people who stayed who just couldn't contribute. Honestly, I didn't particularly need them to suffer. I'd rather they go home, recognize the situation and reflect on whether something should change if this is a semi-regular situation..and then act. That action is within their capacity and it would have been impactful to me.

As it was, they just didn't act, treated the suffering as a badge and pretended they were in the foxhole with us. If they could handle it, why couldn't we? Why were we bitching so much? They were right there next to us! Needless to say, I don't work in those kinds of places anymore.

I guess this means IPO isn't going to happen this year.

First thing that comes to mind is maybe you're underselling your successes in your resume. In particular, if you've been working a long time in the banking sector and you don't mention sufficiently prominently that you have experience outside of that domain, then you could get pigeonholed.

Second would be that there is something about your non-technical skill-set that is sending up red flags. What feedback did google UK give you after the rejection? Was it a bad fit technically or culturally?

Hey, sorry, one more time. The updated link is also wrong, it's relative to your blog rather than absolute i.e. http://technocalifornia.blogspot.com/2011/04/research.yahoo.... as opposed to http://research.yahoo.com/files/recsys2010_submission_150.pd...

Just curious, why would you tell them your current salary? In salary negotiations, wouldn't that give the employer a substantial advantage in bargaining?

I've been asked this before in post-interview negotiation, so evidently this is somewhat common practice. I always respond with, "I'll tell you that if you give me a salary breakdown of your engineering positions by experience level." They always back off at that point.

I just wanted to point out that the link on your blog to PureSVD points to the wikipedia page for Discounted Cumulative Gain.

Well, I do disagree on the quality aspect. I'd like to see a high quality Calc I and II textbook that's free, coherent, and not aimed at MIT students. My wife is a faculty member and if you can find me one, I'll pass it along (they're constantly looking for one). It's just hard to not choose Stewart..it's a solid textbook for a broad range of students, but it's expensive as shit. A lot of professors are getting that this (if they don't, I make damned sure I mention it when we socialize with them) is a barrier for a lot of students.

Anyway, you're absolutely 100% correct about problem sets. Good problem sets go a VERY long way to making a good textbook.

I have not, thanks for the tip. I looked at a book or so, but didn't see an easy away to comment on individual paragraphs like in "Real World Haskell", but maybe more of the interface opens up if you log in. ANyway, thanks for the tip!

Well, I'd love and welcome a good high-quality open source textbook movement that happens to also be high quality. It's the quality issue that's the major problem at the moment, I'd say. I've LOVE to see, for instance, Math textbooks written like they wrote "Real World Haskell" in wiki format with a single (or a small,finite set of) editor(s) ensuring that content and quality is up-to-snuff. I think that there really should be minimal educational infrastructure provided for free or very low cost.

I think you defended your use of satire very well. Good job.

Yeah, I came across Barr and Wells 5 years ago in grad school and found it almost impossible to buy. I ended up getting a copy from the library and photocopying and rebinding it. Like you said, it was ok, but hardly comprehensive. Maybe it's easier to find now.

Anyway, nice paper..and thanks for making me think of category theory again. Maybe I'll pull out my old copy of "CT for the working mathematician" and give it (and haskell) another go.

Edit: I came across this: http://www.amazon.com/gp/product/0521422264/ref=pd_lpo_k2_dp...

Anyone gone through this book?

Any suggestions for a good Category Theory texts that're oriented to CS and go beyond the "classical" connections between cartesian closed categories and lambda calculi? Or am I stuck in another frustrating 2-week long session trying to divine "Category Theory for the working mathematician" and a hand-full of research papers and blogs on Haskell?

Edit: Sorry if I sound bitter, but I've been down the category theory route multiple times in the past, starting in graduate school (in Math) and I've always given up before I "got" the divine revelation that others have talked about.