Microsoft announces major commitment to Apache Spark | Hacker News Reader

Microsoft announces major commitment to Apache Spark

blogs.technet.microsoft.com

124 pointsRmDen48 comments

Usually when I hear about Apache Sparc, there is Scala being mentioned too. Does Scala have the best support for it and what's the situation on .NET side? (F#?)

I would be surprised if it didn't end up being C#, but I have no real foundation for that sentiment.

First class language support in Apache Spark:

  * Scala
  * Python
  * Java
  * R

All these languages are equal, but Scala tends to be more equal than others in some areas of the API. I also believe R is mostly restricted to the DataFrame API.

Third-party language support:

  * Clojure [0]

To develop on Spark in a new, non-JVM language, you'd need a bridge to Java. That's how PySpark works [1], and I believe R follows a similar pattern.

[0] https://github.com/yieldbot/flambo

[1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+In...

I am thinking that R is the reason why this is happening. Now that Microsoft has supported and invested in R (Smart thing on their part) R will get more Spark features. Having better R connections to Spark's API would be amazing.

First class support for running dplyr pipelines on Spark would be killer, imho.

Interesting how the whole R thing happened, about three years ago I wouldn't have any idea what R is about and now see it everywhere, even at customer sites.

Haskell: https://github.com/tweag/sparkle

There's also third-party support for .NET via https://github.com/Microsoft/Mobius

Third-party from Spark's perspective, but first-party from C#'s perspective.

Third-party Node.js bindings:

https://github.com/henridf/apache-spark-node https://github.com/EclairJS/eclairjs-node

More like 3 tiers of support:

Scala -- native API

Java -- java wrappers for Scala API, slightly clunky but same implementation

Python+R -- janky process involving forking a process and feeding strings back and forth between JVM and python/R via pipes

You're right about Python and R having to pass data back and forth to the JVM for certain operations, but also keep in mind that native code still runs in the native interpreter. That means you have access to the full ecosystem of the native language.

For example, if I want to convert an RDD of JSON strings into Python dictionaries:

    import json
    rdd_dict = rdd.map(lambda x: json.loads(x))

Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]!

For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API.

[0] https://spark.apache.org/graphx/

[1] http://stackoverflow.com/q/23995040/877069

[2] https://github.com/graphframes/graphframes

[3] http://stackoverflow.com/a/37150604/877069

[4] https://github.com/apache/spark/pull/2144

Has anyone tried running Jython on top of the native API? Would that make any sense?

Spark's syntax is basically identical to Scala's, so you can use it as a library with standard Scala functions. In contrast PySpark etc. will still let you get all the same features, but you have to write code that doesn't look like Python, by either using Scala-like code with the RDD API or SQL-like code with the SQL API.

Spark's written in Scala, so Scala will presumably always be the best-supported language. On the inside Spark's also pretty intimately tied to some Java implementation details (e.g., it uses Java serialization to push code around), so any non-JVM language is always going to have to deal with a bit of friction.

That said, from my experience the Python integration is still first-class. I think it might lag the Scala APIs a bit, but that isn't a huge deal unless you really need to be on the bleeding edge at all times. The only really big downside in my book is that there's a small performance hit you take by choosing to use the Python API instead of the Scala one, and that might compound into something of practical significance if you're using it for really big data tasks.

I assume you'd have similar problems using it with R or a .NET language.

There are some people doing ML and data analysis in F#, mostly financial institutions, but Spark is written in Scala so it is all about the JVM eco-system.

Can't wait until they install it on my computer!

I assume that Microsoft considers the Windows EULA to let them use your computer as an Azure compute node.

This is actually a rather interesting idea.....allowing users to voluntarily participate as a compute node would be extremely cool.

Users can contribute idle CPU cycles to a variety of projects http://www.hyper.net/dc-howto.html . AFAIK, lot of folks have been doing this for decades.

https://lh3.googleusercontent.com/-45xROH3jwF0/VuFHctFcV_I/A...

Is it just me, or does any technology that Microsoft supports late in the game give you sort of a queasy feeling?

Easy to see their motive. This is good for their cloud business, they already know their monopoly days are over, they're making money with Azure and growing that front (127% [1]). Whatever your platform is, you can host it on Azure. And big data will be more lucrative each day. Spark is a cluster computing framework and clusters lead to cloud.

[1] http://www.businessinsider.com/microsoft-azure-vs-aws-revenu...

"But there's no way to tease out how much of that $5.1 billion is Azure versus the older products. We also don't know if Azure is profitable, althoug the overall business segment earned $2.58 billion in profits — down 1% from a year ago."

Servers and tools did $4.5B of revenue in FY12, Q3 (https://www.microsoft.com/Investor/EarningsAndFinancials/Ear...). This was before Azure really had any footprint.

You do the math.

Why, and what exactly do you mean?

The parent is probably having flashbacks to a time now long gone, in which Microsoft was widely regarded in tech circles as the evil empire. They would attempt to 'embrace and extend' potentially threatening (to them) technology that they didn't control, as a means to hijack or kill it. Many people - including internally at Microsoft - often called it: embrace, extend, extinguish:

https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish

I don't know if Microsoft's management still considers E-E-E to be a viable strategy or not.

However, even if they don't, I haven't seen any indication that they've repented of their past scummy tendencies. The Windows 10 rollout is perhaps the most obvious case in point.

I'm curious if you've seen something that makes you more optimistic than I am.

Whether or not you consider their actions evil, they were demonstrably an extremely destructive force, provenly using very untoward methods, much more so than most of their peers.

It's mind boggling to consider how many years their shenanigans have probably set back computing as a field.

As for trusting them now, I don't believe an organization like that can truly change in such a short time.

I'm not sure they can change fully in this short a time, but they've definitely demonstrated some healthy changes under Nadella, including "embracing" open source, VS code, proper handling of the Mojang acquisition, and others.

They just bought Xamarin and killed Robovm.

No, but I view MS as a collection of employees desperate to learn what they need for the next job now.

Hence, the total apparent lack of desire to do anything new and innovative ("Let's copy AWS! Genius!").

Funny story from my time at Microsoft (this wasn't representative of the entire population, just a few folks). This was circa 2013:

We were in a meeting talking about infrastructure and testing. I said that we used EC2 for some of our infrastructure, and we were thinking of moving some of our local dev infra to AWS. Someone in the room asked me what that was. I responded with "Amazon web services" and they countered "you mean the company that sells books and stuff?" -- they had never heard of AWS, nor EC2

This comes from someone who had been with the organization for some time. He was very well respected by a lot of folks. He had more awards in his office than I could count. I still think he's a great guy.

The Microsoft monoculture hurt quite a bit.

Why wouldn't you use azure instead?

If I were to guess, to avoid having to go through the meetings and internal bureaucracy for it. It'd literally be cheaper (after counting in person-hours of time) to just pay for an external vendor.

There's no extra meetings or internal bureaucracy for using Azure internally. It's painless to get an internal subscription for Azure that can be used by your team.

Perhaps that was not the case in 2013?

Did you try using Azure to run Linux workloads "at scale" (100+ nodes) in 2013?

Cheap shot, but this is the company that created TFS in 2005(?). Which as near as I can tell is an answer to the question "How do we reimplement SVN in an MS environment?" Which seems like something no one should have been asking for by that time.

TFS goes rather a bit further than just being a version control system.

That said, I'm not entirely certain why they decided to implement TFVC (TFS's default - and until recently, only - version control system) instead of just basing it on SVN. Aside from that this was at the height of the period when Microsoft had a compulsive need to sink resources into building their own proprietary clone of every damn thing.

Well, I love those gated check-ins, specially when working in offshoring projects.

Microsoft's dev tools were straight up amazing. As someone who didn't get to take advantage of many of them, I was always envious of engineers using Visual Studio and being able to debug code like it was magic.

Although TFS may seem terrible, it works for super large organizations, and big code bases in a way that SVN and Git just didn't.

I wouldn't say I worked with a super large codebase at the customer who used it, but the only compelling feature seemed to be "it played nice with MS infrastructure."

Teams that had issues grasping the fundamentals of VCS in general still had issues. And in return you got the lesser compatibility and greater number of code warts that came with proprietary enterprise software over open source.

But I'm curious, what worked specifically for you in TFS that wasn't in the SVN ecosystem? It's probably the client was unaware of a lot of features.

I never got to use TFS, but I'd sum up the benefits with /integration/. The MS dev ecosystem is amazing. I write software now for Linux on Mac OS X. I use IntelliJ, Netbeans, virtual box, and a whole bunch of bash to tie it all together.

If I want to do a deploy of my lambda software, I edit some Python in IntelliJ, I then run a build and upload it to lambda using some bash scripts, and test it using the GUI in Chrome.

If I'm editing C code, I edit some code in Netbeans, and edit it over Netbeans SSH integration in VB. I then go compile it using make on the remote machine, and load it. I have a bunch of bash scripts to do testing on that remote machine that are executed via SSH.

----- I have several more duct taped integrations. With VS, this is all in one system.

I don't think there's a lack of desire, I think there's a lack of ability due to size. I also don't feel it's fair to say that their attitude is "Let's copy AWS! Genius!", Azure has become very mature and a delight to work with.

I think Nadella doesn't know what he's doing. They are doing a lot of "cool" stuff but it seems like a disorganized mess.

Reminds me of the Jonathan Schwartz days. At least we don't have to worry about Nadella growing a ponytail.

What part of it seems disorganised? Not trying to be a dick, their roadmap seems clear and detailed and well received by their target audience (primarily MS developers).

I've seen HN light up with joy at news of all the open sourcing and enhancements MS are undertaking in recent years. I don't see the disorganised mess but perhaps I'm missing something?

Not a surprise, since https://www-03.ibm.com/press/us/en/pressrelease/47107.wss

Reply on news.ycombinator.com