samkone

I know two companies running running openshift on Mesos. The reason modern Paas aren't enough, is the factat some scale you're not just running web services. But also rather complex data pipelines involving distributed data systems like kafka, spark, cassandra, etc .. and a simple paas has issues handling those workloads. That's where Mesos shines. As for the user value, I consider having an uptime system, providing reliable and intelligent service based on data processing, by efficiently using your resources is has an important business value.

This cracks me up a bit. I don't have anything against whiteboarding, even though I find it meaningless. With this you more often than not have false positives. And if Google had so many resumes they have to filter through, then why do they have all these recruiters literally spamming you to join? they have recruiting issues like many others tech companies, but the whiteboard is really a bad filter. I've seen very much capable,competent and experienced engineers being denied just because they didn't spend too much time learning cute algorithms or coding in the whiteboard, in front of interviewers sometimes more focused in showing off how smart they are.

Yes opscenter, netlfix's priam, Cassandra cluster manager and the usual monitoring stuff, Datadog in our case. So far so good.

If a company blames its demise on a piece of tech, then they either need better engineers to change that tech. Or the product had no future at all. And Cassandra has changed a lot since then.

So are you still running Cassandra? And why rely on counters for analytics when it's wildly known to be a bad idea.

Google doesn't offer a managed cassandra, rather a manager Bigtable with the HBase Api. But we run a huge Cassandra install in multi dacenters on GCP. And we're good. And yes it requires you to understand distributed storage systems, and the some of Cassandra internals.

Here's laughing. I guess you chose convenience over performance or scalability, which is a valid choice. But We're running a 1000 Cassandra nodes to handle Millions of ops where mongo would simply require too much admin work. the whole clusters are managed by two guys. Mongo's simplicity comes at a cost at some point.

which was also started at Airbnb.

Remember that Borg is an old thing. At the time they probably started out with chroot jails, because cgroups weren't ready yet. But Kubernetes, Mesos use docker and cgroups(Mesos).

to get an idea look at how Hadoop or spark run on Mesos. You push the data locality into tasks, executors and their caches.

Try Mesos/Marathon in addition to Kubernetes. Been running Kubernetes on Mesos and it's just great.

Or you can use the service account to avoid dancing around gcloud and others. But I agree Google has a problem building stuff for us common humans who do not work at the big G.

Lol He can't because Oracle licensing forbids to do this kind of work and make the results public.

Yes you can use IPython with Spark for Pyspark. Spark-notebook and Zeppelin, gives let's you use Scala with Spark,and support all of the other spark libraries. There are other projects like spark-kernel https://github.com/ibm-et/spark-kernel that let's you use IPython (Jupyter ) I guess, supporting spark's scala API.

Yarn tries to do the same thing, but is quite different. It's skewed toward running short running jobs, like batch jobs, it was meant for hadoop and co after all. Borg/Omega seems more like a combination of Mesos at the scheduler layer and Marathon/Kubernetes on top. It's funny to see how many services they run on top of it though.