tristanz

Kubernetes supports this via PersistentVolumes. Supported types include:

GCEPersistentDisk AWSElasticBlockStore AzureFile FC (Fibre Channel) NFS iSCSI RBD (Ceph Block Device) CephFS Cinder (OpenStack block storage) Glusterfs VsphereVolume HostPath (single node testing only – local storage is not supported in any way and WILL NOT WORK in a multi-node cluster)

http://kubernetes.io/docs/user-guide/persistent-volumes/

"The Evolution of Massive-Scale Data Processing" presentation from Google provides a good comparison of Flink with other tools:

https://docs.google.com/presentation/d/10vs2PnjynYMtDpwFsqmS...

Yeah, I'm talking 10k+, so this is only specific types of applications. In basic benchmarks, ZFS seemed to perform fine well into with 10s of thousands of snapshots. This is still far more snapshots than basic containers will create, so it's really only a problem for situations where you're doing tons of checkpointing.

I can confirm that btrfs has major problems with tons of snapshots. The entire system will hang for minutes on random intervals and go completely unresponsive for hours when using admin commands. The many snapshots use case is pretty common when dealing with containers, even it everything works fine for simple setups.

LeCun has a comment on this paper here: https://www.facebook.com/yann.lecun/posts/10153293764562143

I don't get the benefit of this approach -- it's too subjective. Is reset-password part of auth or settings feature? What is the link between routes and features? If I'm on a page and see a bug, how do I quickly find the code behind it if it's not linked to the route hierarchy?

The easiest mental model for me is to just map files/folders directly to the route hierarchy. That way you can rapidly build out the overall navigation scheme while building functionality. There are very few subjective decisions besides the route name. New developers can immediately grasp how things are organized and where things belong by interacting with the actual site. If they see a bug, or somebody suggests an improvement on a specific page, they can immediately find the code they'll need to modify.

With features you have to say something like: "Find routes.js, find the route forgot-password within react-router component, see what component it references, now go to top of the file and see the location for that components import. If it's a folder with an index.js file, look in that file and see what component is actually being exported, once you find the underlying component, open that file and try to find the bug."

If anybody is interested in working on a k8s based data science platform, drop me a line tristan@sense.io.

Sense - https://sense.io - San Francisco - Backend Engineer (Go, Docker, Kubernetes)

We're building a next-generation platform for data science and big data analytics. We're looking for a backend engineer to help scale our data science infrastructure and accelerate new initiatives. You should value reliable code and have experience with with Go, distributed systems, Docker, and cloud orchestration tools. This is the ideal role for somebody that wants to join an early stage startup tackling a deeply technical domain. We're a small team but have some of the largest companies in the world as customers.

Competitive salary and significant equity.

Email tristan@sense.io if you're interested in learning more.

This is terrific if the benchmarks are accurate. The vast majority of these types of libraries end up slower than JSON when implemented in JS. What were the key performance tricks?

The benchmarks show a fair number of -1s (not supported). What isn't supported exactly? I don't see these limitations listed in the limitations section.

Why not just estimate p(A - B | observed data) and be done with it?

The great thing about Stan is it is geared toward practitioners doing real work in science and social science, but still manages to push the boundary of stats research (NUTS, ADVI, penalized ML). Other probabilistic programming languages are more expressive, but are typically much harder to use for day-to-day work.

That being said, there's still a huge gulf between practitioners in science and social science that are building parametric Bayesian models and practitioners in the deep learning / ML community that are focused on building more general, scalable, machine learning algorithms for tasks like machine translation and question answering. It would be amazing if somebody could reconcile these communities by showing deep learning models can be expressed and fit effectively side-by-side with more parametric models.

9/10 cured? Not even close. If you know anybody with Alzheimer and Parkisons you'll know that while exercise can help, it's not remotely a cure for these devastating diseases.

Is Google Brain group primarily using Caffe2, Torch7 or something else? It's great to see deep learning innovation happening in Python. I'd previously heard they were focused on Torch7 like Facebook.

Yeah, except if you look closely at what GraphQL actually is that description is very misleading.

The spec has a better description: https://facebook.github.io/graphql/#sec-Overview

" GraphQL is a query language designed to build client applications by providing an intuitive and flexible syntax and system for describing their data requirements and interactions.

GraphQL is not a programming language capable of arbitrary computation, but is instead a language used to query application servers that have capabilities defined in this specification. GraphQL does not mandate a particular programming language or storage system for application servers that implement it. Instead, application servers take their capabilities and map them to a uniform language, type system, and philosophy that GraphQL encodes. This provides a unified interface friendly to product development and a powerful platform for tool‐building."

This is very different goal that most graph query languages.

No. They're basically totally unrelated despite the name. GraphQL is a way to build and query a typed API that exposes nested resources. It's an alternative to REST. There is no graph computation happening. I'd view it as a more flexible version of REST with types.

Where is this documented?

Interesting comment on that article by Siddharth Agarwal:

"(I work on source control at Facebook.)

Every problem you mentioned with monolithic repositories is a well-known problem with Git (though some of them do have workarounds, such as clone with --depth). None of them are issues in principle.

With Mercurial we're aiming to provide tooling that scales well while still maintaining DVCS workflows like local commits."

Sense - https://sense.io - San Francisco (SOMA)

We're building a next-generation platform for data science.

We're looking for a backend engineer to help build and scale our data science infrastructure. This is the perfect role for somebody that loves Go, Docker, Kubernetes, Mesos, Hadoop/Spark, and distributed systems.

We're a small team and can offer competitive salary and significant equity.

Want to learn more? Email tristan@sense.io with a brief introduction.

Are there any plans to release the Relay (client side) component of this?

The web components spec is starting to look like a huge mess in comparison with modern JS framework component models. I think it's unlikely web components will much traction as a first tier approach to application development.

Very necessary. A few questions:

- How big can records be? Are they stored as a single object in the persistence layer or using something like materialized paths. The firebase data model is one huge JSON structure, so this looks different.

- How are conflicts handled? Last writer on server wins or last writer anywhere? Do you ensure eventual consistency of all the clients even when you apply optimistic writes on the client?

- Can data be larger than the cache size? How is cache eviction handled?

They are competitors, but life isn't simple. The reality is many frameworks currently only run on Mesos or YARN, and Kubernetes has not reached V1, so larger installations typically need multiple frameworks. Mesos is a proven way to run multiple frameworks side-by-side. Ebay's YARN on Mesos is another example. But where this all leads remains to be seen.

Sense - San Francisco - https://sense.io - VP of Engineering

We’re building a cloud data science platform to power the future of science and business. We're a VC-backed seed stage company and are working with some of the largest companies in technology, finance, and pharma.

We're looking for a VP of Engineering to help scale our engineering efforts. We view this role as a late-stage cofounder. You'd be responsible for managing engineering and accelerating product development as we scale. Our stack consists of Go, Kubernetes, Node.js, Docker, React, and Angular.js. You should have strong knowledge of cluster scale distributed systems and a passion for building efficient engineering processes.

Compensation: We're willing to offer competitive salary and significant (late stage co-founder) level equity to the right candidate. If you're looking for such a role, email us to start a conversation.

Email: tristan@sense.io

Website: https://sense.io

The problem with this is there is no way to replay the log when clients connect without replaying the whole topic, which can be huge.

Yeah, this is a good way to look at it. But it's also my point. Partitions are about parallelism and don't always fit the data model or domain. While you can reduce the partitions to 1, this is limits parallelism. It's not always easy to design a partitioning scheme that preserves the replication semantics you want. And these semantics vary from having something totally ordered, to having something that you can replay on a very fine-grained level (to replicate to the client for instance.) Most discussions of the advantages of logs really emphasize how amazing a totally ordered log is for replication, but that's not actually what production deployments look like so you still need to think carefully about what happens when writes are being applied to your datastores without a clear order.

Martin's talks and blog posts are aways awesome. I'm really excited to see how this plays out for real applications.

The one thing I'm always somewhat confused by though is how a "totally ordered log" intersects with the reality of a partitioned log. The simplicity of a log seems to break down a bit when you partition.

For instance, imagine I want to implement multi-key transactions on top of a distributed datastore. With a totally ordered log this is easy. But with a partitioned log, it becomes much harder.

Alternatively, imagine I want to implement a collaborative editing app like Google Docks or something like Slack. A natural design would be to to have millions of independent logs. I can then replay logs to get current state and watch logs to keep it updated. But as far as I'm aware, partitioned logs like Kafka do not actually support millions of topics. So there's no way to replay a log for something like a channel or document.

Can somebody explain to me the underlying theory of this type of book?

The only books that have ever felt coherent to me start with p(data, unknown) as being an approximate model of some domain. Everything then follows smoothly as inference, modeling, and computational methods or shortcuts.

I'd love to see an easy to manage version for containerized cloud stacks like Kubernetes and Mesos. I think the AWS move here validates that NFS is still an ok pattern to use for some applications.

I've had mixed experiences with Gluster a year ago, including lost files, so something that was rock solid and easy to manage would be a great product.

Are there any more details on how Google is designing these new API endpoints? The information here is pretty sparse. It seems like a big strategic initiative to unify API design though.

You can upload small to medium files directly into your project, which lives on a distributed filesystem. For larger data, we typically recommend pulling data directly from S3 or production databases.