rdli

We use Travis for Mac & Linux builds of OSS projects (e.g., https://www.telepresence.io). What's frustrating is that even the paid service for Mac is just as bad. We're on the East Coast, so we can use Travis for Mac in the morning, but it's terrible by lunchtime.

Fun fact: before Travis had multi-stage builds, we were able to get multi-stage builds in Travis by using the fact that mac builds were ALWAYS slower than Linux. So we could have our Linux builds create artifacts that were then used by Mac builds ...

Datawire | C++11 engineer | Boston/SF/remote possible

We are a startup building open source dev tools for Kubernetes developers. We're looking for an open source hacker to work on the Envoy proxy. This means you have to systems programmer fluent in C++11. Lots of bonus points if you've been a committer on an open source project of some reasonable size.

More details: https://angel.co/datawire-io/jobs/265354-principal-software-...

Email careers@datawire.io. Your resume is fine, but even better would be a couple sentences on your C++, OSS, and systems programming experience.

Datawire | Senior/Principal Software Engineer | Boston / Remote possible | Full-time

Datawire is a company focused on building open source tools for organizations adopting Kubernetes and microservices. We're looking for a C++/Go hacker who can work on Telepresence (www.telepresence.io), our open source tool for fast, local dev of services on Kubernetes and Ambassador (www.getambassador.io), our open source API Gateway built on Lyft's Envoy L7 proxy.

We're working on figuring out the best ways to make state-of-the-art tech (Kubernetes, microservices, Envoy, Docker) accessible to developers. Big bonus points if you like to write about technical things, know Kubernetes and/or have prior experience in a microservices architecture, and also know Python / Kotlin / JavaScript (we're a polyglot shop).

Salary $100K - $150K, plus equity and benefits, depending on experience.

Email careers@datawire.io.

What do you mean by PAAS?

In my experience, every company that ends up adopting a microservices architecture ends up building some sort of layer of developer tooling on top of their ops infrastructure (whether it's K8S, EC2, or something else). This is because in microservices, you need to make it easy and cheap to deploy new services, which means developers need some sort of access to operational tooling (or else you need a huge ops team).

In any startup, it's crucial that founders have open lines of communication. There are a lot of decisions to be made, frequently under stress and with imperfect information, and you need to be able to communicate to figure things out.

So, I'd say that 1) you need to sit down with your co-founder and talk about things and 2) if this is going to be a long-term thing, you both need to spend a bunch of time upfront talking about your communication styles and commit to open lines of communication.

The thing you can do at the code level is integrate it with your exceptions and protect against software integration bugs. Imagine that your RPC against a new version of a service fails because of bad data. With a library level circuit breaker, you can catch the failed exception and blacklist the new version of the service. With the network level, your resolution of failure detection is more limited.

Well said.

Besides Netflix, I'd put Twitter as an early pioneer, with their work on Finagle. Both of these companies, for better or worse, took a library-centric approach (Eureka/Hystrix/etc or the Finagle lib). This limited their applicability to the JVM.

The sidecar model that AirBnb pioneered with SmartStack, later adopted by Yelp and others was the cheapest way to get non-Java langs to have similar resilience/observability semantics. And now given the popularity of polyglot architectures, it's probably should be the default choice for companies adopting microservices.

The middleware in this case doesn't work like an ESB in the sense that all connections are point-to-point, intermediated by a service discovery mechanism. So you're not putting all your L7 traffic onto a big bus.

Istio is focused on service-to-service traffic (i.e., in your data center). It uses Lyft Envoy's L7 proxy to add security, resilience, and observability to your L7 traffic.

Imagine you've got 50 microservices and you're using RPC to communicate. You're going to want global rate limiting and circuit breaking behavior to insure resilience, particularly as your topology gets deeper.

There is a use case for extending Istio to the Edge, which is why we wrote Ambassador (it's an API Gateway built on Envoy), and we just released an Istio integration (http://www.getambassador.io/user-guide/with-istio.html).

Full disclosure: Work on Ambassador

We found that setting up Kube on AWS in a prod-ready way was sufficiently complex enough that we wrote up some docs on it: https://www.datawire.io/guide/infrastructure/setting-kuberne..., hoping it would be helpful.

Sorry, I wasn't clear. I was referring to using Envoy as an API gateway, not a specific API gateway product. We had to write some wrapper code on top of Envoy that lets you use a REST API to map URLs to services running in your Kube cluster and then dynamically update Envoy config, etc.

We've been experimenting with Envoy for a few months and have been really impressed with the technology and the responsiveness of the community. We're actually deploying an API Gateway built on Envoy so you can easily extend your service mesh all the way out to the edge. Planning on open sourcing it soon (if anyone wants to give it a try, my email is in my profile).

Here's an alternative, more optimistic view.

CodeStar (and its competitors) doesn't eliminate the future prospects of ops engineers. Instead, they reduce the grunge necessary to get something working. (After all, how much fun is it to actually set up and configure Jenkins ... for the hundredth time.)

So it's possible that over time, ops engineers get more time to worry about things that are harder (or impossible) to automate. For example, organizations run similar, but not exactly the same dev workflows. So understanding your existing dev workflow, and how to tune it is an ongoing thing. And with something like microservices, maybe you have dev workflows that vary by team. Or how do you tune your monitoring infrastructure to let you root cause faster, instead of just taking stuff out of the box. Etc.

I could be wrong, of course. Curious what others think.

If you have funding from some sort of fund or incubator, you can also ask your investors to talk to Google. (Our investors weren't on the official list, but we talked to them, and then Google put them on the list.)

We have a relatively simple cloud app: a couple (micro)services, but we also use Postgres and ElasticSearch. We started using Docker + Spinnaker + k8s, but then we ran into the problem of setting up the app for local dev (where we wanted to use local PG) and prod (where we wanted to use RDS).

<plug>we've been working a bit on an open source tool, pib, that supports setting up multiple environments because we ran into this problem (behind the scenes it uses terraform, k8s, and minikube). would love to hear if anyone here has seen anything similar or has thoughts! https://github.com/datawire/pib</plug>

I love hearing about the evolution of systems, and how different people have converged on the problem from different directions.

[plug] I wrote a post summarizing all that I have figured out about service discovery and wish I had known from the beginning:

https://www.datawire.io/guide/service-discovery-microservice...

Have you considered a strategy of:

- engineering resilience against integration bugs directly into your architecture - setting your infrastructure up so developers can deploy develop versions of new services into production, in a way that only developers can access?

At some level, the complexity of microservices makes a traditional integration testing strategy fairly expensive in terms of infrastructure and process.

It's not about conflict resolution, it's about ordering.

Imagine you have 3 services. A calls B calls C. A request goes into A. The request fails. Since there are 3 services involved, you want to be able to see the log statements of all 3 services, in order. To do this, you need a UUID that is passed from A to B to C and a vector clock that is incremented from A to B to C so you get perfect ordering. Passing the data around is not super hard (you can use HTTP headers, for example). The tricky bit is having a library that does the right thing in A and B and C. And if A is written in Java and B is written in NodeJS, that library better work on both!

Really, it's when you're doing microservices, and each service is deployed to a different container. Typically your business logic is now spread across multiple containers. For example, you might have a recommendation service that looks up location in the geo service. If the geo service returns bad data, then the exception is thrown in the recommendation service. But to debug this properly, you really want to see the logging in both the geo service and the recommendation service, in order. As your topologies get deeper, the problem gets exponentially harder.

I wasn't saying you needed microservices or containers to do distributed logging. My point is that if you're building a distributed system, you need distributed logging (tracing) semantics. The actual logging mechanism (syslog, fluentd, etc) can be used -- as long as you propagate your UUID, etc.

How would passing on the container ID help in mapping multiple log messages to the same request? If a request is going through multiple containers, you'll get multiple container IDs.

To solve this problem, you need a unique trace ID and vector clock, and this actually needs to propagated through your logging system, not through container tags.

I'm surprised the article didn't mention anything about the nuances of actual logging in a distributed architecture. In particular, when a request starts to span multiple containers or microservices, you need a way to correlate all log data to a given request. You typically do this by propagating a trace ID through your logging system, and a vector clock-type thing for causal ordering. If you can't tell which log statements belong to which request and the order in which they occur, debugging is quite hard :-).

(Full disclosure: co-founder of Datawire)

We released a microservices development kit (MDK) last week that includes similar semantics (e.g., circuit breakers, failover) that implements these semantics in Python, JavaScript, Java, and Ruby. The implementation is actually written in a DSL which we transpile into language native impls. We do this to insure interop between different languages. We're working on updating our compiler to support Go and C#, adding richer semantics, and making the service discovery piece pluggable (currently there's a dependency on our own service discovery).

https://github.com/datawire/mdk

There aren't any good or bad answers. There are great people who haven't had a good manager. Good managers also manage not-so-great people. I ask really to understand more about their professional background. Depending on the role, on-the-job training may or may not be OK.

I've experienced the transition from being an engineer to a manager, and then from a manager to director, from director to VP, and now CEO.

I can tell you by far the most material impact on my ability to take on more responsibility has been having a great manager (not just a mentor). A manager knows what work you're being asked to do, sees you try to do that work, and can give you feedback on the work that you're doing. A mentor just does not have that context.

It's so important that one of the questions I ask when interviewing people (at any level) is tell me about a great manager you've had. And a shockingly large number of people (even directors or VPs with impressive resumes) will say "I never had one, but I wish I did."

This is a longer-term advice, but I would make it a career goal if you want to grow to try to find a really great manager.

To expound on the second approach listed, what I've seen is not just a service discovery layer, but client libraries (as opposed to sidecar) that connect to your service discovery (e.g., Netflix Ribbon which is EOL).

Curious what people think of a sidecar approach versus the client library approach? Any preference?

Amazon's fundamental source of competitive advantage is rarely talked about, but I think this is a great example of what that is -- they have built a company that succeeds in very low margin businesses.

So they started in online retail with < 10% gross margin. With AWS with its ~40% GM they have INCREASED the GM of the company. Contrast this with Microsoft and Google which sell software at 80%+ GM. When they do Azure or GCE, they have to decrease their GM. This is hard! FedEx runs around 60% GM. So just think about what this will do for Amazon ...

Datawire | San Francisco | Boston | Backend/Frontend Engineers

We're building open source infrastructure for microservices; see https://github.com/datawire/datawire-connect for details. We're passionate about learning what has actually worked and productizing those concepts; see the Microservices Practitioner Summit we organized online at microservices.com for details.

We're looking for a front-end developer as well as dev infrastructure engineers. If you love learning (and figuring out -- we don't have all the answers) better ways to build software, we'd love to hear from you. Bonus points if you know microservices, multiple languages, AWS, Mesos, or Kubernetes.

https://www.datawire.io/careers/ or marie@datawire.io

There is no valuation in this round. There is no cap. It's just a discount off of a valuation in the future. I would imagine the conversation was "the market is gyrating so much we can't agree on value" so let's defer the valuation until later, and we'll give the investors a 20% (and possibly larger) discount once the public markets establish a valuation.

I don't see a signaling issue here.

Early employees always get diluted. That's what happens when you raise additional capital, regardless of structure.

They make up for this with a combination of a lower exercise price, the vesting clock starting earlier, and typically a larger grant for earlier employees.

Companies retain employees by giving additional refresh grants and/or raises.

I'm not sure why it's OK for Uber to raise a few billion $ and there isn't a fuss about how that's hurting employees when it's not okay here. They both are getting diluted.