trjordan

Kubernetes -> Envoy -> Istio.

Yeah, yeah, it's not rockets to mars or whatever, but I think we've spend DECADES trying to figure out how to abstract away the lower parts of the stack. The last big leap was AWS, and it forced a choice for a lot of people: do you want to work at AWS? or do you want to work on applications that run on AWS? The API is so strong that you can't do both.

There's a similar thing around operating systems, networking tools, and containers. Heroku was a cool PaaS, but ultimately is a little too inflexible to work for everybody. Companies keep rebuilding these internal PaaS systems, and I think Kubernetes has finally nailed it. It's lacking the primitives to do traffic management and introspection, and Envoy / Istio provide those primitives.

Many people see these technologies as an endgame to themselves, but the Istio community has started talking about how they're a platform themselves, and that feels correct. Like AWS, you can abstract away a TON of this stuff, and you will divide people into those that keep a cluster running vs. those that build apps on top of it. The specialization gains will be on the same level as dynamic infrastructure, imho.

Disclaimer: I work at a company that wants to build on top of this new world, but I joined them because I believe this, not the other way around.

Tactically: spot instances, reserved instances, auto-scaling groups, deleting data that we didn't need.

Strategically: I've only ever see it work by making it somebody's job to save money on Amazon, ahead of feature velocity. Find a tool that analyses where you're spending [0], pick the biggest bucket of spend you don't really understand, and drill down until you're sick of looking at it. Make some optimizations around how you're spending that money, rinse, repeat.

Every org I know has done a first pass that's pretty effective, where you buy a bunch of reserved instances for the compute power you need. A couple of the companies I've worked with stalled out at that point. The others figured out real cost / benefit models for real projects and did things like move data from S3 to Glacier, or from hosted MySQL to RDS, or from Cassandra to DynamoDB.

There are only so many free-lunch cuts you can make, so it's worth your while start down a path that allows you to consider trade-offs, which includes empowering somebody to lobby for them.

[0] Seems nice, haven't used it: https://github.com/Teevity/ice. I know the folks at http://cloudhealthtech.com, they're building a pretty solid business if you're willing to spend money to save money.

There is probably something else here that you need to talk to him about.

Why does he feel the need to code up a feature right now, without talking to you about it?

Why is he having trouble keeping everything about the product in his head, with looking in the code as the only fallback?

Why sort of gaps are there in the demo that looking at the code is the best way to address it?

This is not a problem with him, though it's also not a problem with you. It's a problem with the business where the founder who talks to customers feels he has to come back and bang out features. Worst case, he's just bad at his job, but the likely reason for this is that there's a mismatch between you two in terms of stability, urgency, or prioritization.

You're adults. Talk it out.

I saw a great talk on this topic by Dharmesh Shah, founder of Hubspot.

He says that movements are great, but they're necessarily harder to build. Convincing people to change the way they approach their job is _hard_, even if you're right. Then you have to sell them software, which isn't trivial.

Their competition, Marketo, didn't build a movement. They latched on to the existing trend of Marketing Automation, and they built a huge business, at least as successful as HubSpot.

In both cases, the software and ideas were new, but as a company, you don't have to be the standard-bearer for your movement. You just have to support it.

As others in this thread have argued, you don't even have to have a movement to sell software. But fundamentally, if your software doesn't enable people and companies to work differently, you haven't built something deeply valuable. Outside of the early adopter SF-product-hunt folks, nobody wants to buy new software just for kicks. People buy software to change and upgrade their workflow. You can call it a movement or a trend or a wave, but it's always about change. If you want to build something enduring or big or valuable, you need the wind at your back, bringing a useful change to your customers.

Startups are wonderful, freeing experiences. You can build exactly the company and culture you want, balancing so many of the things that you didn't have control over at a bigger company.

Then you get customers.

Customers don't care if you're burnt out.

Customers don't care if you're transparent and fair to your colleagues.

You get grace periods when you're first starting out, or first signed a couple contracts, or after delivering a few key efforts.

But at the end of the day, customers are where the pressure comes from, because they are the sustaining force of your business.

As pointed out in the article, it can be a crime, depending on jurisdiction and local laws. The problem is prosecuting that crime in the same way you'd process a murder with a knife, because the public doesn't see them the same way.

Honestly, I get that. Driving a bike on a road with cars seems dangerous. I do it, and I don't want to die, but it's not the same as walking on the sidewalk or sitting in my yard. If there were bad choices made by the driver, then sure, let's prosecute. But passing a biker going 15mph on a 45mph road doesn't seem crazy. Going 72 in a 65 doesn't seem crazy. I like to think I'm an aware driver, and I haven't hit any bikers yet, but I've seen myself be two distracted moments away from hitting one. It's not crazy that I would be checking my maps app at the same time a police siren comes into view, leading me to miss the biker who'd just turned into the lane next to me.

The problem seems to me that we've built both physical and social structures around roads being for cars. It needs to feel comfortable to bike while taking up a whole lane, both for bikers and cars. ("Drive at the speed bikers go" isn't reasonable, imho.) Bike lanes help a lot, and having a lot of bikers around helps a lot. But we can't just start with "killing a biker is deliberate murder", because it's simply not. It's old attitudes being applied to a new world, and I'd rather find a way to bring people into the new world than punish a few of them for making a mistake.

The point is the OS manages your fingerprint, and even if you have it, you can't easily send a stolen fingerprint to the OS.

If you have access to the hardware, that's a whole different attack vector.

OK, so Facebook kills the home feed tomorrow. What replaces this?

Human-curated news? Either it'll be fast enough that people get sloppy with facts, or it'll be slow enough that people move to faster sources. Clickbait wins.

Another site? Most other sites (Google News, Quora, whatever) use clicks as an important measure of interestingness. Clickbait wins.

Broadcast TV? Have you seen what the networks do when the facts are scarce and the topic is urgent?

Eventually this all gets sorted. In the moment, fake news always wins, especially if there are organizations that are invested in producing it. That's fine. The trick is to help real, verifiable news and level-headed analysis propagate as it comes out as well.

It's a solved problem via reputation, but we've collectively decided that reputation isn't that important in the last 15 years. It's an open but in-progress question for ML / AI. There are probably other models, but we'd first need to change the consumption norms to make them appealing.

Clickbait is always appealing, so it wins when people are trying to find new information before they start to sort through it.

Yep, dropped my phone (totally my fault), got a Pixel XL for $89. Pretty OK, would buy the extended warranty again.

It's easy to be glib about this, but this is actually pretty interesting to me!

This trope shows up in so much sci-fi because it's a pretty quick way to undermine any philosophical questions. What's my purpose? To help run this simulation. Maybe I'll live long enough to find out when the outside computer shuts us down. It's never bothered me personally, because there's not a lot to be done about it. On some level, though, it's a though experiment that I didn't have a good answer to.

Of course, it's still unprovable: the outside simulation might have oodles more atoms than our universe, or different physics, or magic! But that's fundamentally less interesting than knowing there's a "higher" universe with roughly the same physics. That looks much less likely at this point.

It's really cool to read an argument that so strongly disproves something I thought was untestable!

I like feature flags, but it's crazy to suggest that they totally mitigate your risk. Some features will end up with more lines of code hiding the feature than actually implementing it. After it's released, you then have to unwind all that code. That code churn isn't trivial, and "tests pass, ship it" isn't really a good model if you're shipping blocks of work that you don't want to expose to customers.

If your releases processes are perfect, then it's OK, but CI tends to treat release as "if the tests pass, it's perfect" and ships it to all customers. At the very least, you need to be able to compare releases that "don't change anything" to make sure you're not breaking something because some edge of your feature didn't get hidden behind the flag.

I did that, sort of. But I was an exec(ish) before I founded a company.

I think founding a company helped my career, but not necessarily faster than working on a similar exec-level track for 2 years.

Reading this article was interesting to me. I really bristled at the descriptions of founder and employee, but exec feels comfortable. I really enjoy owning a set of responsibilities that's not redundant with anybody else in the organization, and I enjoy working on internal relationships in order to help everybody get their work done better ("politics").

Maybe at some point I'll found another company, because that pressure really does force you to confront the weakest parts of your professional skillset. I learned a ton about myself by doing it.

There are 4 Discovery Service APIs that you can implement: service (SDS), cluster (CDS), route (RDS), and listener (LDS). Colloquially called xDS for all of them.

https://envoyproxy.github.io/envoy/intro/arch_overview/dynam...

You don't call Envoy to make changes. You either update the config file, or you implement an API server with a specific set of endpoints, then have it serve the updated values back. It's a bit tricky, but entirely doable. https://envoyproxy.github.io/envoy/intro/arch_overview/dynam...

(You can do this in nginx, too, and it's similarly annoying. This is a big part of what we do at Turbine Labs -- a UI with a slider between blue and green, and a place to host all those stats, so you don't have to manage it!)

turbinelabs.io is the whole of everything we're building on top of nginx/envoy. Our primary goal is to provide more powerful ways to release software like incremental blue/green deploys or canaries or production previews. That requires a solid data plane (nginx/envoy) and management plane (missing in nginx/envoy xDS APIs) underneath it to work. Nobody runs nginx because it's fun -- one of the logical use cases for a better nginx is better releases, which is what we work on.

The transition has been good! There's certainly a lot more complexity to implement with the Envoy APIs, but that's because there's a lot of power. Working with Matt Klein and the rest of the community has been really positive. We're excited about all the other patterns that unlocks, like circuit breaking and tracing. Those sorts of features are on the wrong side of the cost/benefit equation for most companies, but if you get them for free because you deployed Envoy as ingress and service mesh, they're great to have.

No kidding! I'm glad it exists, but Envoy seems like a much more legit / stable choice.

Do you know if it's been deployed at scale anywhere?

I'm glad this has 2016 in the title! It's now 2017, and we have better options.

Nginx has been the best routing and traffic serving system for a while now, but it's showing its age. In addition to hiding metrics, HTTP 2 support is difficult, there's no clear path to grpc, and (as mentioned in tfa) there is no way to get stats out of it.

Envoy (envoyproxy.github.io) seems like the next thing. Nginx won because of the predominant patterns at the time, and it's still totally fine for serving lots of HTTP traffic. The problem is that if you're thinking microservices or rely on metrics to do your job or have elastic infrastructure, Envoy's feature set is the only real way to handle traffic routing in that world.

Disclaimer: I work for turbinelabs.io, and we're moving from nginx to Envoy to take advantage of a lot of these features to power our product. We've put a lot of effort into that decision, and believe it's the right one.

We at Turbine Labs have been contributing to Envoy recently, and it now builds on MacOS!

https://github.com/envoyproxy/envoy/issues/128

Good.

Twitter needs to keep moving forward. There have been several features that feel like the core of their platform, but based on their revenue growth, they may be holding them back. Personally, eh, 280-length tweets feel weird and long and I don't see the need, but I'll probably get over it. If they can change how Twitter is used open up new audiences and conversations, this is good.

They are trying something big, and that is Good.

Since I don't think it was explicitly mentioned, this is because __slots__ is effectively a named tuple. The size reduction comes from dropping the keys in the dictionary (they're stored once on the class, not every time on the instances), and the speed reduction come from it being an array lookup instead of a dict.

It's also a great way to introduce some static typing. No more setting mistyped class attributes without errors!

I think this blog post is one turn of the crank away from a truth we're all about to learn: don't hand roll your own Kubernetes ingress.

Dealing with the traffic handling between your users and your code is not a trivial problem. Like all good ops problems, you can fix it with good tools, deep knowledge of those tools, fine-grained observability, and smart people running all that.

This has been the recipe for a couple of really successful SaaS offerings. Individual servers? Datadog. CDN? Akamai / Fastly.

Disclaimer: I work at one of those companies, Turbine Labs, and we're trying to make ingress better. Here's a presentation from our CEO on Kubernetes ingress, and why the specification creates the problems that this blog post is trying to fix. https://www.slideshare.net/mobile/MarkMcBride11/beyond-ingre...

One of the dangers of writing structural critiques like this is that it's easy to feel like the problems are structural in a way that makes individual effort irrelevant. On this particular topic, I think there is a huge amount of individual latitude to attack the problem.

I'm lucky enough to work on problems that I find interesting, important, and personally challenging. I optimize for this at the expense of salary and stability. One of the biggest things that's helped me in a cultural safety net of believing that this is OK, with a network of people that help me out. I think there's also enormous potential for individuals to reinforce the values they think will define work in the right way.

- Can you support your friends when their companies attack their work/life balance? Can you help them come up with ways to fight back without hurting their careers?

- Can you share information that's not typically public, in order to help others make better decisions? Salary to peers is an easy place to start. Harder is telling younger folks what Sales Engineers and Product Managers do (and how it can be really fun, fulfilling, and lucrative!).

- Can you normalize behaviors that benefit workers at the expense of companies? Unions used to be great, but I personally think they're the wrong tool for the problems of 2017. Working from home 2 days a week shouldn't break any team, but it's still a cultural taboo.

I'll admit that these are pretty 1st-world problems and I can't do much for people outside of Bay Area tech, but Bay Area tech is where I live and work. We can't change where we are right now, but there's a pile of small problems that are right on your doorstep that you can shove in the right direction. I think there's more ground to be won there than trying to overhaul the whole system.

We overestimate what we can do in a year and underestimate what we can do in a decade ;)

Aside: I disagree somewhat with the framing as a Wicked Problem. That method seems to deliberately conflate multiple problems, in order to make the discussion more sweeping. I think there's value to breaking down "Work" into smaller, more tractable problems. That aside, there's a lot of meat on this bone, so it's good to acknowledge that.

It's hard to prove intent.

If there's a history of these execs selling similar amounts of stock in other quarters, then it's probably not insider trading.

If the paperwork trail started an hour after the first meeting where they learned about the breach, and they had never sold stock before, then it's probably insider trading.

Since the truth is somewhere in between, it's hard to say. It's hard to believe that these people didn't recognize the optics of what they were doing. But, that's why we have investigators and courts.

Marketing, especially for technical B2B products | SF | TR Jordan | terraljordan gmail

I used to be a developer, now I do sales and marketing. It's fun. I like talking to people who are thinking about making the switch, or people who have found themselves trying to sell/market a technical product.

One of the undersold (imho) parts of this post is the system that allows tiering of releases with gates for their release. When most companies talk about CI/CD, they mean that master gets deployed to production, full stop. Rollbacks mean changing the code. In reality, when code hits master, there is ALWAYS a lag while it gets deployed, and it's worth having a system that holds that source of truth. Where release engineering gets interesting is how you handle the happy path vs. a breaking release.

I like that Facebook separated out deploy from release. It means that you can roll the release out relatively slowly, checking metrics as you go. Bad metrics mean blocking the release, which means turning off the feature via feature flag. I think for the rest of the world, that would mean halting the release and notifying the developer.

Disclosure: I work with smart people who spend lots of time thinking about this and writing blog posts like "Deploy != Release": https://blog.turbinelabs.io/deploy-not-equal-release-part-on...

No doubt we talk about it that way.

Not every company is like that. LinkedIn was famously 40-hour weeks in the early days. I just started at a company that's pretty aggressively work-life balanced. We're no doubt a startup (VC funding, high growth, the whole 9 yards), but we just don't believe that 80 hour weeks translates to the best work. You are first and foremost your own caretaker, and it's probably bad if you're in the office 8-6 and working at home.

Culture matters, and it's different at every company. Don't let what's said on Twitter fool you into thinking it's the same everywhere.

[x-posted from the other thread]

No doubt we talk about it that way.

Not every company is like that. LinkedIn was famously 40-hour weeks in the early days. I just started at a company that's pretty aggressively work-life balanced. We're no doubt a startup (VC funding, high growth, the whole 9 yards), but we just don't believe that 80 hour weeks translates to the best work. You are first and foremost your own caretaker, and it's probably bad if you're in the office 8-6 and working at home.

Culture matters, and it's different at every company. Don't let what's said on Twitter fool you into thinking it's the same everywhere.

Of course I've tried a classic HTML app. That seems disingenuous?

You're right! If we just implemented everything correctly, there wouldn't be problems!

Anecdotally, every team I've worked with has converted to a SPA, and the main ergonomic upgrade is that updates to the signup page or the "not found" widget quit getting forgotten whenever we change tweak our design. There always seems to be one page that's out-of-date for 2 months before anybody notices, then it quietly gets fixed up because we forgot that the design tweak was a CSS+HTML change, not pure CSS.

Not that that's perfect -- once you have a SPA, nobody remembers the 404 page, because it's a different HTML base...