kelseyhightower

It's rare that I get to reflect back on my entire tech career and my philosophy towards work and life, but this interview captures it perfectly.

Google Cloud products including GKE (Kubernetes), Cloud Run/Functions, the gcloud CLI, and a number of other utilities and control plane components sit it direct revenue paths. In the case of Cloud Run/Functions (Go support) and GKE, those products generate direct revenue, and the amount is much higher than you would think.

The one benefit you get is protection from bugs in Kubernetes itself and a reduced blast radius. Even if you could produce a secure and H/A cluster, you still leave yourself open to Kubernetes bugs and configuration mistakes such as adding a network policy that blocks all communication across all namespaces.

Multiple clusters protects you from these types of configuration mistakes by reducing the blast radius and providing an additional landing zone to roll out changes over time.

I'm currently testing our (GCP) solution to the CPU throttling you've highlighted. I've been using Vault[1] as my test case, and so far so good. Be on the look out for early sign up if you're interested.

[1] https://github.com/kelseyhightower/serverless-vault-with-clo...

Stay tuned. A lot of the tech behind El Carro can be extracted into a generic controller and serve as the foundation for other databases including Postgres.

Cloud run is attempting to enable that "Just Works" experience, but we are trying to do it by leveraging the best of open source, and native client libraries. Feels like we are getting close and I'll be sure to share this feedback with the rest of the team.

Thanks for your comment. This is something the team keeps top of mind and influences our decisions and road maps. Please continue to keep us honest here. I'll get others to weight in on this, but here's my personal opinion on the matter.

AppEngine has been around for a long time, 12 years to be exact, and in my mind it's a completely different product, and still one of our most successful. While Cloud Run competes with AppEngine on some fronts, customers really appreciate AppEngine's deep integration with other GCP managed services. It's a full blown PaaS.

I expect Cloud Run to improve over time and gain more features that will no doubt match the requirements of many AppEngine customers, some may switch to Cloud Run, but we are not forcing them to.

Cloud Functions already shares a lot of underlying infrastructure with Cloud Run, which shares a lot of underlying infrastructure with AppEngine, and that's by design. These days I like to think of Cloud Functions as a simplified developer experience on top of Cloud Run focused on task orientated workloads backed by events and triggers.

Can we continue to support 3 somewhat overlapping Serverless platforms going forward? I believe the answer is yes, thanks to the shared infrastructure, and the fact we truly believe all three platforms offer a unique set of developer experiences worth preserving. Maybe there is future state where one product satisfies all use cases, but that's not today, so we plan to keep listening to customers, and invest across the board.

One recent example of investing across the board: you can now leverage global load balancing[1] across all three Serverless platforms.

[1] https://cloud.google.com/load-balancing/docs/negs/serverless...

Cold starts can be one of the biggest trade offs when adopting a Serverless platform like Cloud Run, especially with 30 second start up times. We introduced minimum instances[1], currently in beta, which will allow you to specify a minimum number of container instances to be kept warm and ready to serve requests.

There is a cost for doing so as instances kept running in this way do incur billing costs [1].

I've done some work with Ruby in the past, and based on my experience, you might not be able to take advantage of Cloud Run's best feature, concurrency[2], which is another way to reduce cold starts by routing concurrent requests to the same container instance. Ruby maybe blocking in flight requests and forcing us to fire up another instance to handle it. Once the first request has completed there is a chance the container instance processing that request will be spun down, this is something minimum instances can help with. You might need more than one minimum instance to compensate for the lack of concurrency if you really want to see improvement in overall latency.

[1] https://cloud.google.com/run/docs/configuring/min-instances

[2] https://cloud.google.com/run/docs/about-concurrency

McDonald's was my starting point. It's where I developed much of my work ethic and my first taste of leadership in a professional setting. I was a shift manager for most of that time and remember learning about the restaurant business, food cost, working with customers, and managing people.

A lot of stuff did not make the article but who I am today was greatly influenced by that job. I chipped in on the bills, bought my own school clothes, and my first car (1987 Jeep Cherokee), thanks to that job, so for me it was very foundational.

1) The answer to this lies in the question. Agile is a practice and it'll take some to get up to speed. One path I took was taking jobs in tech support, answering phones calls, and finding opportunities to engage with the product teams. You can start by giving feedback on the top issues you're seeing and breaking down ways the product can improve to reduce related support calls. And Boom, you are now apart of the Agile process, providing a feedback loop that helps development teams incrementally improve the product. You also help reduce support cost; don't worry, if you automate yourself out of a job, there will be a better one waiting for you.

That's how you open doors for yourself. Many great Q/A and operations engineers started in tech support where they honed their troubleshooting skills.

2) Yes, I use to get those questions. My answer was, "I'm starting a family, and I'm looking for something a bit more stable, and bigger challenges than the ones I was getting on my own".

It's all about being able to demonstrate your skills. Some times it's whiteboard coding exercises or logging into a live system and "making it work". My IT certifications helped me earlier in my career and now things like GitHub and blog posts are a great way to showcase your skills.

3) Remember, you can always tailor your resume for the job you want. If you want to avoid looking over qualified, then re-frame your experience to align with the job requirements. Instead of "I ran a business doing X,Y,Z", you can re-frame it, "As a _ I did X,Y,Z".

During the interview you can show off your full skill set by giving deep answers demonstrating your understanding of the big picture and how to make a business impact.

If you ever want to discus this stuff further, shoot me a DM on Twitter, I've been where you are, and I know what's possible.

Not at all! I read it as an honest question so I answered it.

I could not fit the entire title "From McDonald's to Google: How Kelsey Hightower became one of the most respected people in cloud computing" when submitting the post.

Ronnie taught me that making people smile should be part of the job. We both learned that you really need to understand the world around you if you want get people to laugh at it.

I read it for the first time today like everyone else and love the way it turned out.

A lot of the technical stuff has been covered in other places. Tom pulled on a different thread, one that even taught me some things about myself. Tom did the homework, interviewed a lot of people, and presented the person behind the keyboard.

My current role does little to describe where I am today. The path for others will be different, and what I think is most important, beyond the technical achievements is the person I've become. The higher you go up in the engineering world the less you lean on the skills that got you there.

In my opinion the best engineer can change the world with zero lines of code.

This is a side of me I would like more people to see. The whole person. So I submitted the article. I grew up in the South and don't live in the Valley. It required a bit of humility to even share this side of me, it's very personal, and in someways, not very flattering, but I wish I knew that successful people also come from very average backgrounds.

Make it happen.

Many of those Tweets reminded me how much I've grown as a person. Those Tweets reminded me that I made the right choice treating everyone with respect and in some cases helped them in their own careers.

It was my way of saying thank you and hoping those stories would inspire others and bring a little joy to their day.

McDonald's helped me establish a work ethic and learn what it means to be a professional and earn a paycheck. I was a shift manager in the 11th grade so I had to learn how to manage people and make sure the numbers added up at the end of the night, while doing homework in the back office, with one eye on the drive through times.

Running a shift at McDonald's required some leadership, you have to be able to work the drive through and clean the bathrooms when the time came. You have to be able to handle any tasks in a fast paced environment. I learned how to be a team player and keep the customers happy. Kinda of the same things I'm doing now.

Maybe I can help fill the gaps.

I ran my own computer store with a small IT consultancy attached to it for a few years. Then I chose to pivot and get a "real job". Things change once you're married with a child on the way.

Like many, I started out doing 3 months to perm contract jobs. The first contract was a Linux system administrator at Google in Atlanta automating the huge fleet of servers there. I learned enough shell scripting to be dangerous, but it was mostly racking and stacking servers, and provisioning top of rack switches -- hello minicom.

3 months later I was working in tech support, for more money, at a company called Vocalocity, who was early in the VoIP game. That's where I learned how to PXE boot and flash Cisco IP phones to work with our custom Asterisk based backends. I was there almost a year and then it was time to move on.

This would continue every three months or so. I held jobs at places like Cox Communications working in the NOC during the night shift so I could be home with my daughter. Three to six months later I quit.

I know what you're thinking, this guy jumped around a lot. I had to, money was tight, and it was the fastest way to get a raise, and it also accelerated my learning. Coming from being your own boss it's really hard to get excited about an entry level job and look forward to working your way up the corporate ladder.

My skills really leveled up when I landed a full time job at Peer 1 Web Hosting, where I started in Tech Support working tickets and taking calls helping people with Linux servers, Plesk, and MySQL. It's true, it's always a DNS problem.

Peer 1 is where I really learned how to write code, it started with bash, and eventually Python. I automated the SSL certificate provisioning system, and wrote some scripts that allowed me to close tickets faster than anyone else.

About 6 months later I was promoted to the engineering team and worked on our automated provisioning system for Server Beach, acquired from Rackspace, which was the part of Peer 1 that hosted YouTube before YouTube was bought by Google. Server Beach ran those "Latency Kills" ads to help sale dedicated gaming servers.

That provisioning system was responsible for allowing people to order a server back in the early 2000s from a web form and have it provisioned in less than an hour. We PXE booted servers, configured RAID controllers, and bootstrapped the OS, including Windows, and handed back an IP address and login creds to the larger system.

I was there for over a year before landing a job that would double my salary around 2008, 2009.

I joined the company mentioned in the article, TSYS, where I brought in a lot of automation, thanks Puppet, and learned enough Java to earn the respect of the broader organization and really help transform the place.

I was a Red Hat Certified Engineer (RHCE) from my days at Peer 1 and I leveraged that set of skills to package all the production applications into fat RPMs (Java, JBoss, and all the war files required to make it work) in the same way we use containers today. I also revamped the CI/CD system leveraging Bamboo with tight Jira integration. I also helped the company move on from CVS to SVN. Don't ask.

We had automated deployments and tight integration with our apps over the course of the 3 years I was leading the team. We automated everything from Oracle running on AIX, to provisioning SSH keys and access to production servers based on Jira tickets and Puppet.

On the software development side I learned enough COBOL to port some of our mainframe jobs to Python. I wrote packed-decimal libraries and EBCDIC encoders so we could use Python going forward to process batch jobs. A big deal in the payments industry.

During my time at TSYS I really got exposed to open source and made some major contributions to Puppet and Cobbler -- I added a feature to Cobbler that enabled us to configure servers while leveraging Cobbler metadata and tools like Puppet.

I also started contributing to distutils and pip back in the day. I did some of the work that made pip and virtulenv play nice together. I also started public speaking at local meetup, PyATL, in Atlanta, and found my voice in the Python community.

It's my PuppetConf 2012 talk that landed me a job at Puppet Labs, the rest is history.

It's me, Kelsey Hightower.

Coinbase built and maintains their own platform that's working for them.

Coinbase provided an analysis worth studying. The major takeaway for me: asking people to manage their own Kubernetes cluster is like asking people to manage their own hypervisors when they just want VMs.

"Unfortunately I forgot the exact quote and the author. (If anyone knows please let me know)."

I said something similar in 2018[1], "You haven't mastered a tool until you understand when it should not be used."

[1]: https://twitter.com/kelseyhightower/status/96342809329245798...

All of the following assumes you plan on running more than a single instance of your monolithic application. If that's not the case, then ignore Kubernetes, and be glad you don't have the problems it was designed to solve.

If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configuration, high availability, log and metrics aggregation, autoscaling, and load balancing across multiple application instances.

Kubernetes provides an opinionated way of doing all of those things. For example, Kubernetes leverages container images and declarative configs for packaging and deploying applications. For many people this approach is much simpler than what Puppet, Chef, and Ansible bring to the table in terms managing applications.

When it comes to high availability Kubernetes provides an orchestration layer across multiple machines, grouped in clusters, that deals with distributing applications based on resource requirements and automatically responding to node and application failures. When applications crash, Kubernetes restarts them. When nodes fail, Kubernetes reschedules the applications to healthy nodes, and avoids the failed nodes in the future.

Many of the patterns for managing applications, even monoliths across a handful of nodes, Kubernetes provides out of the box. In essence, Kubernetes is the sum of all the bash scripts and best practices that most system administrators would cobble together over time, presented as a single system behind a declarative set of APIs.

One other major caveat to all of this.

Just like I would not recommend standing up OpenStack from the ground up in order to deploy your monolithic application across a set of virtual machines, I don't recommend rolling your own Kubernetes cluster either. You should strongly consider leveraging a fully managed Kubernetes offering such as Google Kubernetes Engine, Digital Ocean's Managed Kubernetes, or Azure Kubernetes Service.

Kubernetes External Secrets essentially copies secrets from an external source, in this case AWS Secrets Manager, and creates a native Kubernetes secret in the same namespace where the ExternalSecret custom object was defined.

Kubernetes is being leveraged as a secrets cache and transport layer, with just enough access control thanks to the built-in RBAC functionality. Kubernetes is not the source of truth for the contents of the secret, but provides APIs for referencing, and ultimately transmitting cached secrets to containers under management.

I like this approach. Nice and simple. I do have one suggestion that might improve External Secrets for use with multiple clusters.

By running the External Secrets "control plane" inside the Kubernetes cluster you end up with what I described above. Now consider the case where you have multiple Kubernetes clusters in the same environment -- HA across multiple regions in production.

You have a few options on how to leverage External Secrets in this context. The obvious approach is to deploy the "control plane" across every cluster. There are pros and cons. While you have the ability to do a canary roll out of the External Secrets control plane across multiple clusters, you also take up compute resource across every cluster, and must store the credentials required for the External Secrets control plane to sync secrets from an external store.

The other option would be to enable the External Secrets control plane to live outside of any individual Kubernetes cluster and work as a "global control plane". In this configuration the External Secrets control plane can push secrets across multiple clusters. Users can still define ExternalSecrets objects in each cluster, but you now have the ability to support "global" ExternalSecrets objects which are replicated to every cluster, or a limited set of clusters and namespaces based on configuration.

You can still host the External Secrets control plane using Kubernetes, but in a "admin" cluster, which is separate from the clusters where normal applications are deployed.

No comment.

Any load balancer can be configured or modified to target routable Pod IP addresses and skip node ports altogether. You'll have to integrate with the Kubernetes Endpoints API[1] and support dynamic backends. Another option would be to leverage Kubernetes' DNS and the SRV records[2] backing each service.

The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, not arbitrary endpoints such as containers, a limitation that will go away over time.

[1] Envoy with Kubernetes Endpoints integration: https://github.com/kelseyhightower/kubernetes-envoy-sds

[2] https://kubernetes.io/docs/concepts/services-networking/dns-...

It will as I continue to work on echo until it's "finished". This is the result of 4 hours of learning and writing my first assembly program.

This is great feedback, which I plan to use to improve the echo program. I'm just learning (on my own), and I figured I would just post my progress and I would get some feedback; it worked!

echo is far from finished, and it's safe to say "I don't know what the hell I'm doing", but hey, I gotta start somewhere.

Yes, this is my first assembly program. I had to look up every instruction and it took me hours to understand even the basics, but it was worth it. I have a much better understanding of x86 assembly and plan to write larger programs to continue learning in 2017.

I went with 32 bit because all the examples were 64 bit so I forced to learn the nasm and ld flags to get my program to compile, link, and run. I also learned a lot about the different registers available to 32 and 64 bit programs.

ConfigMaps offer everything I need for a tool like konfd, and ConfigMaps work with the majority of clusters deployed today. While I could have used a ThirdPartyResource, that would require me to design a custom scheme and write a little more code up front.

Also, there are a few issues [1] with ThirdPartyResource objects including the lack of validation and inconsistencies when interacting with ThirdPartyResource objects through the Kubernetes API [2]. That's not to say ThirdPartyResource's should not be used, but it's not a decision take lightly.

Now, if konfd were to grow or add more features, then I think a ThirdPartyResource would be the way to go -- mainly because ConfigMaps require the use of "reserved" keys and annotations to configure behavior.

[1] https://github.com/kubernetes/kubernetes/issues/22768

[2] https://github.com/kubernetes/kubernetes/issues/29542